Node.js has become one of the most popular runtimes for executing server-side JavaScript programs. Node.js applications typically resort to hundreds of publicly available third-party packages hosted in the Node package manager (npm) registry, a large repository containing over a million of ready-to-use publicly available third-party libraries. While the availability of a large number of packages is appealing and eases the development of complex software, it is fundamental that such packages have high quality, to reduce chances that a buggy package affects potentially many applications and users. Unfortunately, the lack of uniform code-quality standards exposes the npm ecosystem to bugs and vulnerabilities, whose negative impact increases with the popularity of the affected package. One important aspect of the test quality of a package is its code coverage. Assessing the code coverage of the packages in the npm ecosystem is crucial and has to be done via a fully automated tool, due to the large amount of packages that should be analyzed.
We developed TESA, a novel tool that enables a comprehensive and automatic evaluation of code-coverage metrics on an arbitrary set of npm packages. TESA runs on GraalVM and is built on top of NodeProf. Our tool evaluates and assembles test suites for any package, including the original tests from the package developers, located in the package’s development repository.
As the original tests often yield only limited coverage, it is beneficial to find additional third-party tests as a complement or alternative. We propose a novel methodology to automatically extend the code coverage of a target package by treating the tests of third-party packages that rely on the target package (called dependents) as tests for the target package itself. By executing the tests of the dependents (called dependent tests) the resulting workload can exercise the code of the target package, thus behaving as if the tests were written for the target package itself. We implement our test-coverage extension methodology in TESA. To the best of our knowledge, TESA is the first framework for the automatic assembly of test suites fully leveraging the abundance of code available in the npm registry. Thus, TESA assembles a new test suite for arbitrary packages that combines two complementary sets of tests, i.e., the original tests and the dependent tests.
TESA provides a compaction algorithm that produces an optimal compacted test suite, i.e., containing only those tests necessary to maximizing code coverage while minimizing the total test execution time. Finally, in addition to extending code coverage, TESA enables the automatic execution of arbitrary dynamic program analyses (DPAs) on the testing code (including both original and dependent tests) of any set of npm packages, allowing the automated execution of large-scale DPAs “in the wild”. Thanks to TESA, we run a state-of-the-art DPA for identifying performance problems in npm packages. Our results demonstrate that dependent tests allow finding performance problems that cannot be detected with the original tests of a package, further confirming the benefits of extending code coverage by exploiting tests written for dependents.
A research paper describing this work has been accepted at AST’21 [1]. We are actively working on releasing TESA as open-source software.
Key Publications
[1] Haiyang Sun, Andrea Rosà, Daniele Bonetta, Walter Binder: Automatically Assessing and Extending Code Coverage for NPM Packages. AST 2021: 40-49 [pdf][video][slides]