In Vivo Testing
About In Vivo Testing
In this project, we are seeking to develop a new testing approach in which an application automatically tests itself while it is running in the deployment environment (the field), as opposed to in the development and testing environments (the lab). The thought is that an application running in the field is more likely to encounter states that were not anticipated prior to release, or could not have been tested due to various constraints (e.g. time).
In Fall 2007, we developed a prototype framework called Invite, which is described in our tech report and was presented as a poster at ISSTA 2008 (a variant of this paper was presented at ICST 2009, and is available here). This implementation uses an AspectJ component to instrument selected classes in a Java application, such that each method call in those classes has some chance (configurable on a per-method basis) of executing the method’s corresponding unit test. When a test is run, Invite forks off a new process in which to run the test, and the results are logged.
We also developed a distributed version of Invite, which seeks to amortize the testing load across a community of applications; a paper was published in the student track of ICST 2008. This version currently uses only one global value for the probability of running a test, instead of one per method, however. That value is set by a central server, depending on the size of the “application community”.
In Spring 2008, we looked at various mechanisms for reducing the performance impact of Invite, e.g. by assigning tests to different cores/processors on multi-core/multi-processor machines, or by limiting the number of concurrent tests that may be run. We also looked at ways of balancing testing load across members of a community so that instances under light load pick up more of the testing. Lastly, we created a modified JDK that allows Invite to create copies of files so that in vivo tests do not alter the “real” file system.
In Fall 2008, we ported the Invite framework to C and evaluated more efficient mechanisms for injecting the instrumentation and executing the tests. We also investigated fault localization techniques, which collect data from failed program executions and attempt to discover what caused the failure.
Recently we have investigated ways to make the technique more efficient by only running tests in application states it hasn’t seen before. This cuts down on the number of redundant states that are tested, thus reducing the performance overhead. This work has potential application to domains like model checking and dynamic analysis and was presented in a workshop paper at AST 2010.
Currently we are looking at ways to apply the In Vivo approach to the domain of security testing. Specifically, we devised an approach called Configuration Fuzzing in which the In Vivo tests make slight changes to the application configuration and then check “security invariants” to see if there are any vulnerabilities that are configuration-related. This work was presented at the 2010 Workshop on Secure Software Engineering.
In 2012-2013, we are investigating techniques to efficiently isolate the state of the tests, so as to avoid the effect of the tests on external systems.
Open research questions include:
- Can the overhead be reduced by offloading test processes to other machines? This is especially important when the application is running on a single-core machine.
- What sorts of defects are most likely to be detected with such an approach? How can we objectively measure the approach’s effectiveness at detecting defects?
- How can the tests be “sandboxed” so that they do not affect external entities like databases? We currently assure that there are no changes to the in-process memory or to the file system, but what about external systems?
Interested students should view our project student ads.
Contact: Jon Bell (jbell@cs.columbia.edu)
Team Members
Faculty
Prof. Gail Kaiser, kaiser@cs.columbia.edu
Graduate Students
Jonathan Bell, jbell@cs.columbia.edu
Former Members
Matt Chu
Waseem Ilahi
Chris Murphy
Del Slane
Ian Vo
Moses Vaughan
Links
Publications
ICST 2009 paper on in vivo testing
ICST 2008 paper on distributed in vivo testing
SecSE 2010 paper on configuration fuzzing
AST 2010 on conducting tests only in previously-unseen states
Related Work
Gamma project at Georgia Tech
Skoll project at Univ of Maryland
Cooperative Bug Isolation at Univ of Wisconsin
Available student project positions:
Projects in In Vivo Testing
We are seeking to develop new approaches to detecting defects in fielded systems that are not possible to fully test and debug in the lab – which includes virtually every complex software system. One promising approach is for an application to automatically test itself while it is running in the deployment environment. This kind of testing can done at the system level, e.g., to detect configuration errors peculiar to the installation, or at the internal unit (function or class) level, which can detect a much wider range of problems by running crafted test cases in the varied application states reached during customer operation. We call the latter in vivo testing (‘in vivo’ refers to inside the living organism – the software system – in the field as opposed to ‘in vitro’ in the glass, i.e., the development lab). We have previously developed a proof-of-concept implementation of an in vivo framework that required source code modification to insert test case instrumentation. We now plan to implement a more robust, higher performance framework using dynamic binary instrumentation, and apply the approach more generally, including to security concerns (see below) and concurrent systems. We also need to develop a more concrete approach to devising the in vivo test cases.
Contact: Jon Bell jbell@cs.columbia.edu