In Vivo Testing Project

In this project, we are seeking to develop a new testing approach in which an application's unit tests are executed while the program is running in the deployment environment (the field), as opposed to in the development and testing environments (the lab). The thought is that an application running in the field is more likely to encounter states that were not anticipated prior to release, or could not have been tested due to various constraints (e.g. time).

In Fall 2007, we developed a prototype framework called Invite, which is described in our tech report and was presented as a poster at ISSTA 2008 (a variant of this paper was presented at ICST 2009, and is available here). This implementation uses an AspectJ component to instrument selected classes in a Java application, such that each method call in those classes has some chance (configurable on a per-method basis) of executing the method's corresponding unit test. When a test is run, Invite forks off a new process in which to run the test, and the results are logged.

We also developed a distributed version of Invite, which seeks to amortize the testing load across a community of applications; a paper was published in the student track of ICST 2008. This version currently uses only one global value for the probability of running a test, instead of one per method, however. That value is set by a central server, depending on the size of the "application community".

In Spring 2008, we looked at various mechanisms for reducing the performance impact of Invite, e.g. by assigning tests to different cores/processors on multi-core/multi-processor machines, or by limiting the number of concurrent tests that may be run. We also looked at ways of balancing testing load across members of a community so that instances under light load pick up more of the testing. Lastly, we created a modified JDK that allows Invite to create copies of files so that in vivo tests do not alter the "real" file system.

In Fall 2008, we ported the Invite framework to C and evaluated more efficient mechanisms for injecting the instrumentation and executing the tests. We also investigated fault localization techniques, which collect data from failed program executions and attempt to discover what caused the failure.

We are currently looking at ways to apply the In Vivo approach to the domain of security testing. We are also integrating the framework with a lightweight virtualization technology to provide a more robust "sandbox" for the tests.

This semester, we intend to address the following research questions:

• Can the technique be made more efficient by only running tests in application states it hasn't seen before? We currently run tests probabilistically or according to the system load; however, certain states may unnecessarily be tested multiple times.
• Can the overhead be reduced by offloading test processes to other machines? This is especially important when the application is running on a single-core machine.
• What sorts of defects are most likely to be detected with such an approach? How can we objectively measure the approach's effectiveness at detecting defects?
• How can the tests be "sandboxed" so that they do not affect external entities like databases? We currently assure that there are no changes to the in-process memory or to the file system, but what about external systems?

Faculty
  Prof. Gail Kaiser, kaiser [at] cs.columbia.edu

Graduate Students
  Chris Murphy, cmurphy [at] cs.columbia.edu
  Huning Dai, hdd2210 [at] columbia.edu
  Waseem Ilahi, wki2001 [at] columbia.edu

Former Members
  Matt Chu, mwc2110 [at] columbia.edu
  Del Slane, djs2160 [at] columbia.edu
  Ian Vo, idv2101 [at] columbia.edu
  Moses Vaughan, mjv2123 [at] columbia.edu