• What sort of data do users consider private or sensitive?
• In what ways to users use this data?
• How can we reconcile existing encryption techniques with user requirements?

We intend to describe further details on this system publicly later in 2013.

Team Members

Faculty
Prof Gail Kaiser, kaiser@cs.columbia.edu
Prof Roxana Geambasu, roxana@cs.columbia.edu

Graduate Students
Jonathan Bell, jbell@cs.columbia.edu
Riley Spahn, riley@cs.columbia.edu

Advances in machine learning provide an opportunity to overcome the intuition wall. In the last decade there have been many major advances in machine learning on graphs motivated by need/benefit of mining behaviors in social networks and enabled by cheap commodity computing. The PIs propose to leverage these advances to discover and program new computer architectures. By viewing program execution as a graph, clustering these
graphs, and mining them for similarities, the PIs propose to discover new behaviors that architects and microarchitects can use to develop new on-chip acceleration structures.  The PIs also propose to study how legacy code can semi-automatically be converted to execute on the architectures with the new accelerators.

We are working with Profs. Simha Sethumadhavan and Tony Jebara to improve workload characterization in a very fundamental way: traditionally computer architects have relied on scalar characteristics (such as instruction mixes, hit/miss rate, etc.) to characterize groups of programs. We aim to discover deep program behaviors on a large scale by using program graphs as the intermediate representation.

To enable this advance, the three groups are working on:

1. Hardware feature specification: development of internal representations that capture behaviors/features that are suited for architectural discover.
2. Machine Learning: development of graphical clustering algorithms that are tuned to discover architectural behaviors.
3. Software Engineering: techniques for adapting legacy code clones to exploit new architectures.

The Neurokernel will:

1. Enable computational/systems neuroscientists to exploit new connectome data by directing emulation efforts at interoperable local processing units (LPUs), functional subdivisions of the brain that serve as its computational substrate;
2. Capitalize on the representation of stimuli in the time domain to enable the development of novel asynchronous algorithms for processing spikes with neural circuits;
3. Serve as an extended machine that will provide abstractions and interfaces for scalably leveraging a powerful commodity parallel computing hardware platform to study a tractable neural system;
4. Serve as a resource allocator that will enable researchers to transparently take advantage of future improvements in this hardware platform;
5. Enable testing of models, both by easing the detection and localization of programming errors and by operationally verifying the models’ designs against time-encoded signals to/from live fly brains in real-time;
6. Accelerate the research community’s progress in developing new brain circuit model by facilitating the sharing and refinement of novel and/or improved models of LPUs and their constituent circuits by different  groups.

To ease its use by the neuroscience community and enable synergy with existing computational tools and packages, we are developing our software framework in Python, a high-level language that has enjoyed great popularity amongst computational neuroscientists.

As we enhance Neurokernel to model new regions of the fly brain, there may be a negative effect on previous models for other regions.  As the fly brain model(s) will be developed in iterative software development cycles, it will be imperative to ensure that each iteration re-verifies the platform and its individual LPUs against the actual fly brain neuropils.  We would like these tests on the Python code to be conducted automatically, without requiring the use of our fly interface equipment — which is manually intensive to operate.  We are constructing a tool to simulate the fly brain interface for software testing purposes that will capture the stimuli provided to the fly along with its responses.  From these sets of inputs and outputs, the tool will automatically generate test cases that recreate the same experiment without the need for repeated interfacing with the fly. This tool will also be used to automatically generate regression tests for the Neurokernel software that depend on other external factors.

Additional information is available on the Bionet Lab website.  Interested students should also view Bionet’s project student ads.

Contact: Lev Given (lev@columbia.edu)

Team Members

Faculty
Prof Gail Kaiser, kaiser@cs.columbia.edu
Prof Aurel Lazar, aurel@ee.columbia.edu

Graduate Students
Lev Givon, lev@columbia.edu
Nikhil Sarda, ns2847@columbia.edu

Conventional software testing checks whether each output is correct for the set of test inputs. But for some software, it is not known what the correct output should be for some inputs — yet it is still important to detect coding errors in that software, so they can be fixed. This dilemma arises frequently for machine learning, simulation and optimization applications, often “Programs which were written in order to determine the answer in the first place. There would be no need to write such programs, if the correct answer were known.”  [Weyuker EJ. On testing non-testable programs. Computer Journal, November 1982; 25(4):465–470.] As these kinds of applications are frequently used in public infrastructure and biomedical research (domains targeted in this research), it is critical to detect and fix errors before a calamity occurs.

In this project, we seek to develop testing techniques that successfully find defects in software without a test oracle (known as “Non-Testable Programs” although that phrase is not intended to be literal).  The conventional way to test software is to devise pairs of input data and expected output results, run the program with the prepared input data and then compare the actual results to those expected; alternatively, preparation of expected output results can be omitted if there is some easy way to check whether the actual results are correct.  The mechanism to do this is in either case called a test oracle.  However, there are large and important classes of software applications – in machine learning, simulation, optimization, etc. domains – where there simply is no test oracle, and for the general case it is impossible (or at least impractical, e.g., it would take years at prohibitive cost, and/or would require unacceptable experimentation with human subjects) to determine whether the actual output is indeed correct for given input.  Yet we would still like to be able to devise test cases that enable us to find coding errors.  We have investigated several approaches, of which the most successful (at finding both previously unknown bugs and seeded bugs in real-world applications) has been “metamorphic testing”.   Fortunately, many such applications reflect ‘metamorphic properties’ that define a relationship between pairs of inputs and outputs, such that for any previous input i with its already known output o, one can easily derive a test input i’ and predict the expected output o’. If the actual output o” is different from o’, then there must be an error in the code.

Metamorphic testing was previously designed by others as a general technique for creating follow-up test cases based on existing ones, particularly those that have not revealed any failure, in order to try to find uncovered flaws. Instead of being an approach for test case selection, it is a methodology of reusing input test data to create additional test cases whose outputs can be predicted. In metamorphic testing, if input x produces an output f(x), the function’s (or application’s) metamorphic properties can then be used to guide the creation of a transformation function t, which can then be applied to the input to produce t(x); this transformation then allows us to predict the output f(t(x)), based on the (already known) value of f(x). If the output is not as expected, then a defect must exist. Of course, this can only show the existence of defects and cannot demonstrate their absence, in the case of software with no test oracle, since the correct output cannot be known in advance (and even if the outputs are as expected, both could be incorrect). But metamorphic testing provides a powerful technique to reveal defects in such non-testable programs by use of a built-in pseudo-oracle.

Our initial work evaluated the feasibility of applying metamorphic testing to machine learning apps, resulting in a paper published at SEKE 2008 in which we categorized six types of metamorphic properties that such applications may have.

We then created a tool called Corduroy to automate the process by allowing developers to specify individual functions’ metamorphic properties using the specification language JML; these properties could then be checked using JML Runtime Assertion Checking. This yielded a paper that was presented at ICST 2009. We further developed a technique called Metamorphic Runtime Checking, and an implementation framework called Columbus, that are described in a tech report.

For system-level testing, we developed a tool called Amsterdam and an approach called Automated Metamorphic System Testing. This allows for checking of the application’s metamorphic properties at runtime, using the real input from actual executions. However, the checks are conducted in parallel (for performance reasons), and any side effects are hidden from the user.  Amsterdam was presented at ISSTA 2009, and is presented in more detail in a tech report.

We have recently completed two studies where we applied the metamorphic testing approach to healthcare simulation software with no test oracle: the JSim discrete event simulation system (developed by the University of Massachusetts at Amherst) for modeling the flow of patients through a hospital emergency room, and the GCS Glycemic Control Simulator (developed by the University of Pennsylvania) to simulate the behavior of different closed-loop insulin titration algorithms on a virtual patient.  Both studies were in collaboration with the developers of the software under test, and involved bugs seeded through the standard mutation testing technique.  Our metamorphic testing approach was able to detect all of the errors in JSim and 60% of the inserted errors in GCS.    Our experiments in these and other domains demonstrate that metamorphic testing is more effective at detecting defects in applications without test oracles than using a “partial oracle” (simple input for which the correct output can be known) or runtime assertion checking, although these techniques find some defects not found by metamorphic testing and thus should be used in combination.

We now plan to investigate methodology for determining the metamorphic properties of software and for devising good test cases from which the secondary tests can be derived. The project will extend the inputs/outputs considered in our previous work on metamorphic testing to focus on application state, before and after, rather than just functional parameters and results. The research will also extend the pairwise relations implied by metamorphic properties to ‘semantic similarity’ for nondeterministic applications, applied to profiles from numerous executions, since an exact relation cannot be expected to hold for a single pair of test executions. These extensions will enable treatment of more sophisticated properties that preliminary experiments have shown to reveal defects that were not detected otherwise.

Interested students should view our project student ads.

Contact: Jon Bell (jbell@cs.columbia.edu)

Team Members:

Faculty
Prof. Gail Kaiser, kaiser [at] cs.columbia.edu

Graduate Students

Jonathan Bell, jbell@cs.columbia.edu

Former Members
Sahar Hasan
Lifeng Hu
Chris Murphy
Kenny Shen
Miriam Melnick
Qiang Deng
Kunal Mishra

Links

Publications

Christian Murphy, M. S. Raunak, Andrew King, Sanjian Chen, Christopher Imbriano, Gail Kaiser, Insup Lee, Oleg Sokolsky, Lori Clarke, Leon Osterweil. On Effective Testing of Health Care Simulation Software. 3rd International Workshop on Software Engineering in Health Care, May 2011.

Xiaoyuan Xie, Joshua W. K. Ho, Christian Murphy, Gail Kaiser, Baowen Xu and Tsong Yueh Chen.  Testing and Validating Machine Learning Classifiers by Metamorphic Testing.  Journal of Systems and Software, Elsevier, 84(4):544-558, April 2011.

Christian Murphy and Gail Kaiser. Automatic Detection of Defects in Applications without Test Oracles. Technical Report CUCS-027-10, Dept. of Computer Science, Columbia University, October 2010. (Amsterdam)

Christian Murphy and Gail Kaiser.  Metamorphic Runtime Checking of Non-Testable Programs. Technical Report CUCS-042-09, Dept. of Computer Science, Columbia University, October 2009. (Columbus)

Christian Murphy, Kuang Shen and Gail Kaiser. Automatic System Testing of Programs without Test Oracles. International Symposium on Software Testing and Analysis, July 2009.

Christian Murphy, Kuang Shen and Gail Kaiser. Using JML Runtime Assertion Checking to Perform Metamorphic Testing in Applications without Test Oracles. 2nd IEEE International Conference on Software Testing, Verification and Validation, April 2009.

Christian Murphy, Gail Kaiser, Lifeng Hu and Leon Wu. Properties of Machine Learning Applications for Use in Metamorphic Testing. 20th International Conference on Software Engineering and Knowledge Engineering, July 2008.

Christian Murphy, Gail Kaiser and Marta Arias. Parameterizing Random Test Data According to Equivalence Classes. 2nd ACM International Workshop on Random Testing, November 2007.

Christian Murphy, Gail Kaiser and Marta Arias. An Approach to Software Testing of Machine Learning Applications. 19th International Conference on Software Engineering and Knowledge Engineering, July 2007, pp. 167-172.

In this project, we are seeking to develop a new testing approach in which an application automatically tests itself while it is running in the deployment environment (the field), as opposed to in the development and testing environments (the lab). The thought is that an application running in the field is more likely to encounter states that were not anticipated prior to release, or could not have been tested due to various constraints (e.g. time).

In Fall 2007, we developed a prototype framework called Invite, which is described in our tech report and was presented as a poster at ISSTA 2008 (a variant of this paper was presented at ICST 2009, and is available here). This implementation uses an AspectJ component to instrument selected classes in a Java application, such that each method call in those classes has some chance (configurable on a per-method basis) of executing the method’s corresponding unit test. When a test is run, Invite forks off a new process in which to run the test, and the results are logged.

We also developed a distributed version of Invite, which seeks to amortize the testing load across a community of applications; a paper was published in the student track of ICST 2008. This version currently uses only one global value for the probability of running a test, instead of one per method, however. That value is set by a central server, depending on the size of the “application community”.

In Spring 2008, we looked at various mechanisms for reducing the performance impact of Invite, e.g. by assigning tests to different cores/processors on multi-core/multi-processor machines, or by limiting the number of concurrent tests that may be run. We also looked at ways of balancing testing load across members of a community so that instances under light load pick up more of the testing. Lastly, we created a modified JDK that allows Invite to create copies of files so that in vivo tests do not alter the “real” file system.

In Fall 2008, we ported the Invite framework to C and evaluated more efficient mechanisms for injecting the instrumentation and executing the tests. We also investigated fault localization techniques, which collect data from failed program executions and attempt to discover what caused the failure.

Recently we have investigated ways to make the technique more efficient by only running tests in application states it hasn’t seen before. This cuts down on the number of redundant states that are tested, thus reducing the performance overhead. This work has potential application to domains like model checking and dynamic analysis and was presented in a workshop paper at AST 2010.

Currently we are looking at ways to apply the In Vivo approach to the domain of security testing. Specifically, we devised an approach called Configuration Fuzzing in which the In Vivo tests make slight changes to the application configuration and then check “security invariants” to see if there are any vulnerabilities that are configuration-related. This work was presented at the 2010 Workshop on Secure Software Engineering.

In 2012-2013, we are investigating techniques to efficiently isolate the state of the tests, so as to avoid the effect of the tests on external systems.

Open research questions include:

• Can the overhead be reduced by offloading test processes to other machines? This is especially important when the application is running on a single-core machine.
• What sorts of defects are most likely to be detected with such an approach? How can we objectively measure the approach’s effectiveness at detecting defects?
• How can the tests be “sandboxed” so that they do not affect external entities like databases? We currently assure that there are no changes to the in-process memory or to the file system, but what about external systems?

Interested students should view our project student ads.

Contact: Jon Bell (jbell@cs.columbia.edu)

Team Members

Faculty
Prof. Gail Kaiser, kaiser@cs.columbia.edu

Graduate Students
Jonathan Bell, jbell@cs.columbia.edu

Former Members
Matt Chu
Waseem Ilahi
Chris Murphy
Del Slane
Ian Vo
Moses Vaughan

Links

Publications
ICST 2009 paper on in vivo testing
ICST 2008 paper on distributed in vivo testing
SecSE 2010 paper on configuration fuzzing
AST 2010 on conducting tests only in previously-unseen states

Related Work
Gamma project at Georgia Tech
Skoll project at Univ of Maryland
Cooperative Bug Isolation at Univ of Wisconsin