Regression testing

From Jacobson Lab Wiki

Jump to: navigation, search
  1. The daily build and regression testing only occurs if there have been changes in the CVS repository.
  2. Tests are performed for each .test file in the plop/scripts/regresstest/regression_tests directory, all input files required by these .test files must be placed in the regresstest/input_files directory.
  3. .test files don't really need to contain anything special, it should know where to look for your input files, though currently the only command which is defined as an input command is load.
  4. The real power is in user defined tests. You can define a test to compare a value after change in the code. These tests are defined by 4 things:
    1. A plain english name to make things easy for us to understand.
    2. A file extension denoting which files should be compared, e.g. should this test be run on .rmsd files or .log files.
    3. A threshold - values which differ by more than this amount will be considered a failure.
    4. A set of regular expressions telling it how to find the values to compare in the file.
      1. Most perl regular expressions are supported, / must be used inside regular expressions, i.e. no s#a#b# syntax.
      2. Multiple regular expressions must be separated by ";" (and possible whitespace which will be ignored).
      3. A special "n=#" expression, which will capture the nth, whitespace separated word, i.e. if you want to compare the third column, n=3 would be sufficient.
      4. Unfortunately capturing parens and "$" substitution is not supported (there are presently no plans to change this).
      5. We recommend having a matching expression first if possible.
  5. A TOTALE test will be generated automatically. It compares final energies. So it is not necessary for the user to add their own TOTALE E test. (To do this, an "energy calc" command is added to your con file).
  6. If all tests pass the minor version number is incremented.
  7. In order to increment the major version number just touch your_plop/major_version_flag_file, then add the file to cvs, finally run cvs commit. This file will be removed by the regression test that day, the major version incremented, and the minor version reset.
  8. If a user wishes to reset a test, you should move all the testname.* files to a subfolder in /plop_build/archive this will cause the regression testing to treat the test as a new test.
Personal tools