Context

The background to the testing framework is discussed on the Testing Proposal page. One vision of what a full testing system might include is given in Spronk Contract 2010. Note that this proposal has not been adopted, and may not be.

Tasks

The task for the first 12-month period will be to set up full-system tests for the CcpNmr code and the Extend-NMR pipeline. This includes a framework, test data and results, and the actual tests. The tests will be used in the leading-edge development branch, as well as the stable branch. The tasks to test will be high-level program runs. Programs will be run using CCPN projects as input and output (where possible), and results will be compared to standard results. There will be no requirement for in-code test data, and the high level nature of the tests will make them relatively stable towards code changes.

• Tests must start with install-and-build, or download-and-patch, to catch errors at this stage.
  
• Tests should run nightly in batch mode, and send a highly condensed report with changes and problems only. It should be possible to see at a  glance if there are new problems that require investigation. A detailed report should be available to further identify problems. 
• It must be possible to re-run individual tests so that problem fixes can be evaluated quickly.
• The organisation, use, and maintenance of the test data and framework must be clearly documented so that they can be used and maintained by different parties.
  
• Tests should include:
• FormatConverter, starting and ending on non-CCPN data.
• Data upgrade and downgrade, possibly together with format conversion
  
• ECI entry completion and deposition
• ARIA, Haddock, CING
• Extend-NMR pipeline, where a CCPN project is passed to ARIA, then to CING, then to deposition.
• The tests will require large amounts of test data, which must be gathered and organised in a repository for easy and systematic access form several sites. Providing the test data would have to be the responsibility of CCPN, while organising it would be a SpronkNMR deliverable.
  
• The tests and data management framework must continue to work in the face of changes to data model and consequently CCPN data storage files.
• It must be possible to mark as 'unimportant' changes differences that are judged not to be cause for concern, without effectively neutering future tests on the specific case.   
• Comparing CCPN projects is a non-trivial problem, especially in the presence of data model changes. Possible solutions would be making a CCPN project comparison routine, comparing XML files directly, after modifying the XML storage to make files more 'diffable', and/or relying on export to external data formats. The road to take will need to be discussed, and some components, like a CCPN project comparator, would have to be written wholly or partially by CCPN.