The potential for using the Met Office User Test Framework(UTF) with NEMO. -------------------------------------------------------------------------- Draft: Date: 05/11/2010 Author: Richard Hill Introduction ------------ This document is intended to be an initial assessment of the Met Office UTF with regard to the possibilities of using it as a replacement for the NEMO NVTK. Initially, only the technical functionality and potential is considered. Other issues such as licensing and distribution will also need consideration however, it is believed that these are not likely to present major stumbling blocks. Background ---------- The UTF is developed in the Met Office, initially as a test framework for the Met Office Unified Model (UM) system, but extends or will extend to cover other systems (such as full coupled models involving NEMO and CICE). The UTF describes itself as follows: "The User Test Framework (UTF) is a joint effort between the UM System Development team and the CRUM team to create a way of testing of the UM simply and robustly. It is intended to fulfill two main purposes: * The UTF should enable people developing code for the UM to easily perform a selection of tests specified by the UM System Development Team on their branch prior to submitting it for code/system review. * The UTF should be a more robust and extensible, in terms of both jobs and tests performed on them, replacement for the current Test Harness. We see the second requirement as being a specific instance of the first (i.e. the test harness replacement will be a nightly run of the UTF started by cron). The UTF is written in Perl, using an object-orientated approach. It is capable of submitting UM extract-and-build..... and model run jobs to the IBM and monitoring the job's progress, then once the job has completed performing a number of pre-specified tests on the output (e.g. does standard output contain specific text such as 'failed' or 'succeeded', does the output dump compare with a pre-specified one..... etc)...... It is possible for a job to depend on the success of prior jobs, so for example a UM run job (or multiple UM run jobs) could depend on a previous UM build job. The OO nature means that creating and adding new jobs is a relatively simple task enabling easy future expansion of tests or even whole new standard jobs, while the use of 'status objects' for each action means that in the event of failure a helpful error message should be produced." NEMO Requirements ----------------- Any NEMO test system requires us to consider the following: 1) NEMO has lots of users 2) Any test system needs to be runnable on various machine architectures 3) We need to be able to test against reference configurations 4) We need to be able to add and change model configurations easily 5) We need to be able to add functionality for new machine achitectures easily 6) We need to be able to add validation tests easily Initial assessment ------------------ The UTF is stored and developed under FCM at the Met Office. It has already been sent to a few external users of the unified model for their use in testing the unified model Atmosphere code. Having familiarised myself with some of its capabilities with regard to the UM and discussed the technical background and possibilities with the UTF developers, it seems that the system is likely to be quite suitable for use by NEMO, with a little extra work. Specifically with regard to NEMO requirements 1) - 6) above, these are addressed as follows: 1) The UTF is designed to be used by a very large user base (e.g. all users of the Unified Model) - within and outside the Met Office. So relative ease of use and modularity has been an important consideration. At present, it is a relatively youg system, just starting to come on line at the Met Office, so we still have the opportunities to influence its development. 2) The UTF's object orientated approach means that it is expected to be easily extended to cover mose machine types and operating systems. (It will not work under MS Windows!) 3) The primary purpose of the UTF is to provide a facility to test model developments against reference configurations from standard runs. There is exepected to be a facility to be able to store the results of standard runs in the UTF repository. This will not be essential, but may prove useful if people want to avoid having to store and maintain the output from standard runs elsewhere. 4) The UTF assumes nothing about the model configurations which are to be run - i.e. unlike the NVTK, it does not contain specific code for a particular configuration. It simply builds and runs whatever it is told to do. The details of each configuration are provided via the contents of configuration files lie outside of the UTF. Hence changing model configurations requires no changes to the UTF code at all. 5) Adding functionality for new machines is expected to be straightforward. See item 2) above. This will probably have to be done by the users on each new platform the first time they try it (e.g. to set up the appropriate commands to interrogate batch queues, such as "llq" on IBM AIX, "qstat" on NEC, etc.) 6) Validation tests are definable by the user and may be different from run to run. Again there are no hard coded tests assumed. The user may use simple system level commands such as "cmp" to comapre two different files (e.g. one might imagine doing this for the solver.stat files). Or it is possible to devise new special tests which carry out operations to perform more complicated tests between data files or to carry out post processing operations (e.g. one might imagine combining all the partial NEMO restart files from an MPP run into a single global file before perfroming some comparison operation). Maintenance and support Long term maintenance and user support. Initially, one would probably expect any support requests from users (probably IPSL, Met Office and NOCS) to be directed to the UTF development team. However, once IPSL and other NEMO users are experienced in its use, user support is expected to be directed to the NEMO system team in the usual way. The UTF team would be keen to include any appropriate additional functionality in the base code, and would expect to be kept informed of any potential developments. In reality, the modular and object orientated nature of the UTF should mean that any changes required to the code will be small. Outstanding issues ------------------ The issues to be resolved are: 1) Specific NEMO functionality. Currently the UTF functionality manily covers the the UM. In order to allow it to work with NEMO, it will require further relatively small developments to the code. This will almost certainly have to be done by the UTF development team. The potential availability of staff and timescales for this still need to be clarified. 2) Portability. Although the UTF is designed to be portable, it will probably require some work in relation to 5) above when being set up on architectures which it has not previously been run on. This is expected to be done by the user at the point of use, but is not expected to be a major issue or a large amount of work. 3) Licensing. A strategy for releasing the UTF under a license which will make it freely available and usable in the NEMO community and which would allow IPSL to distribute it needs to be agreed by the Met Office. The hope would be to issue it under a license similar to that used for FCM which will allow people to modify things for their own needs. 4) Anything identified by NEMO Systems Team. Summary ------- It seems that the UTF is designed in such a way that incorporation of support for NEMO would be relatively straightforward and that once this were done, the modular nature of the system would provide the type of flexible system which a test NEMO system requires. This draft will be forwarded to the NEMO Systems Team for their comments and input.