2.4. Testing the UPP

2.4.1. Running UPP Regression Tests

To run the full regression test (RT) suite in preparation for opening a pull request (PR):

  1. Navigate to the local clone of your UPP fork containing the changes you would like to introduce, and run the included RT script within /ci

cd /path/to/UPP/ci
nohup ./rt.sh -a <my_account> -r $PWD/rundir -t $PWD/../ &

where my_account is the name of an account where you have permissions to run jobs. The terminal will print a message like:

nohup: ignoring input and appending output to ‘nohup.out’

The user can continue to issue commands in the Terminal while the RTs run in the background.

Note

The time it takes for tests to run is queue-dependent. RTs can take as little as half an hour to run, but on machines with long queue times, it can take several hours to complete the full set of tests.

  1. Check rt.log.<machine>/nohup.out for a short summary of any changes in results. The tests are finished when there are 16 timestamps and a final results summary (e.g., “No changes in test results detected.”).

    • The /work directory generated in UPP/ci contains out.post.<test_name> files, which list output from each test, including any unexpected errors during runtime.

    • The /rundir directory generated within UPP/ci will include test case results, and .diff files located within each test’s directory will outline changes in fields with the current baselines.

    • Confirm expected changes within the run directory .diff files if any are present.

      • Changes in the rap_pe_test case only consisting of field 708 Convective Cloud Layer may be ignored; this is a known bug and will always be present within the WRFPRS.diff file.

2.4.2. Additional Configuration

For repeated regression test runs, users can edit the rt.sh file and disable the specified test cases by changing their respective values to “no.” Users can disable the build step as well with the same value for the build variable above the tests. Please be sure to enable all test cases and build settings and conduct a full RT run in preparation for a pull request so that code managers (CMs) can confirm all changes in results are expected and consistent with the developer’s results.

rt.sh will allow for changing the configuration of the regression tests if users desire to do so with the following available options:

  • w – specify the work directory for test case job output

  • r – specify the run directory containing baselines and .diff files for comparison of changes in results

The following are legacy options for when rt.sh was not included within the UPP repository and may be ignored by developers: -b, -u, -c, -t.