Testing¶

Module “Unit” Tests¶

A comprehensive set of tests is provided in orchestrator/test/ to ensure code consistency during development. These module tests cover basic use cases for individual modules, aiming for robust code coverage of required methods. The Simulator and Oracle tests, among others, can be run independently, while the Trainer and Potential tests are run together.

Although a fully automated test suite is planned for the future, modules can currently be tested manually. To facilitate this, a setup script setup_tests.zsh is provided, which prepares the test directories for execution. Run the script as follows:

$ zsh setup_tests.zsh /path/where/tests_are_run SELECTION

The first argument specifies the directory where the tests will be copied and executed.
The second argument is either all (to run all tests) or the name of a specific module (e.g., descriptor, oracle, etc.).

Once set up, you can run individual tests by executing the associated driver.py file:

[.../test_path/oracle/]$ python driver.py

Tests must be run in an environment that includes all code dependencies (as well as the KIM API, if testing KIM functionality). See the Installation section for more details.

A run_all.zsh script is also provided to automatically run the full test suite (or the loaded subset). This script checks whether a given module has already been tested and skips rerunning completed tests.

Automatic checking with pytest is also available. See below for more information.

Trainer Test¶

The Trainer and Potential tests cover the generation and training of an IAP using a small set of nine structures. The trained potential is then saved.

Currently, tests are provided for the following implementations:

DUNNTrainer with KliffBPPotential
ParametricModelTrainer with KIMPotential
FitSnapTrainer with FitSnapPotential
Tests are included for both command-line training and job queue submission.
Integration tests are also provided for ensuring the KIM API and kimkit backend functionality is working as expected.

Oracle Test¶

The Oracle test uses three structures to compute ground-truth forces, energies, and stresses (except for KIMOracle, which does not compute stresses). Outputs are saved as extended XYZ files using ASE’s write function. Most tests also integrate with ColabfitStorage to test saving and extraction of configurations.

Tested modules:

Workflow Test¶

Several tests use different workflow implementations, such as:

Refer to the workflow block in the input files for usage examples.

Simulator Test¶

The Simulator test generates a molecular dynamics simulation using a randomly selected initial configuration. Output is parsed using the parse_for_storage() method, with excerpts saved for verification.

Tested modules:

LAMMPSSimulator

Target Property Test¶

The TargetProperty tests run copper MeltingPoint calculations at standard conditions. Example files are also provided for carbon (diamond) melting point calculations and copper at high pressure, using both regular and KIM potentials.

ElasticConstants tests are provided for Si and Ta, using different types of KIM potentials. This module currently supports only KIM potentials.

KIMRun has two tests for W, confirming correct predictions for some basic material properties using a KIM Simulator Model and KIM Tests found in KIMKit. The two tests invoke the Singularity container using two different workflows – LocalWF and SlurmWF. An additional two tests, both demonstrating the calculation of a cold curve for diamond Si using Stillinger-Weber using LocalWF, test the usage of KIMRun with Potential. KIMRun works with Potential objects whose save_potential_files() function saves an archive of a directory that is installable using the KIM API.

Note

The KIMRun test requires the following OpenKIM items and dependencies to be downloaded and added to KIMKit (separate from the KIM API). See the KIMRun documentation for instructions:

Descriptor Test¶

The DescriptorBase tests computes descriptors in both single and batched execution modes.

Tested modules:

KLIFFDescriptor module to compute ACSF descriptors
QUESTSDescriptor module to compute QUESTS descriptors

Score Test¶

The ScoreBase tests demonstrate the use of both AtomCenteredScore and DatasetScore modules to compute various score quantities. Tests include both single and batched calculations, as well as saving/accessing data with ColabfitStorage.

Tests cover:

Note

All FIM tests currently only support integration with KIMPotential.

Semi-Automated Checks with pytest¶

Curated outputs for unit tests are saved in the reference_output/ directory and can be automatically checked against new test runs using pytest with the provided test files in each module’s subdirectory. pytest compares new outputs to the reference set.

pytest is automatically run for any test completed via the driver.py file. You can also run pytest manually - it will search for any files that begin with test_ in the current directory and all subdirectories and run these tests.:

$ pytest -v

The -v flag increases verbosity, showing which tests are run. To run a specific test file simply specify it:

$ pytest test_simulator.py

Adding Tests¶

To add tests for new functionality or to increase coverage:

Check if existing unit tests apply to your use case. This will generally be the case if you are adding a new concrete module to a pre-existing module type. If not, write a new test and add it to the appropriate MODULE_unit_testers.py file.
Write the input file for your test and add it to the appropriate test_inputs directory. If additional inputs are needed, add them here as well. Input files should be machine-agnostic. If machine/user specific inputs are needed, they should be entered as <ABSTRACT_VALUES> which can be substituted by the setup_tests.zsh script.
Add adequate reference data to confirm successful test completion. Place this data in the appropriate reference_data/ subdirectory, using distinguishable filenames. Keep files small where possible. Shared reference data across tests is allowed.
Update the pytest test_MODULE.py file to check your test output against your reference data.
Update the relevant driver.py file to run your new test condition, specifying any dependencies.
Update setup_tests.zsh to properly “install” the test module on different machines. Replace any machine-specific input as needed based on architecture. Refer to current tests and the setup script for examples.
Run setup_tests.zsh to load your new tests and ensure they run and pass their pytests.