Testing process description

Algorithm

1. Environment is set up (environment variables, common files).

2. Solution is placed in the filesystem.

3. Precompile checkers are run in order of declaration.

   If at least one fails, solution scores 0 points.

4. Solution is compiled.

   If it compiles for too long, it scores 0 points with CE.

   If it fails to compile, it scores 0 points with CE.

5. For each test group,

   1. If one of dependencies failed, do not run.

   2. For each test case,

      1. Place specified in test case files

      2. Run solution

         If a solution has been running for too long, test case ends with TL.

         If a solution has been taking too much RAM, test case ends with ML.

      3. Retrieve required output files

         If a required file is not present, test case ends with PE.

      4. Run all validators.

         If at least one fails, test case ends with WA.

         If all pass, test case ends with OK.

      5. Cleanup files

   3. Determine how many points to score based on defined scoring policy for group.

6. Determine final score (sum of all group scores) and verdict (first non-OK verdict).

Used definitions

Test suite

A collection of test groups, precompile checkers, and environment settings that is used to test and estimate code task solutions.

Precompile checker

A declaration of how to check a code task solution source code before compiling and running.

Test group

A declared collection of test cases, test group dependencies, a fixed maximum points value and a scoring policy that describe how to test and estimate code task solutions.

Test group dependency

Another test group in the same test suite which is required to be in state where all its test cases pass in order to run this, dependent group.

Scoring policy

Describes how many points from maximum to score based on how successful test runs were.

See also: Test suite declaration

Test case

A collection of validators and settings that describe environment of a single test code task solution run: what data to run with and how to check for right answer.

Test case validator

A declaration of how to check results of running a code task solution.

Verdict

A short code telling status of a submission.

See also: List of verdicts

Points

An integer number, estimates how good a solution is and/or how did it pass code task test cases.