IN THIS ARTICLE
General Maintenance Information
How do I turn the TIAF on/off in Automated Review?
Method 1: Using the TIAF CMake Switches
This method will switch off TIAF at the CMake option level, leaving all tests currently opted-in to TIAF to be run by CTest. This method is the least intrusive and thus only if this method fails should the following method be used.
When using this method, the TIAF CMake function
o3de_test_impact_apply_test_labels will remove the label
REQUIRES_tiaf from any native or Python test target that has been disabled with the
O3DE_TEST_IMPACT_PYTHON_TEST_TARGETS_ENABLED CMake variables. As such, it requires no further modification at either the CMake or Jenkins level to gracefully pass the burden of test running back over to CTest.
- Navigate to
O3DE_TEST_IMPACT_NATIVE_TEST_TARGETS_ENABLED(for native C++ tests) and/or
O3DE_TEST_IMPACT_PYTHON_TEST_TARGETS_ENABLED(for Python tests) to
Method 2: Removing the TIAF Stages from AR
This method will remove TIAF completely from AR. It should be used as a method of last resort as it requires intrusive edits to the AR Jenkins configuration.
- Navigate to
profile_pipeand under steps remove
How do I enroll a test into TIAF?
All tests enrolled into the TIAF are run by the appropriate TIAF runtime (
native for native C++ tests,
python for Python tests) instead of CTest. Likewise, all tests not enrolled into the TIAF are run instead by CTest. In order to enroll a test target into TIAF, simply add the
REQUIRES_tiaf label when registering the test, like so:
ly_add_googletest( NAME AZ::MyExample.Tests LABELS REQUIRES_tiaf )
Caution:It is advised that you place your
LABELSafter the test target name and before any other custom attributes.
How do I enrol a native test for test sharding optimization?
Test sharding is an optional optimization that can greatly boost the speed at which test targets are run (see
NativeInstrumentedTestRunner for more information). As the question implies, the test sharding optimization is only available for native tests. In order to opt a test target into test sharding, simply add either
TIAF_shard_fixture label when registering the test, like so:
ly_add_googletest( NAME AZ::MyExample.Tests LABELS REQUIRES_tiaf;TIAF_shard_test )
When using the
TIAF_shard_test label, each individual test will be interleaved across the available shards. This delivers the greatest performance boost but increases the brittleness of the sharded test target as resource race conditions for badly behaving tests are more likely to manifest. On the contrary,
TIAF_shard_fixture interleaves each individual fixture across the available shards, delivering less performance than
TIAF_shard_test but less brittleness. Always test your targets when enrolling them into test sharding optimization to ensure that they behave well when running as shards.
Note:The test target must be enrolled into native TIAF in order to benefit from test sharding, else the test target will not be run by TIAF.
What are the command line options for the TIAF runtimes?
The following table details the command line options common to both the native and Python runtimes:
|Configuration File||-config=<filename>||<tiaf binay build dir>.<tiaf binary build type>.json||Path to the configuration file for the TIAF runtime.|
|Test Impact Data File||-testimpactdatafile=<filename>||None||Optional path to a test impact data file that will be used instead of that specified in the config file.|
|Previous Test Runs File||-previousrundatafile=<filename>||None||Optional path to a test impact data file that will be used instead of that specified in the config file.|
|Change List File||-changelist=<filename>||None||Path to the JSON of source file changes to perform test impact analysis on.|
|Global Timeout||-gtimeout=<seconds>||No timeout||Global timeout value to terminate the entire test sequence should it be exceeded.|
|Test Target Timeout||-ttimeout=<seconds>||No timeout||Timeout value to terminate individual test targets should it be exceeded.|
|Sequence Type||-sequence=<none, seed, regular, tia, tianowrite, tiaorseed>||None||The type of test sequence to perform, where:|
|Safe Mode||-safemode=<on,off>||Off||Flag to specify a safe mode for tia and tiaorseed sequences where the set of unselected tests is run without instrumentation after the set of selected instrumented tests is run (this has the effect of ensuring all tests are run regardless).|
|Draft Failing Tests||-draftfailingtests=<on,off>||Off||If enabled, will attempt to read the previous test runs data as specified in the config file and draft in any failing tests into tiaf and tiafnowrite sequences to be run in conjunction with the selected tests.|
|Shard Tests||-shard=<on,off>||No sharding||Break any test targets with a sharding policy into the number of shards according to the maximum concurrency value.|
|Capture Test Output||-targetout=<stdout, file>||None||Capture of individual test run stdout, where:|
|Failed Test Coverage Policy||-cpolicy=<discard, keep>||Keep||Policy for handling the coverage data of failing tests, where:|
|Execution Failure Policy||-epolicy=<abort, continue, ignore>||Continue||Policy for handling test execution failure (test targets could not be launched due to the binary not being built, incorrect paths, etc.), where:|
|Test Failure Policy||-fpolicy=<abort, continue>||Abort||Policy for handling test failures (test targets report failing tests), where:|
|Integrity Failure Policy||-ipolicy=<abort, continue>||Abort||Policy for handling coverage data integrity failures, where:|
|Test Prioritization Policy||-ppolicy=<none, locality>||None||Policy for prioritizing selected test targets, where:|
|Test Suites||-suites=<…>||None||The comma-separated test suites to select from for this test sequence.|
|Sequence Report File||-report=<filename>||None||Path to where the sequence report file will be written (if this option is not present, no report will be written).|
|Suite Label Excludes||-labelexcludes=<…>||None||The list of labels that will exclude any tests with any of these labels in their suite.|
The native runtime has the following additional options:
|Max Concurrency||-maxconcurrency=<number>||Max hardware concurrency||The maximum number of concurrent test targets/shards to be in flight at any given moment (a value of 0 signifies to use the architecture’s maximum hardware concurrency).|
What are the command line options for the TIAF AR scripts?
The following table details the command line options common to both the native and Python AR scripts:
|Runtime Type||runtime-type=<native, python>||Yes||None||The runtime TIAF should run tests for.|
|Runtime Sequence Override||sequence-override=<tianowrite, seed, tia, regular>||No||None||Manually override the sequence to run with the specified type.|
|Configuration File||config||Yes||<tiaf binay build dir>.<tiaf binary build type>.json||The path to the configuration file for the TIAF runtime.|
|Source Branch||src-branch||Yes||None||The branch that is being built.|
|Destination Branch||dst-branch||No||None||For PR builds, the destination branch to be merged to, otherwise empty.|
|Commit Hash||commit||Yes||None||The commit that is being built.|
|Build Number||build-number||Yes||None||The build number this run of TIAF corresponds to.|
|Test Suites||suites||Yes||None||The test suites to select test targets from.|
|Test Label Excludes||label-excludes||No||None||The CTest labels to exclude test targets from selection if matched.|
|Test Failure Policy||test-failure-policy||Yes||None||The test failure policy for regular and test impact sequences (ignored when seeding).|
|Safe Mode||safe-mode||No||None||Run impact analysis tests in safe mode (ignored when seeding).|
|Test Timeout||test-timeout||No||None||The maximum run time (in seconds) of any test target before being terminated.|
|Global Timeout||global-timeout||No||None||The maximum run time (in seconds) of the sequence before being terminated.|
|Test Target Exclusion File||exclude-file||No||None||The path to file containing tests to exclude from this run.|
|Test Target Output Routing||target-output=<stdout>||No||None||The test target std/error output routing (if not specified, no test target output will be routed to the console).|
|S3 Bucket Name||s3-bucket||No||None||The location of S3 bucket to use for persistent storage, otherwise local disk storage will be used.|
|S3 Bucket Name Top Level Directory||s3-top-level-dir||No||None||The top level directory to use in the S3 bucket.|
|MARS Index Prefix||mars-index-prefix||No||None||The index prefix to use for MARS, otherwise no data will be transmitted to MARS.|