The mtimeit CLI tool may be used to run multivariate performance tests. This program wraps
rics.performance.run_multivariate_test().
Output when running mtimeit --help.
$ mtimeit --help
Usage: mtimeit [OPTIONS]
Run a multivariate performance test.
This is the https://pypi.org/project/rics/ version of the python timeit
module. It may be used to run performance tests evaluating one or more
candidate functions ('candidates.py') on one or more different kinds of
inputs ('test_data.py'). See below for details on these modules.
This script will:
0. Create 'candidates.py' and 'test_data.py' (iff --create is set)
1. Quickly evaluate each candidate on all test data "a few times".
2. Decide how many times to evaluate each candidate, such that the
--time-per-candidate argument is respected.
3. Print the best times per candidate/test_data
combination to stdout.
4. Save a performance overview figure to disk.
5. Save raw timing data to disk as csv.
Required files:
candidates.py - Members starting with 'candidate_' are used as candidates.
test_data.py - Members starting with 'case_' are used as the case case data.
Hint: Define a 'ALL'-attributes in 'candidates' and 'test_data' to declare
explicit members to use.
Options:
--time-per-candidate FLOAT Time in seconds to allocate to each candidate
function. [default: 2.0]
--name FILE Name to use for artifacts produced. Also used as
the figure title (stylized). [default:
performance.png]
--create / --no-create Create files 'candidates.py' and 'test_data.py'
and run a demo. Will not overwrite existing
files. [default: no-create]
--help Show this message and exit.
Output when running mtimeit --create. This flag may be used to initialize working dummy implementations of the
required candidates.py and test_data.py modules.
$ (mkdir /tmp/example && cd /tmp/example/ && (echo y | mtimeit --create))
========================= Begin Performance Evaluation =========================
| 'Create Example Run' |
--------------------------------------------------------------------------------
| Found 2 candidates and 2 data variants. |
| Started: 2022-11-26 16:58:46, ETA: Saturday 26, 16:58:47 |
================================================================================
2022-11-26T16:58:48.289 [rics.performance:INFO] Evaluate candidate 'do_nothing' 5x55164 times..
/home/docs/checkouts/readthedocs.org/user_builds/rics/envs/v1.0.0/lib/python3.10/site-packages/rics/performance/_multi_case_timer.py:83: UserWarning: The test results may be unreliable for ('do_nothing', 'small_array'). The worst time 1.8341e-05 sec was ~23.8 times slower than the best time (7.71e-07 sec).
warnings.warn(
/home/docs/checkouts/readthedocs.org/user_builds/rics/envs/v1.0.0/lib/python3.10/site-packages/rics/performance/_multi_case_timer.py:83: UserWarning: The test results may be unreliable for ('do_nothing', 'big_array'). The worst time 2.292e-05 sec was ~29.7 times slower than the best time (7.71e-07 sec).
warnings.warn(
2022-11-26T16:58:48.489 [rics.performance:INFO] Evaluate candidate 'do_something' 5x2 times..
================================================================================
| Best Times |
| 'Create Example Run' |
================================================================================
Time [s] Time [ms] Time [μs]
Candidate Test data
do_nothing big_array 1.39765e-11 1.39765e-08 1.39765e-05
small_array 1.39765e-11 1.39765e-08 1.39765e-05
do_something big_array 0.0206174 20.6174 20617.4
small_array 7.09e-07 0.000709 0.709
================================================================================
Figure saved: '/tmp/example/create-example-run.png'
WARNING: The full timing report has 110332 rows, which may take a while to serialize.
Really print full report to '/tmp/example/create-example-run.csv'? [y/N]: Data saved: '/tmp/example/create-example-run.csv'
$ (tree /tmp/example/ -L 1)
/tmp/example/
├── __pycache__
├── candidates.py
├── create-example-run.csv
├── create-example-run.png
└── test_data.py
1 directory, 4 files