Multivariate performance testing from the command line.
$ mtimeit --help
Usage: mtimeit [OPTIONS]
Run a multivariate performance test.
This is the https://pypi.org/project/rics/ version of the python timeit
module. It may be used to run performance tests evaluating one or more
candidate functions ('candidates.py') on one or more different kinds of
inputs ('test_data.py'). See below for details on these modules.
This script will:
0. Create 'candidates.py' and 'test_data.py' (iff --create is set)
1. Quickly evaluate each candidate on all test data "a few times".
2. Decide how many times to evaluate each candidate, such that the
--time-per-candidate argument is respected.
3. Print the best times per candidate/test_data
combination to stdout.
4. Save a performance overview figure to disk.
5. Save raw timing data to disk as csv.
Required files:
candidates.py - Members starting with 'candidate_' are used as candidates.
test_data.py - Members starting with 'case_' are used as the case case data.
Hint: Define a 'ALL'-attributes in 'candidates' and 'test_data' to declare
explicit members to use.
Options:
--time-per-candidate FLOAT Time in seconds to allocate to each candidate
function. [default: 2.0]
--name FILE Name to use for artifacts produced. Also used as
the figure title (stylized). [default:
performance.png]
--create / --no-create Create files 'candidates.py' and 'test_data.py'
and run a demo. Will not overwrite existing
files. [default: no-create]
--help Show this message and exit.
Output when running mtimeit --create. This flag may be used to initialize working dummy implementations of the
required candidates.py and test_data.py modules.
$ (mkdir /tmp/example && cd /tmp/example/ && (echo y | mtimeit --create))
========================= Begin Performance Evaluation =========================
| 'Create Example Run' |
--------------------------------------------------------------------------------
| Found 2 candidates and 2 data variants. |
| Started: 2022-11-30 21:35:55, ETA: Wednesday 30, 21:35:55 |
================================================================================
2022-11-30T21:35:56.557 [rics.performance:INFO] Evaluate candidate 'do_nothing' 5x54381 times..
/home/docs/checkouts/readthedocs.org/user_builds/rics/envs/v2.0.0/lib/python3.10/site-packages/rics/performance/_multi_case_timer.py:83: UserWarning: The test results may be unreliable for ('do_nothing', 'small_array'). The worst time 1.1659e-05 sec was ~14.6 times slower than the best time (8.01e-07 sec).
warnings.warn(
/home/docs/checkouts/readthedocs.org/user_builds/rics/envs/v2.0.0/lib/python3.10/site-packages/rics/performance/_multi_case_timer.py:83: UserWarning: The test results may be unreliable for ('do_nothing', 'big_array'). The worst time 1.3158e-05 sec was ~16.3 times slower than the best time (8.09e-07 sec).
warnings.warn(
2022-11-30T21:35:56.761 [rics.performance:INFO] Evaluate candidate 'do_something' 5x2 times..
================================================================================
| Best Times |
| 'Create Example Run' |
================================================================================
Time [s] Time [ms] Time [μs]
Candidate Test data
do_nothing big_array 1.48765e-11 1.48765e-08 1.48765e-05
small_array 1.47294e-11 1.47294e-08 1.47294e-05
do_something big_array 0.0201347 20.1347 20134.7
small_array 7.14e-07 0.000714 0.714
================================================================================
Figure saved: '/tmp/example/create-example-run.png'
WARNING: The full timing report has 108766 rows, which may take a while to serialize.
Really print full report to '/tmp/example/create-example-run.csv'? [y/N]: Data saved: '/tmp/example/create-example-run.csv'
$ (tree /tmp/example/ -L 1)
/tmp/example/
├── __pycache__
├── candidates.py
├── create-example-run.csv
├── create-example-run.png
└── test_data.py
1 directory, 4 files