Multivariate performance testing from the command line.
[1]:
!mtimeit --help
Usage: mtimeit [OPTIONS]
Run a multivariate performance test.
This is the https://pypi.org/project/rics/ version of the python timeit
module. It may be used to run performance tests evaluating one or more
candidate functions ('candidates.py') on one or more different kinds of
inputs ('test_data.py'). See below for details on these modules.
This script will:
0. Create 'candidates.py' and 'test_data.py' (iff --create is set)
1. Quickly evaluate each candidate on all test data "a few times".
2. Decide how many times to evaluate each candidate, such that the
--time-per-candidate argument is respected.
3. Print the best times per candidate/test_data
combination to stdout.
4. Save a performance overview figure to disk.
5. Save raw timing data to disk as CSV.
Required files:
candidates.py - Members starting with 'candidate_' are used as candidates.
test_data.py - Members starting with 'case_' are used as the case case data.
Hint: Define a 'ALL'-attributes in 'candidates' and 'test_data' to declare
explicit members to use.
Options:
--time-per-candidate FLOAT Time in seconds to allocate to each
candidate function. [default: 2.0]
--name FILE Name to use for artifacts produced. Also
used as the figure title (stylized).
[default: performance.png]
--create / --no-create Create files 'candidates.py' and
'test_data.py' and run a demo. Will not
overwrite existing files. [default: no-
create]
--per-candidate / --no-per-candidate
Enable to print per-candidate best times.
Just shows the best overall per data if
disabled. [default: per-candidate]
--help Show this message and exit.
Output when running mtimeit --create. This flag may be used to initialize working dummy implementations of the required candidates.py and test_data.py modules.
[2]:
!mkdir /tmp/example
!(cd /tmp/example/ && (echo y | mtimeit --create))
========================= Begin Performance Evaluation =========================
| 'Create Example Run' |
--------------------------------------------------------------------------------
| Found 2 candidates and 2 data variants. |
| Started: 2022-12-10 19:31:05, ETA: Saturday 10, 19:31:05 |
================================================================================
2022-12-10T19:31:05.877 [rics.performance:INFO] Evaluate candidate 'do_nothing' 5x193003 times..
/home/dev/git/rics/src/rics/performance/_multi_case_timer.py:83: UserWarning: The test results may be unreliable for ('do_nothing', 'small_array'). The worst time 8.4125e-05 sec was ~389.5 times slower than the best time (2.16001e-07 sec).
warnings.warn(
/home/dev/git/rics/src/rics/performance/_multi_case_timer.py:83: UserWarning: The test results may be unreliable for ('do_nothing', 'big_array'). The worst time 3.1537e-05 sec was ~146.0 times slower than the best time (2.15998e-07 sec).
warnings.warn(
2022-12-10T19:31:06.093 [rics.performance:INFO] Evaluate candidate 'do_something' 5x8 times..
/home/dev/git/rics/src/rics/performance/_multi_case_timer.py:83: UserWarning: The test results may be unreliable for ('do_something', 'small_array'). The worst time 1.743e-06 sec was ~4.4 times slower than the best time (3.97999e-07 sec).
warnings.warn(
Figure(1600x700)
================================================================================
| Best Times |
| 'Create Example Run' |
================================================================================
Candidate Test data Run no Time [s] Time [ms] Time [μs] Times min Times mean
368577 do_nothing big_array 175574 1.11914e-12 1.11914e-09 1.11914e-06 1 1.63327e-05
175715 do_nothing small_array 175715 1.11916e-12 1.11916e-09 1.11916e-06 1 0.258281
386013 do_something small_array 7 4.97498e-08 4.97498e-05 0.0497498 44452.8 11481.3
386016 do_something big_array 2 0.00141416 1.41416 1414.16 1.26361e+09 20638.1
================================================================================
Figure saved: '/tmp/example/create-example-run.png'
WARNING: The full timing report has 386022 rows, which may take a while to serialize.
Really print full report to '/tmp/example/create-example-run.csv'? [y/N]: Data saved: '/tmp/example/create-example-run.csv'
Contents of /tmp/example
[3]:
!tree /tmp/example/ -L 1
/tmp/example/
├── candidates.py
├── create-example-run.csv
├── create-example-run.png
├── __pycache__
└── test_data.py
1 directory, 4 files
candidates.py#[4]:
!pygmentize /tmp/example/candidates.py
"""Module defining candidate functions.
Any top-level members that start with `"candidate_"` will be automatically
imported. These are assumed to be callable. The candidates will be evaluated
for all data defined in ``test_data.py``.
Alternatively, you may define an "`ALL`" attribute of explicit members to use.
"""
def candidate_do_nothing(data):
pass
def candidate_do_something(data):
sum(data)
def candidate_ignored_since_not_in_all(data):
pass
# Explicit members to use. Use a dict to specify names manually.
ALL = [
candidate_do_nothing,
candidate_do_something,
]
test_data.py#[5]:
!pygmentize /tmp/example/test_data.py
"""Module defining test data.
Any top-level members that start with `"data_"` will be automatically imported
and used on all candidates as defined by ``candidates.py``
Alternatively, you may define an "`ALL`" attribute of explicit members to use.
"""
data_small_array = [0]
data_big_array = list(range(10**6))
data_ignored_since_not_in_ALL = 0
# Explicit members to use.
ALL = {
"small_array": data_small_array,
"big_array": data_big_array,
}
[ ]: