mtimeit#
Multivariate performance testing from the command line.
Synopsis#
[1]:
!mtimeit --help
Usage: mtimeit [OPTIONS]
Run a multivariate performance test.
This is the https://pypi.org/project/rics/ version of the python timeit
module. It may be used to run performance tests evaluating one or more
candidate functions ('candidates.py') on one or more different kinds of
inputs ('test_data.py'). See below for details on these modules.
This script will:
0. Create 'candidates.py' and 'test_data.py' (iff --create is set)
1. Quickly evaluate each candidate on all test data "a few times".
2. Decide how many times to evaluate each candidate, such that the
--time-per-candidate argument is respected.
3. Print the best times per candidate/test_data
combination to stdout.
4. Save a performance overview figure to disk.
5. Save raw timing data to disk as CSV.
Required files:
candidates.py - Members starting with 'candidate_' are used as candidates.
test_data.py - Members starting with 'case_' are used as the case case data.
Hint: Define a 'ALL'-attributes in 'candidates' and 'test_data' to declare
explicit members to use.
Options:
--time-per-candidate FLOAT Time in seconds to allocate to each
candidate function. [default: 2.0]
--name FILE Name to use for artifacts produced. Also
used as the figure title (stylized).
[default: performance.png]
--create / --no-create Create files 'candidates.py' and
'test_data.py' and run a demo. Will not
overwrite existing files. [default: no-
create]
--per-candidate / --no-per-candidate
Enable to print per-candidate best times.
Just shows the best overall per data if
disabled. [default: per-candidate]
--help Show this message and exit.
Example run#
Output when running mtimeit --create. This flag may be used to initialize working dummy implementations of the required candidates.py and test_data.py modules.
[2]:
!mkdir /tmp/example
!(cd /tmp/example/ && (echo y | mtimeit --create))
========================= Begin Performance Evaluation =========================
| 'Create Example Run' |
--------------------------------------------------------------------------------
👻 Configured some stuff just the way I like it!
| Found 2 candidates and 2 data variants. |
| Started: 2024-02-03 14:44:56, ETA: Saturday 03, 14:44:57 |
================================================================================
2024-02-03T14:44:58.196 [rics.performance:INFO] Evaluate candidate 'do_nothing' 5x237328 times..
/home/jovyan/work/src/rics/performance/_multi_case_timer.py:83: UserWarning: The test results may be unreliable for ('do_nothing', 'small_array'). The worst time 22μs was ~126.9 times slower than the best time (175ns).
warnings.warn(
/home/jovyan/work/src/rics/performance/_multi_case_timer.py:83: UserWarning: The test results may be unreliable for ('do_nothing', 'big_array'). The worst time 33μs was ~189.3 times slower than the best time (176ns).
warnings.warn(
2024-02-03T14:44:58.485 [rics.performance:INFO] Evaluate candidate 'do_something' 5x7 times..
/home/jovyan/work/src/rics/performance/_multi_case_timer.py:83: UserWarning: The test results may be unreliable for ('do_something', 'small_array'). The worst time 2μs was ~4.6 times slower than the best time (338ns).
warnings.warn(
/home/jovyan/work/src/rics/performance/_util.py:104: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning.
best = data.groupby(["Test data", "Candidate"]).min().reset_index()
Figure(1600x700)
================================================================================
| Best Times |
| 'Create Example Run' |
================================================================================
Candidate Test data Run no Time [s] Time [ms] Time [μs] Times min Times mean
179621 do_nothing small_array 179621 7.37376e-13 7.37376e-10 7.37376e-07 1 0.195489
466672 do_nothing big_array 229344 7.4159e-13 7.4159e-10 7.4159e-07 1 1.27682e-05
474660 do_something small_array 4 4.82857e-08 4.82857e-05 0.0482857 65483.2 12801.3
474669 do_something big_array 6 0.00179516 1.79516 1795.16 2.42069e+09 30907.9
================================================================================
Figure saved: '/tmp/example/create-example-run.png'
WARNING: The full timing report has 474670 rows, which may take a while to serialize.
Really print full report to '/tmp/example/create-example-run.csv'? [y/N]: Data saved: '/tmp/example/create-example-run.csv'
Generated files#
Contents of /tmp/example
[3]:
!tree /tmp/example/ -L 1
/bin/bash: line 1: tree: command not found
candidates.py#
[4]:
!pygmentize /tmp/example/candidates.py
"""Module defining candidate functions.
Any top-level members that start with `"candidate_"` will be automatically
imported. These are assumed to be callable. The candidates will be evaluated
for all data defined in ``test_data.py``.
Alternatively, you may define an "`ALL`" attribute of explicit members to use.
"""
def candidate_do_nothing(data):
pass
def candidate_do_something(data):
sum(data)
def candidate_ignored_since_not_in_all(data):
pass
# Explicit members to use. Use a dict to specify names manually.
ALL = [
candidate_do_nothing,
candidate_do_something,
]
test_data.py#
[5]:
!pygmentize /tmp/example/test_data.py
"""Module defining test data.
Any top-level members that start with `"data_"` will be automatically imported
and used on all candidates as defined by ``candidates.py``
Alternatively, you may define an "`ALL`" attribute of explicit members to use.
"""
data_small_array = [0]
data_big_array = list(range(10**6))
data_ignored_since_not_in_ALL = 0
# Explicit members to use.
ALL = {
"small_array": data_small_array,
"big_array": data_big_array,
}
[ ]: