rics timeit

Multivariate performance testing.

Synopsis

!rics timeit --help

Usage: rics timeit [OPTIONS]

  Multivariate performance testing.

  This is the https://pypi.org/project/rics/ version of the python timeit
  module. It may be used to run performance tests evaluating one or more
  candidate functions ('candidates.py') on one or more different kinds of
  inputs ('test_data.py').

  This script will:
      0. Create 'candidates.py' and 'test_data.py' (only if --create is set; will never overwrite).
      1. Quickly evaluate each candidate on all test data "a few times".
      2. Decide how many times to evaluate each candidate, such that the
         --time-per-candidate argument is respected.
      3. Print the best times per candidate/test_data
         combination to stdout.
      4. Save a performance overview figure to disk.
      5. Save raw timing data to disk as CSV.

  Required files:
      candidates.py - Members starting with 'candidate_' are used as candidates.
      test_data.py - Members starting with 'case_' are used as the case case data.

  Hint: Define a 'ALL'-attributes in 'candidates' and 'test_data' to declare
  explicit members to use.

Options:
  -t, --time-per-candidate FLOAT  Time in seconds to allocate to each
                                  candidate function.  [default: 2.0]
  -n, --name FILE                 Name to use for artifacts produced. Also
                                  used as the figure title (stylized).
                                  [default: performance.png]
  --create / --no-create          Create files 'candidates.py' and
                                  'test_data.py' and run a demo. Will not
                                  overwrite existing files.  [default: no-
                                  create]
  --per-candidate / --no-per-candidate
                                  Enable to print per-candidate best times.
                                  Just shows the best overall per data if
                                  disabled.  [default: per-candidate]
  -y, --yes                       Automatic yes to prompts for things like
                                  writing large (>100_000 rows) result files.
  --help                          Show this message and exit.

  The CLI does not support facets (rows and columns). Please use
      https://rics.readthedocs.io/en/stable/api/rics.performance.html
  if your test data should be categorized into subgroups.

Example run

Output when running mtimeit --create. This flag may be used to initialize working dummy implementations of the required candidates.py and test_data.py modules.

%%sh
mkdir /tmp/example
cd /tmp/example/
rics timeit --yes --create

========================= Begin Performance Evaluation =========================
|                             'Create Example Run'                             |
--------------------------------------------------------------------------------
| Found 2 candidates and 2 data variants:                                      |
|  -    Candidates: ['do_nothing', 'do_something']                             |
|  - Data variants: ['small_array', 'big_array']                               |
| Started: 2025-05-19 19:14:00, ETA: Monday 19, 19:14:00                       |
================================================================================
================================================================================
|                                  Best Times                                  |
|                             'Create Example Run'                             |
================================================================================
       Candidate  Run no    Time [s]    Test data   Time [ms]  Time [μs]   Time [ns]  Times min  Times mean
6     do_nothing       1 2.91255e-08    big_array 2.91255e-05  0.0291255     29.1255          1 1.76045e-05
4     do_nothing       4 2.91507e-08  small_array 2.91507e-05  0.0291507     29.1507          1    0.366268
14  do_something       4 6.72857e-08  small_array 6.72857e-05  0.0672857     67.2857     2.3082    0.845421
19  do_something       4  0.00294402    big_array     2.94402    2944.02 2.94402e+06     101081     1.77947
================================================================================
Figure saved: '/tmp/example/create-example-run.jpg'
Data saved: '/tmp/example/create-example-run.csv'

Generated files

Contents of /tmp/example

!tree /tmp/example/ -L 1

/tmp/example/
├── candidates.py
├── create-example-run.csv
├── create-example-run.jpg
├── __pycache__
└── test_data.py

2 directories, 4 files

The candidates.py file

This is the candidate functions file generated by the --create flag.

!pygmentize /tmp/example/candidates.py

"""Module defining candidate functions.

Any top-level members that start with `"candidate_"` will be automatically
imported. These are assumed to be callable. The candidates will be evaluated
for all data defined in ``test_data.py``.

Alternatively, you may define an "`ALL`" attribute of explicit members to use.
"""


def candidate_do_nothing(data):
    pass


def candidate_do_something(data):
    sum(data)


def candidate_ignored_since_not_in_all(data):
    pass


# Explicit members to use. Use a dict to specify names manually.
ALL = [
    candidate_do_nothing,
    candidate_do_something,
]

The test_data.py file

This is the data file generated by the --create flag.

!pygmentize /tmp/example/test_data.py

"""Module defining test data.

Any top-level members that start with `"data_"` will be automatically imported
and used on all candidates as defined by ``candidates.py``

Alternatively, you may define an "`ALL`" attribute of explicit members to use.
"""

data_small_array = [0]
data_big_array = list(range(10**6))
data_ignored_since_not_in_ALL = 0

# Explicit members to use.
ALL = {
    "small_array": data_small_array,
    "big_array": data_big_array,
}