Benchmark configuration

Configuring a benchmark can be quite extensive, as this framework focuses on flexibility. For this, the documentation will be divided in main sections.

The benchmark configuration file describes precisely how the benchmarking should be done, for example, specifying where the executable is located, the options to pass to the application, and how the tests will be parametrized.

The base of the configuration file is shown below.

{
    "executable": "",
    "use_case_name": "",
    "timeout":"",
    "env_variables":{},
    "options": [],
    "resources":{},
    "platforms":{},
    "additional_files":{},
    "scalability":{},
    "sanity":{},
    "parameters":{},
}

Users can add any field used for refactoring. For example, one can do the following.

"output_directory":"/data/outputs" // This is a custom field
"options":["--output {{output_directory}}"]

1. Fields on JSON root

executable [str]

Path to the application executable or command to execute.

use_case_name [str]

Custom name given to the use case. Serves as an ID of the use case, must be unique across use cases.

timeout [str]

Job execution timeout, starting when the job is launched (and not pending). Format: days-hours:minutes:seconds This field is notably important so that HPC resources are not wasted unnecessarily. For example, for a 10 minutes timer: 0-00:10:00

env_variables [Dict[str,str]]

key:value pairs for benchmark related environment variables. These variables will be set after the init phase of ReFrame tests.

options [List[str]]

List of application’s options. Input arguments can be parameterized in here. For example, [ "--number-of-elements={{parameters.elements.value}}", "--number-of-points={{parameters.points.value}}", "--verbose" ]

2. Resources

The resources field is used for specifying the computing resources that each test will use. Users can specify a combination of tasks, tasks_per_node, gpus_per_node, nodes, memory and exclusive_access. However, only certain combinations are supported, and at least one must be provided. The resource fields are meant to be parameterized, so that the application scaling can be analyzed, but this is completely optional.

At the moment, multithreading is not supported. The number of tasks per cpu is set to 1.
tasks [int]

Total number of tasks to launch the test with.

tasks_per_node [int]

Number of tasks per node to use. Must be specified along nodes OR tasks. If this number cannot be greater than ReFrame’s systems.partitions.processor.num_cpus configuration value.

nodes [int]

Number of nodes to launch the tests on.

gpus_per_node [int] (Optional)

Number of GPUs per node to use. Defaults to None. The test will not be launched on any gpu.

memory [int] (Optional)

Total memory used by the test, in Gb. If this field is provided, the number of tasks per node and number of nodes will be recalculated so that the application will have enough memory to run.

If using custom ReFrame configuration files, users must ensure that the `extras.memory_per_node` field is present on the ReFrame configuration file.
exclusive_access [bool] (Optional)

If true, the scheduler will reserve the totality of the nodes where tests will run. Defaults to true.

Valid combinations are the following: - tasks and tasks_per_node - nodes and tasks_per_node - Only tasks

Other fields can be specified along these combinations as needed.

2.1. Examples

2.1.1. Non-parameterized resources field

  • Tasks and tasks per node

"resources":{
    "tasks": 256,
    "tasks_per_node":128,
    "exclussive_access":true
}

This configuration will run ALL tests on 2 nodes (reserved exlusively), using 128 task per node. If systems.partitions.processor.num_cpus configuration field is inferior to 128, an error will be raised before submitting the test job.

  • Memory

Concerning memory, let’s suppose that our system has 256Gb of RAM per node, and that our application requires a total of 1000Gb of memory.

"resources":{
    "tasks": 256,
    "tasks_per_node":128,
    "exclussive_access":true,
    "memory":1000
}

This means that in order for the application to run, it needs at least 4 nodes. As we specified that we want to run on 256 tasks, and we need at least 4 nodes, the number of tasks per node cannot be greater than 64. The final number of tasks per node will be recomputed as the minimum between the requested number of tasks per node, and 64. In this case, all tests will run on 4 nodes, using 64 tasks per node.

2.1.2. Parameterized resources field

Suppose that the following parameters are defined:

"parameters":[
    {
        "name":"resources",
        "sequence":[
            { "tasks":128, "tasks_per_node":32 },
            { "tasks":128, "tasks_per_node":64 },
            { "tasks":128, "tasks_per_node":128 },
            { "tasks":256, "tasks_per_node":128 }
        ]
    }
]

We would need to define the resources field like this:

"resources":{
    "tasks":"{{parameters.resources.tasks.value}}",
    "tasks_per_node":"{{parameters.resources.tasks_per_node.value}}",
}

This configuration will execute one test for each one of the combinations below: - 4 nodes, 32 tasks per node (total of 128 tasks) - 2 nodes, 64 tasks per node (total of 128 tasks) - 1 node, 128 tasks per node (total of 128 tasks) - 2 nodes, 128 tasks per node (total of 256 tasks)

3. Platforms

The platforms object lists all options and directories related to the benchmark execution for each supported platform. A platform present on this object does not imply that it will be benchmarked, but it rather lists all possible options. The actual platform where tests will run is defined by either the access field of the machine configuration, or the platform field.

input_dir [str]

Indicates the directory path where input files can be found, INSIDE the given platform. For the built-in platform, it corresponds to where input files can be found on the system.

append_app_options [List[str]] (Optional)

Describes the options to pass to the application. It is equivalent to the options field on the configuration root. However, it is used for having different application options depending on the platform.

options [List[str]] (Optional)

Describes the options to pass to the platform launcher. For example, "options":["--bind a:b"] will be interpreted, for the apptainer platform, as apptainer exec --bind a:b your_image.sif your_application.exe …​

image [Dict[str,str]] (Conditional)

Contains information related to the container image. For any platform other than built-in, the image field must be specified

-filepath [str]

Filepath containing the location of the container image. If provided along the url field, the image will be pulled and placed here before tests are executed, otherwise the framework will assume that the image exists on the given filepath.

-url [str] (Optional)

URL to pull the image from. If this field is specified, feelpp.benchmarking will pull the image and place it under the filepath field. If this field is not provided, the framework will assume that the image exists under filepath.

The platforms field is optional, if not provided, the builtin platform will be considered. The syntax for builtin platform is the following:

"platforms": {
    "builtin":{
        "input_dir":"",
        "append_app_options":[]
    }
}

The following shows an example how to configure the Apptainer platform:

"platforms":{
    "apptainer":{
        "input_dir":"/input_data/",
        "options":["--bind /data/custom_data/:{{platforms.apptainer.input_dir}}"],
        "append_app_options":["--my_custom_option_for_apptainer"],
        "image":{
            "filepath":"/data/images/my_image.sif",
            "url":"oras://ghcr.io/your-image.sif"
        }
    }
}

In this case, input_dir represents the directory where input files will be found INSIDE the container. If there is no input data present in the container, you might need to bind a local input data directory to it, using options

The options field contains a list of all the options to include on the container execution. It is equivalent to the machine’s containers.apptainer.options field. However, users should only include benchmark dependent options in this list.

Note that the placeholder syntax is used to tell to the container to bind a directory to the one specified in ìnput_dir.

The append_app_options lists all the options to add to the application execution. It does the same as the options field in the root of the file, but can be used for case handling.

The image field indicates that the image should be pulled from oras://ghcr.io/your-image.sif and placed in /data/images/my_image.sif.

To summarize, feelpp.benchmarking will first execute (even before ReFrame is launched):

apptainer pull -F /data/images/my_image.sif oras://ghcr.io/your-image.sif

And then, ReFrame will submit a job executing (for local scheduler):

apptainer exec --bind /data/custom_data/:/input_data/ /data/images/my_image.sif --my_custom_option_for_apptainer

For the filepath field, it is very useful to make use of the {{machine.containers.apptainer.image_base_dir}} field from the machine configuration.

4. Scalability

Lists all the files where performance variables can be found.

directory [str]

Common directory where files containing performance variables can be found.

clean_directory [bool] (Optional)

If true, it will delete the contents of inside directory. Defaults to false.

stages [List[Stage]]

Describes the files containing performance variables, and how to extract them.

-name [str]

Name to describe the stage. It is used as prefix to add to the performance variables found in the file. If no prefix is needed, the name can be "".

-filepath [str]

Relative filepath of the file containing performance variables, relative to the directory field.

-format [str]

Format of the stage file. Supported values are "csv" and "json".

-units [Dict[str,str]] (Optional)

Custom units for certain performance variables. key:value pairs correspond to performance-variable:unit. For example, "my-variable":"m/s" implies that the variable will have "my-variable" has "m/s" as unit. By default, all columns have "s" as unit. To change the default behavior, users need to pass the "*":"custom-unit" key:value pair. This will associate the "custom-unit" to ALL performance variables inside the file, excepting other units specified inside this object.

-variables_path [str, List[str]]

Only valid if format is "json". Defines where, in the JSON hierrarchy, performance variables will be found. Supports the use of one or multiple wildcards (*).

custom_variables [List[Dict[str,str]]] (Optional)

Contains a list of objects describing custom performance variables to create, based on extracted ones (from stages). An aggregation will be performed using provided columns and valid operations. For more information, see the advanced Configuration

-name [str]

The name to give to the custom performance variable.

-columns [List[str]]

List of columns to aggregate, accepts both variables existing in the performance files, as well as other custom variables.

-op [str]

The aggregation operation to apply to the performance columns to create the custom one. Valid operations are "sum","min","max","mean".

-unit [str]

The unit to assign to the created performance variable.

Recursive creation of custom_variables is supported!

Deeply nested and complex JSON scalability files are supported, using multiple wildcard syntax!

4.1. Examples

Let’s assume our application exports the following files:

The {{instance}} keyword on the export implies that each test exports this files on its own directory, using the test’s hashcode.

  • /data/outputs/{{instance}}/exports.csv

a,b,c
1,2,3

  • /data/outputs/{{instance}}/logs/timers.json

{
    "function1":{
        "constructor":1.0,
        "init":0.1,
    },
    "function2":{
        "constructor":1.0,
        "init":0.1,
    },
    "execution":{
        "step1":0.5,
        "step2":0.7,
        "step3":1.0,
    }
}

An example of the scalability field to extract values on these files is found and explained below.

"scalability": {
    "directory": "/data/outputs/{{instance}}/",
    "stages": [
        {
            "name":"myExports",
            "filepath": "export.csv",
            "format": "csv",
            "units":{ "*":"meters", "a":"kg" }
        },
        {
            "name":"myTimers",
            "filepath": "logs/timers.json",
            "format": "json",
            "variables_path":"*"
        }
    ]
}

The common directory path of these exports is /data/outputs/{{instance}}.

Let’s analyse the first stage:

{
    "name":"myExports",
    "filepath": "export.csv",
    "format": "csv",
    "units":{ "*":"meters", "a":"kg" }
}

The name myExports means that performance variables from this file will appear in the exported report (and available for plotting) as myExports_a:1, myExports_b:2, myExports_c:3.

Concerning the units, "*":"meters" means that all of the variables in this CSV should have the "meters" unit. However, by specifying "a":"kg" we indicates that all columns should be "meters", except a who should have "kg" as unit.

Let’s now consider the second stage:

{
    "name":"myTimers",
    "filepath": "logs/timers.json",
    "format": "json",
    "variables_path":"*"
}

Performance variables on this file will be prefixed by "myTimers_".

As the units field is not specified, all variables will have the default ('s') unit.

Having only * as variables_path, means that all variables should be exported into the performance report. Variables will be exported as follows:

  • myTimers_function1.constructor : 1.0

  • myTimers_function1.init : 0.1

  • myTimers_function2.constructor : 1.0

  • myTimers_function2.init : 0.1

  • myTimers_execution.step1 : 0.5

  • myTimers_execution.step2 : 0.7

  • myTimers_execution.step3 : 1.0

4.1.1. Filtering with variables_path

  • "variables_path":"function1.*":

Exported performance variables:

  • myTimers_constructor : 1.0

  • myTimers_init : 0.1

Using the wildcards removes the part of the json that is not variable.

  • "variables_path":"exectution.step1"

Exported performance variables:

  • myTimers_step1 : 0.5

If a full path is passed, the variable name corresponds to the key of the leaf element of the JSON.

variables_path can be a list.

5. Additional Files

This field allows the export of custom files, as well as use case descriptions and logs. It is optional.

description_filepath [str] (Optional)

Filepath of a file to be used as benchmark description. This file will be copied to the same directory of the performance report, named description.adoc, after all ReFrame tests are completed (successfully or not). This file will automatically appear in the website for the current report, treated as an Antora partial.

parameterized_descriptions_filepath [str] (Optional)

Parameterized filepath (using either {{instance}} or {{parameters.MY_PARAM.value}}) of a file generated by a test. This file will be copied right after a test finishes (only if successful). The copied file will be located in a partials directory, under the current exported report directory. It will be named after the tests hashcode. These files will automatically appear in the website, inside the parameter table of the current report, and are treated as Antora partials.

custom_logs [List[str]] (Optional)

List of parameter depdendent filepaths of logs to be included in the website.

Currently, it is imposed for description and parameterized_description files to be in AsciiDoc format.

5.1. Examples

Let’s suppose that our benchmarked application produces an information.adoc file each time it runs. And that we have a use case description named description.adoc. Also, let’s assume that our app logs some information under logs/log.INFO and under logs/log.WARNING each time it runs.

If we have set our application to write its outputs under /data/outputs/{{instance}}. Then our additional_files field will look like this.

"additional_files":{
    "description_filepath":"/data/outputs/description.adoc",
    "parameterized_descriptions_filepath":"/data/outputs/{{instance}}/information.adoc",
    "custom_logs":[
        "/data/outputs/{{instance}}/logs/log.INFO",
        "/data/outputs/{{instance}}/logs/log.WARNING"
    ]
}

All these files will automatically appear on the website’s report page.

6. Sanity

The sanity field is used to validate the application execution.

The syntax is the following:

"sanity":{
    "success":[],
    "error":[]
}

  • The success field contains a list of patterns to look for in the standard output. If any of the patterns are not found, the test will fail.

  • The error field contains a list of patters that will make the test fail if found in the standard output. If any of these paterns are found, the test will fail.

At the moment, only validating standard output is supported. It will soon be possible to specify custom log files.

6.1. Examples

"sanity": {
    "success": ["[SUCCESS]"],
    "error": ["[OOPSIE]","Error"]
}

Will check if "[SUCCESS]" is found on the application’s standard output. If not, the test will failt.

It will also check that neither "[OOPSIE]" nor "Error" appear in the standard output.

Regex patterns are supported.

7. Parameters

The parameters field list all parameters to be used in the test. The cartesian product of the elements in this list will determine the benchmarks to be executed.

Parameters are accessible across the whole configuration file by using the syntax {{parameters.my_parameter.value}}.

Each parameter is described by a name and a generator.

Valid generators are :

  • linspace:

{
    "name": "my_linspace_generator",
    "linspace":{
        "min":2,
        "max":10,
        "n_steps":5
    }
}

The example will yield [2,4,6,8,10]. Min and max are inclusive.

  • geomspace:

{
    "name": "my_geomspace_generator",
    "geomspace":{
        "min":1,
        "max":10,
        "n_steps":4
    }
}

The example will yield [2,16,128,1024]. Min and max are inclusive.

  • range:

{
    "name": "my_range_generator",
    "range":{
        "min":1,
        "max":5,
        "step":1
    }
}

The example will yield [1,2,3,4,5]. Min and max are inclusive.

  • geometric:

{
    "name": "my_geometric_generator",
    "geometric":{
        "start":1,
        "ratio":2,
        "n_steps":5
    }
}

The example will yield [1,2,4,8,16].

  • repeat:

{
    "name": "my_repeat_generator",
    "repeat":{
        "value":"a repeated value",
        "count":3
    }
}

The example will yield ["a repeated value", "a repeated value", "a repeated value"].

  • sequence:

{
    "name": "my_sequence_generator",
    "sequence":[ 1, 2, 3, 4]
}

Sequence is the simplest generator. It will yield exactly the given list. It accepts dictionnaries as items, which can then be accessed via the . separator.

  • zip and subparameters:

Parameters can contain subparameters, which can be accessed recursively via the . separator. Its objective is to have parameters that depend on eachother, without producing a cartesian product. Aditionnaly, parameters can be zipped together via the zip generator. The zip generator takes a list of parameters to produce a list of python dictionaries. Each param inside the list can then have any desired generator from above.

{
    "name": "my_zip_generator",
    "zip":[
        {
            "name":"param1",
            "sequence":[
                {"val1":1,"val2":2},
                {"val1":3,"val2":4},
                {"val1":5,"val2":6}
            ]
        },
        {
            "name":"param2",
            "repeat":{
                "value":"a repeated value",
                "count":3
            }
        }
    ]
}

This example will yield [{'param1': {'val1': 1, 'val2': 2}, 'param2': 'a repeated value'}, {'param1': {'val1': 3, 'val2': 4}, 'param2': 'a repeated value'}, {'param1': {'val1': 5, 'val2': 6}, 'param2': 'a repeated value'}]

Zipped parameters need to have the same lenght.

Parameter filtering is supported, visit the Advanced Configuration for more information.

More advanced features are available (Advanced Configuration)such as:

  • Downloading remote data

  • Copying input file between disks

  • Pruning the parameter space

  • Specifying custom performance variables