Figures

In order to generate reports, the Feel++ benchmarking framework requires a figure description to specify what the website report page should contain.

These descriptions should be provided either with a specific JSON file with the structure containing uniquely

{
    "plots":[]
}

Or by specifying the plots field on the benchmark configuration JSON file.

Each figure description should contain the following fields

{
    "title": "The figure title",
    "plot_types": [], //List of figure types
    "transformation": "", //Transformation type
    "variables":[], // List of variables to consider
    "names":[], //Respective labels for variables
    "yaxis":{},
    "xaxis":{},
    "color_axis":{}, //Default: performance variables
    "secondary_axis":{}
}

Figures will appear in the same order as they appear on the list.

Users can provide multiple plot_types in the same description field.

Only performance variables specified under the variables list will be considered. If the list is empty, ALL variables inside the ReFrame report will be taken into account.

1. Axis

Each axis (with the exception of the yaxis) take a parameter and a label field. The yaxis will contain performance values by default, but it can be overwritten by specifying the parameter field.

The parameter field of each axis should correspond to either a single dimension parameter specified on the benchmark configuration. In the case of subparameters, the syntax should be the following: parameter.subparameter.

By default, the color axis will contain the performance variables, but this can be customized.

2. Transformations

The ReFrame report will be used to create a Master DataFrame, which will contain all performance variables and their respective values, as well as all parameters and environments.

To explain how transformation and plot types work, we can consider the following example.

records = [{"perfvalue": "elapsed_fill", "value": 28.9239, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 1.0, "elements": 10000.0}, {"perfvalue": "elapsed_compute", "value": 1.92806, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 1.0, "elements": 10000.0}, {"perfvalue": "elapsed_fill", "value": 63.0273, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 1.0, "elements": 20000.0}, {"perfvalue": "elapsed_compute", "value": 3.74504, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 1.0, "elements": 20000.0}, {"perfvalue": "elapsed_fill", "value": 92.358, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 1.0, "elements": 30000.0}, {"perfvalue": "elapsed_compute", "value": 3.29597, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 1.0, "elements": 30000.0}, {"perfvalue": "elapsed_fill", "value": 99.9142, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 1.0, "elements": 40000.0}, {"perfvalue": "elapsed_compute", "value": 2.37914, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 1.0, "elements": 40000.0}, {"perfvalue": "elapsed_fill", "value": 24.8636, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 2.0, "elements": 10000.0}, {"perfvalue": "elapsed_compute", "value": 0.629236, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 2.0, "elements": 10000.0}, {"perfvalue": "elapsed_fill", "value": 14.9139, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 2.0, "elements": 20000.0}, {"perfvalue": "elapsed_compute", "value": 0.498528, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 2.0, "elements": 20000.0}, {"perfvalue": "elapsed_fill", "value": 98.9536, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 2.0, "elements": 30000.0}, {"perfvalue": "elapsed_compute", "value": 2.3708, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 2.0, "elements": 30000.0}, {"perfvalue": "elapsed_fill", "value": 45.4764, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 2.0, "elements": 40000.0}, {"perfvalue": "elapsed_compute", "value": 1.56548, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 2.0, "elements": 40000.0}, {"perfvalue": "elapsed_fill", "value": 2.89716, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 4.0, "elements": 10000.0}, {"perfvalue": "elapsed_compute", "value": 0.0434488, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 4.0, "elements": 10000.0}, {"perfvalue": "elapsed_fill", "value": 36.537, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 4.0, "elements": 20000.0}, {"perfvalue": "elapsed_compute", "value": 0.519222, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 4.0, "elements": 20000.0}, {"perfvalue": "elapsed_fill", "value": 52.302, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 4.0, "elements": 30000.0}, {"perfvalue": "elapsed_compute", "value": 1.23368, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 4.0, "elements": 30000.0}, {"perfvalue": "elapsed_fill", "value": 98.397, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 4.0, "elements": 40000.0}, {"perfvalue": "elapsed_compute", "value": 1.55742, "unit": "s", "result": "pass", "system": "gaya", "partition": "public", "tasks": 4.0, "elements": 40000.0}]

Out[0]:
         perfvalue     value unit result system partition  tasks  elements
0     elapsed_fill  28.92390    s   pass   gaya    public    1.0   10000.0
1  elapsed_compute   1.92806    s   pass   gaya    public    1.0   10000.0
2     elapsed_fill  63.02730    s   pass   gaya    public    1.0   20000.0
3  elapsed_compute   3.74504    s   pass   gaya    public    1.0   20000.0
4     elapsed_fill  92.35800    s   pass   gaya    public    1.0   30000.0

We can see that this dataframe contains the parameters: - system - result - tasks - elements - value - perfvalue - unit

By having this common structure, we can make use of transformation strategies to manipulate values depending on the desired output.

Strategies will depend on the figure axis. All strategies will create a pivot dataframe that will contain the parameter specified as color_axis as columns, xaxis as first level index and secondary_axis as second level index. Values of the dataframe will always be the values of the master dataframe.

As an example, we will consider the following axis:

"xaxis":{
    "parameter":"tasks",
    "label":"Number of tasks"
},
"yaxis":{
    "label":"Execution time (s)"
},
"secondary_axis":{
    "parameter":"elements",
    "label":"N"
},
"color_axis":{
    "parameter":"perfvalue",
    "label":"Performance variable"
}

Available strategies are:

performance

This strategy should be seen as the "base" strategy. No transformation, other that a pivot, is done. For the given example, it produces the following dataframe

from feelpp.benchmarking.json_report.figures.transformationFactory import TransformationStrategyFactory
from feelpp.benchmarking.reframe.schemas.benchmarkSchemas import DefaultPlot
plot_config = DefaultPlot(**{
    "title": "Absolute performance",
    "plot_types": [ "stacked_bar", "grouped_bar" ],
    "xaxis":{ "parameter":"tasks", "label":"Number of tasks" },
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{"parameter":"elements", "label":"N"}
})
strategy = TransformationStrategyFactory.create(plot_config)
df = strategy.calculate(master_df)
print(df)

perfvalue       elapsed_compute  elapsed_fill
elements tasks
10000.0  1.0           1.928060      28.92390
         2.0           0.629236      24.86360
         4.0           0.043449       2.89716
20000.0  1.0           3.745040      63.02730
         2.0           0.498528      14.91390
         4.0           0.519222      36.53700
30000.0  1.0           3.295970      92.35800
         2.0           2.370800      98.95360
         4.0           1.233680      52.30200
40000.0  1.0           2.379140      99.91420
         2.0           1.565480      45.47640
         4.0           1.557420      98.39700

relative_performance

The relative performance strategy computes the proportion of the time that a a color_axis variable takes with regards of the total.

plot_config = DefaultPlot(**{
    "title": "Absolute performance",
    "plot_types": [ "stacked_bar", "grouped_bar" ],
    "transformation": "relative_performance",
    "xaxis":{ "parameter":"tasks", "label":"Number of tasks"
    },
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{ "parameter":"elements", "label":"N"
    }
})
strategy = TransformationStrategyFactory.create(plot_config)
df = strategy.calculate(master_df)
print(df)

perfvalue       elapsed_compute  elapsed_fill
elements tasks
10000.0  1.0           6.249392     93.750608
         2.0           2.468286     97.531714
         4.0           1.477544     98.522456
20000.0  1.0           5.608670     94.391330
         2.0           3.234584     96.765416
         4.0           1.401174     98.598826
30000.0  1.0           3.445722     96.554278
         2.0           2.339812     97.660188
         4.0           2.304407     97.695593
40000.0  1.0           2.325801     97.674199
         2.0           3.327843     96.672157
         4.0           1.558130     98.441870

The sum along the column axis will always be equal to 1.

speedup

The speedup strategy computes the speedup of the color_axis variables. The minimum of the xaxis values is taken as the base of the speedup. For the example, this strategy will produce the following.

plot_config = DefaultPlot(**{
    "title": "Absolute performance",
    "plot_types": [ "stacked_bar", "grouped_bar" ],
    "transformation": "speedup",
    "xaxis":{"parameter":"tasks", "label":"Number of tasks"},
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{"parameter":"elements", "label":"N"}
})
strategy = TransformationStrategyFactory.create(plot_config)
df = strategy.calculate(master_df)
print(df)

perfvalue       elapsed_compute  elapsed_fill  optimal  half-optimal
elements tasks
10000.0  1.0           1.000000      1.000000      1.0           1.0
         2.0           3.064129      1.163303      2.0           1.5
         4.0          44.375449      9.983536      4.0           2.5
20000.0  1.0           1.000000      1.000000      1.0           1.0
         2.0           7.512196      4.226078      2.0           1.5
         4.0           7.212791      1.725027      4.0           2.5
30000.0  1.0           1.000000      1.000000      1.0           1.0
         2.0           1.390235      0.933347      2.0           1.5
         4.0           2.671657      1.765860      4.0           2.5
40000.0  1.0           1.000000      1.000000      1.0           1.0
         2.0           1.519751      2.197056      2.0           1.5
         4.0           1.527616      1.015419      4.0           2.5

3. Plot types

Considering the same example axis as above, the software can generate the following figures:

scatter

from feelpp.benchmarking.json_report.figures.figureFactory import FigureFactory
figures = FigureFactory.create(DefaultPlot(**{
    "title": "Absolute performance - Scatter Plot",
    "plot_types": [ "scatter" ],
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{"parameter":"elements", "label":"N"},
    "xaxis":{"parameter":"tasks", "label":"Number of tasks"}
}))
fig = figures[0].createFigure(master_df)
fig.show()

marked_scatter

The marked scatter plot type supports from 2 to 4 dimensions. The symbol/marks axis will correspond to the secondary_axis parameter. This plot type will behave as follows:

If 1 or 2 dimensions are specified (x-axis and optionally color-axis), then this plot type will be equivalent to scatter.
If 3 dimensions are specified (x-axis, color-axis and secondary-axis), then the secondary_axis will correspond to the symbol/marks axis.
If 4 dimensions are specified (x-axis, color-axis, secondary-axis and one extra-axis), then the first element of the extra_axes list will correspond to the symbol/marks axis, and the secondary_axis will correspond to the slider of the returned animation.

figures = FigureFactory.create(DefaultPlot(**{
    "title": "Absolute performance - Marked Scatter Plot",
    "plot_types": [ "marked_scatter" ],
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{"parameter":"elements", "label":"N"},
    "xaxis":{"parameter":"tasks", "label":"Number of tasks"}
}))
fig = figures[0].createFigure(master_df)
fig.show()

stacked_bar

figures = FigureFactory.create(DefaultPlot(**{
    "title": "Absolute performance - Stacked Bar Plot",
    "plot_types": [ "stacked_bar" ],
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{"parameter":"elements", "label":"N"},
    "xaxis":{"parameter":"tasks", "label":"Number of tasks"}
}))
fig = figures[0].createFigure(master_df)
fig.show()

grouped_bar

figures = FigureFactory.create(DefaultPlot(**{
    "title": "Absolute performance - Grouped Bar Plot",
    "plot_types": [ "grouped_bar" ],
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{"parameter":"elements", "label":"N"},
    "xaxis":{"parameter":"tasks", "label":"Number of tasks"}
}))
fig = figures[0].createFigure(master_df)
fig.show()

heatmap

For this case, we will consider the elements (N) as color_axis and performance_variable for secondary axis (slider).

figures = FigureFactory.create(DefaultPlot(**{
    "title": "Absolute performance - HeatMap",
    "plot_types": [ "heatmap" ],
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{"parameter":"perfvalue", "label":"Performance Variable"},
    "xaxis":{"parameter":"tasks", "label":"Number of tasks"},
    "color_axis":{"parameter":"elements", "label":"N"},
}))
fig = figures[0].createFigure(master_df)
fig.show()

table

figures = FigureFactory.create(DefaultPlot(**{
    "title": "Absolute performance - Table",
    "plot_types": [ "table" ],
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{"parameter":"elements", "label":"N"},
    "xaxis":{"parameter":"tasks", "label":"Number of tasks"}
}))
fig = figures[0].createFigure(master_df)
fig.show()

sunburst

This figure considers the color_axis parameter as the outer-most ring. Users can supply an extra_axes field, containing a list of additional parameters. Values for these parameters whill be shown on the rings that follow the color_axis ring, in the order they are provided. The secondary_axis and xaxis parameter are present respectively on the inner-most and second inner-most rings.

figures = FigureFactory.create(DefaultPlot(**{
    "title": "Absolute performance - Sunburst Plot",
    "plot_types": [ "sunburst" ],
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{"parameter":"elements", "label":"N"},
    "xaxis":{"parameter":"tasks", "label":"Number of tasks"}
}))
fig = figures[0].createFigure(master_df)
fig.show()

parallelcoordinates

Axes will be shown on the following order: secondary_axis, xaxis, all additional extra_axes, color_axis. The yaxis will be shown in the line color.

figures = FigureFactory.create(DefaultPlot(**{
    "title": "Absolute performance - Parallel Coordinates Plot",
    "plot_types": [ "parallelcoordinates" ],
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{"parameter":"elements", "label":"N"},
    "xaxis":{"parameter":"tasks", "label":"Number of tasks"}
}))
fig = figures[0].createFigure(master_df)
fig.show()

3.1. 3D Plots

3D plots are also supported, and then can show up to 4 dimensions. At least 3 parameters must be provided (xaxis,color_axis and secondary_axis ). Axes correspondance is as follows:

x-axis of the 3D plot: xaxis
y-axis of the 3D plot: secondary_axis if no extra axes are provided, else, the first element of the extra_axes list.
z-axis of the 3D plot: yaxis (contains the measured values)
color of the 3D traces: color_axis
Slider: secondary_axis if extra axes are provided.
scatter3d

figures = FigureFactory.create(DefaultPlot(**{
    "title": "Absolute performance - Scatter 3D",
    "plot_types": [ "scatter3d" ],
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{"parameter":"elements", "label":"N"},
    "xaxis":{"parameter":"tasks", "label":"Number of tasks"}
}))
fig = figures[0].createFigure(master_df)
fig.show()

surface3d

figures = FigureFactory.create(DefaultPlot(**{
    "title": "Absolute performance - Surface 3D",
    "plot_types": [ "surface3d" ],
    "yaxis":{"label":"Execution time (s)"},
    "secondary_axis":{"parameter":"elements", "label":"N"},
    "xaxis":{"parameter":"tasks", "label":"Number of tasks"}
}))
fig = figures[0].createFigure(master_df)
fig.show()

4. Aggregations

Depending on the dashboard level that we are located at, it might be necessary to aggregate the data on the master dataframe. For example, if we have all use cases, applications and machines on the dataframe, and we want to see how a certain use case performs on different machines, we can make use of the aggregations field to group the data accordingly.

"aggregations":[
    {"column":"date","agg":"max"},
    {"column":"applications","agg":"filter:my_app"},
    {"column":"use_cases","agg":"filter:my_use_case"},
    {"column":"performance_variable","agg":"sum"}
]

The previous example will first get only the latest benchmarks (by getting the maximum date), then it will filter the application and the use case to find applications and use cases that correspond to "my_app" and "my_use_case". And finally it will compute the sum of all performance variables for the remaining rows.

Users must provide a column and an aggregation function as a string.

Available aggregations are:

mean : Computes the mean of the column
mean : Computes the sum of the column
max : Computes the maximum of the column
min : Computes the minimum of the column
filter:value: Filters the column by value.

The order of the aggregations list is important.

5. Custom layouts

By providing the layout_modifiers field, users can pass custom layout options for rendering the figures. These options correspond to the accepted layout reference for Plotly: Plotly layout reference It accepts a nested dictionnary just as Plotly does.

For example, we could customize a figure to have have its x-axis on a logscale.

"layout_modifiers":{
    "xaxis":{
        "type":"log"
    }
}