Online Performance Tools

# On-line Performance Feedback

## Enabling On-line Performance Monitoring

In order to enable online performance monitoring, the application can call starpu_profiling_status_set() with the parameter STARPU_PROFILING_ENABLE. It is possible to detect whether monitoring is already enabled or not by calling starpu_profiling_status_get(). Enabling monitoring also reinitialize all previously collected feedback. The environment variable STARPU_PROFILING can also be set to `1` to achieve the same effect. The function starpu_profiling_init() can also be called during the execution to reinitialize performance counters and to start the profiling if the environment variable STARPU_PROFILING is set to `1`.

Likewise, performance monitoring is stopped by calling starpu_profiling_status_set() with the parameter STARPU_PROFILING_DISABLE. Note that this does not reset the performance counters so that the application may consult them later on.

More details about the performance monitoring API are available in Profiling.

## Per-task Feedback

If profiling is enabled, a pointer to a structure starpu_profiling_task_info is put in the field starpu_task::profiling_info when a task terminates. This structure is automatically destroyed when the task structure is destroyed, either automatically or by calling starpu_task_destroy().

The structure starpu_profiling_task_info indicates the date when the task was submitted (starpu_profiling_task_info::submit_time), started (starpu_profiling_task_info::start_time), and terminated (starpu_profiling_task_info::end_time), relative to the initialization of StarPU with starpu_init(). It also specifies the identifier of the worker that has executed the task (starpu_profiling_task_info::workerid). These date are stored as `timespec` structures which the user may convert into micro-seconds using the helper function starpu_timing_timespec_to_us().

It it worth noting that the application may directly access this structure from the callback executed at the end of the task. The structure starpu_task associated to the callback currently being executed is indeed accessible with the function starpu_task_get_current().

## Per-codelet Feedback

The field starpu_codelet::per_worker_stats is an array of counters. The `i`-th entry of the array is incremented every time a task implementing the codelet is executed on the `i`-th worker. This array is not reinitialized when profiling is enabled or disabled.

## Per-worker Feedback

The second argument returned by the function starpu_profiling_worker_get_info() is a structure starpu_profiling_worker_info that gives statistics about the specified worker. This structure specifies when StarPU started collecting profiling information for that worker (starpu_profiling_worker_info::start_time), the duration of the profiling measurement interval (starpu_profiling_worker_info::total_time), the time spent executing kernels (starpu_profiling_worker_info::executing_time), the time spent sleeping because there is no task to execute at all (starpu_profiling_worker_info::sleeping_time), and the number of tasks that were executed while profiling was enabled. These values give an estimation of the proportion of time spent do real work, and the time spent either sleeping because there are not enough executable tasks or simply wasted in pure StarPU overhead.

Calling starpu_profiling_worker_get_info() resets the profiling information associated to a worker.

To easily display all this information, the environment variable STARPU_WORKER_STATS can be set to `1` (in addition to setting STARPU_PROFILING to 1). A summary will then be displayed at program termination. To display the summary in a file instead of the standard error stream, use the environment variable STARPU_WORKER_STATS_FILE.

    Worker stats:
    CUDA 0.0 (4.7 GiB)
    	480 task(s)
    	total: 1574.82 ms executing: 1510.72 ms sleeping: 0.00 ms overhead 64.10 ms
    	325.217970 GFlop/s

    CPU 0
    	22 task(s)
    	total: 1574.82 ms executing: 1364.81 ms sleeping: 0.00 ms overhead 210.01 ms
    	7.512057 GFlop/s

    CPU 1
    	14 task(s)
    	total: 1574.82 ms executing: 1500.13 ms sleeping: 0.00 ms overhead 74.69 ms
    	6.675853 GFlop/s

    CPU 2
    	14 task(s)
    	total: 1574.82 ms executing: 1553.12 ms sleeping: 0.00 ms overhead 21.70 ms
    	7.152886 GFlop/s

The number of GFlops/s is available because the starpu_task::flops field of the tasks were filled (or STARPU_FLOPS used in starpu_task_insert()).

When an FxT trace is generated (see Generating Traces With FxT), it is also possible to use the tool `starpu_workers_activity` (see Monitoring Activity) to generate a graphic showing the evolution of these values during the time, for the different workers.

## Bus-related Feedback

The bus speed measured by StarPU can be displayed by using the tool `starpu_machine_display`, for instance:

    StarPU has found:
            3 CUDA devices
                    CUDA 0 (Tesla C2050 02:00.0)
                    CUDA 1 (Tesla C2050 03:00.0)
                    CUDA 2 (Tesla C2050 84:00.0)
    from    to RAM          to CUDA 0       to CUDA 1       to CUDA 2
    RAM     0.000000        5176.530428     5176.492994     5191.710722
    CUDA 0  4523.732446     0.000000        2414.074751     2417.379201
    CUDA 1  4523.718152     2414.078822     0.000000        2417.375119
    CUDA 2  4534.229519     2417.069025     2417.060863     0.000000

Statistics about the data transfers which were performed and temporal average of bandwidth usage can be obtained by setting the environment variable STARPU_BUS_STATS to `1`; a summary will then be displayed at program termination. To display the summary in a file instead of the standard error stream, use the environment variable STARPU_BUS_STATS_FILE.

    Data transfer stats:
    	RAM 0 -> CUDA 0	319.92 MB	213.10 MB/s	(transfers : 91 - avg 3.52 MB)
    	CUDA 0 -> RAM 0	214.45 MB	142.85 MB/s	(transfers : 61 - avg 3.52 MB)
    	RAM 0 -> CUDA 1	302.34 MB	201.39 MB/s	(transfers : 86 - avg 3.52 MB)
    	CUDA 1 -> RAM 0	133.59 MB	88.99 MB/s	(transfers : 38 - avg 3.52 MB)
    	CUDA 0 -> CUDA 1	144.14 MB	96.01 MB/s	(transfers : 41 - avg 3.52 MB)
    	CUDA 1 -> CUDA 0	130.08 MB	86.64 MB/s	(transfers : 37 - avg 3.52 MB)
    	RAM 0 -> CUDA 2	312.89 MB	208.42 MB/s	(transfers : 89 - avg 3.52 MB)
    	CUDA 2 -> RAM 0	133.59 MB	88.99 MB/s	(transfers : 38 - avg 3.52 MB)
    	CUDA 0 -> CUDA 2	151.17 MB	100.69 MB/s	(transfers : 43 - avg 3.52 MB)
    	CUDA 2 -> CUDA 0	105.47 MB	70.25 MB/s	(transfers : 30 - avg 3.52 MB)
    	CUDA 1 -> CUDA 2	175.78 MB	117.09 MB/s	(transfers : 50 - avg 3.52 MB)
    	CUDA 2 -> CUDA 1	203.91 MB	135.82 MB/s	(transfers : 58 - avg 3.52 MB)
    Total transfers: 2.27 GB

## MPI-related Feedback

Statistics about the data transfers which were performed over MPI can be obtained by setting the environment variable STARPU_COMM_STATS to `1`; a summary will then be displayed at program termination:

    [starpu_comm_stats][1] TOTAL:	456.000000 B	0.000435 MB	 0.000188 B/s	 0.000000 MB/s
    [starpu_comm_stats][1:0]	456.000000 B	0.000435 MB	 0.000188 B/s	 0.000000 MB/s

    [starpu_comm_stats][0] TOTAL:	456.000000 B	0.000435 MB	 0.000188 B/s	 0.000000 MB/s
    [starpu_comm_stats][0:1]	456.000000 B	0.000435 MB	 0.000188 B/s	 0.000000 MB/s

These statistics can be plotted as heatmaps using StarPU tool `starpu_mpi_comm_matrix.py` (see Debugging MPI).

# Task And Worker Profiling

A full example showing how to use the profiling API is available in the StarPU sources in the directory `examples/profiling/`.

struct starpu_task *task = starpu_task_create();
task->cl = &cl;
task->synchronous = 1;
/* We will destroy the task structure by hand so that we can
* query the profiling info before the task is destroyed. */
task->destroy = 0;
/* Submit and wait for completion (since synchronous was set to 1) */
starpu_task_submit(task);
/* The task is finished, get profiling information */
struct starpu_profiling_task_info *info = task->profiling_info;
/* How much time did it take before the task started ? */
double delay += starpu_timing_timespec_delay_us(&info->submit_time, &info->start_time);
/* How long was the task execution ? */
double length += starpu_timing_timespec_delay_us(&info->start_time, &info->end_time);
/* We no longer need the task structure */
starpu_task_destroy(task);
starpu_task::synchronous
unsigned synchronous
Definition: starpu_task.h:907
starpu_task::destroy
unsigned destroy
Definition: starpu_task.h:945
starpu_task::profiling_info
struct starpu_profiling_task_info * profiling_info
Definition: starpu_task.h:1137
starpu_task::cl
struct starpu_codelet * cl
Definition: starpu_task.h:584
starpu_task_create
struct starpu_task * starpu_task_create(void) STARPU_ATTRIBUTE_MALLOC
starpu_task_destroy
void starpu_task_destroy(struct starpu_task *task)
starpu_task_submit
int starpu_task_submit(struct starpu_task *task) STARPU_WARN_UNUSED_RESULT
starpu_task
Definition: starpu_task.h:567
starpu_profiling_task_info::submit_time
struct timespec submit_time
Definition: starpu_profiling.h:51
starpu_profiling_task_info::start_time
struct timespec start_time
Definition: starpu_profiling.h:68
starpu_profiling_task_info::end_time
struct timespec end_time
Definition: starpu_profiling.h:70
starpu_timing_timespec_delay_us
double starpu_timing_timespec_delay_us(struct timespec *start, struct timespec *end)
starpu_profiling_task_info
Definition: starpu_profiling.h:49
/* Display the occupancy of all workers during the test */
int worker;
for (worker = 0; worker < starpu_worker_get_count(); worker++)
{
struct starpu_profiling_worker_info worker_info;
int ret = starpu_profiling_worker_get_info(worker, &worker_info);
STARPU_ASSERT(!ret);
double total_time = starpu_timing_timespec_to_us(&worker_info.total_time);
double executing_time = starpu_timing_timespec_to_us(&worker_info.executing_time);
double sleeping_time = starpu_timing_timespec_to_us(&worker_info.sleeping_time);
double overhead_time = total_time - executing_time - sleeping_time;
float executing_ratio = 100.0*executing_time/total_time;
float sleeping_ratio = 100.0*sleeping_time/total_time;
float overhead_ratio = 100.0 - executing_ratio - sleeping_ratio;
char workername[128];
starpu_worker_get_name(worker, workername, 128);
fprintf(stderr, "Worker %s:\n", workername);
fprintf(stderr, "\ttotal time: %.2lf ms\n", total_time*1e-3);
fprintf(stderr, "\texec time: %.2lf ms (%.2f %%)\n", executing_time*1e-3, executing_ratio);
fprintf(stderr, "\tblocked time: %.2lf ms (%.2f %%)\n", sleeping_time*1e-3, sleeping_ratio);
fprintf(stderr, "\toverhead time: %.2lf ms (%.2f %%)\n", overhead_time*1e-3, overhead_ratio);
}
starpu_profiling_worker_info::executing_time
struct timespec executing_time
Definition: starpu_profiling.h:107
starpu_profiling_worker_info::total_time
struct timespec total_time
Definition: starpu_profiling.h:105
starpu_profiling_worker_info::sleeping_time
struct timespec sleeping_time
Definition: starpu_profiling.h:109
starpu_timing_timespec_to_us
double starpu_timing_timespec_to_us(struct timespec *ts)
starpu_profiling_worker_get_info
int starpu_profiling_worker_get_info(int workerid, struct starpu_profiling_worker_info *worker_info)
starpu_profiling_worker_info
Definition: starpu_profiling.h:101
STARPU_ASSERT
#define STARPU_ASSERT(x)
Definition: starpu_util.h:205
starpu_worker_get_count
unsigned starpu_worker_get_count(void)
starpu_worker_get_name
void starpu_worker_get_name(int id, char *dst, size_t maxlen)

# Performance Model Example

To achieve good scheduling, StarPU scheduling policies need to be able to estimate in advance the duration of a task. This is done by giving to codelets a performance model, by defining a structure starpu_perfmodel and providing its address in the field starpu_codelet::model. The fields starpu_perfmodel::symbol and starpu_perfmodel::type are mandatory, to give a name to the model, and the type of the model, since there are several kinds of performance models. For compatibility, make sure to initialize the whole structure to zero, either by using explicit memset(), or by letting the compiler implicitly do it as examplified below.

- Measured at runtime (model type STARPU_HISTORY_BASED). This assumes that for a given set of data input/output sizes, the performance will always be about the same. This is very true for regular kernels on GPUs for instance (<0.1% error), and just a bit less true on CPUs (~=1% error). This also assumes that there are few different sets of data input/output sizes. StarPU will then keep record of the average time of previous executions on the various processing units, and use it as an estimation. History is done per task size, by using a hash of the input and ouput sizes as an index. It will also save it in `$STARPU_HOME/.starpu/sampling/codelets` for further executions, and can be observed by using the tool `starpu_perfmodel_display`, or drawn by using the tool `starpu_perfmodel_plot` (Performance Model Calibration). The models are indexed by machine name. To share the models between machines (e.g. for a homogeneous cluster), use `export STARPU_HOSTNAME=some_global_name`. Measurements are only done when using a task scheduler which makes use of it, such as `dmda`. Measurements can also be provided explicitly by the application, by using the function starpu_perfmodel_update_history().

  The following is a small code example.

  If e.g. the code is recompiled with other compilation options, or several variants of the code are used, the `symbol` string should be changed to reflect that, in order to recalibrate a new model from zero. The `symbol` string can even be constructed dynamically at execution time, as long as this is done before submitting any task using it.

  static struct starpu_perfmodel mult_perf_model =
  {
  .type = STARPU_HISTORY_BASED,
  .symbol = "mult_perf_model"
  };
  struct starpu_codelet cl =
  {
  .cpu_funcs = { cpu_mult },
  .cpu_funcs_name = { "cpu_mult" },
  .nbuffers = 3,
  .modes = { STARPU_R, STARPU_R, STARPU_W },
  /* for the scheduling policy to be able to use performance models */
  .model = &mult_perf_model
  };
  starpu_codelet::cpu_funcs
  starpu_cpu_func_t cpu_funcs[STARPU_MAXIMPLEMENTATIONS]
  Definition: starpu_task.h:326
  starpu_codelet
  Definition: starpu_task.h:250
  STARPU_W
  @ STARPU_W
  Definition: starpu_data.h:58
  STARPU_R
  @ STARPU_R
  Definition: starpu_data.h:57
  starpu_perfmodel::type
  enum starpu_perfmodel_type type
  Definition: starpu_perfmodel.h:221
  STARPU_HISTORY_BASED
  @ STARPU_HISTORY_BASED
  Definition: starpu_perfmodel.h:170
  starpu_perfmodel
  Definition: starpu_perfmodel.h:189

- Measured at runtime and refined by regression (model types STARPU_REGRESSION_BASED and STARPU_NL_REGRESSION_BASED). This still assumes performance regularity, but works with various data input sizes, by applying regression over observed execution times. STARPU_REGRESSION_BASED uses an `a*n^b` regression form, STARPU_NL_REGRESSION_BASED uses an `a*n^b+c` (more precise than STARPU_REGRESSION_BASED, but costs a lot more to compute).

  For instance, `tests/perfmodels/regression_based.c` uses a regression-based performance model for the function memset().

  Of course, the application has to issue tasks with varying size so that the regression can be computed. StarPU will not trust the regression unless there is at least 10% difference between the minimum and maximum observed input size. It can be useful to set the environment variable STARPU_CALIBRATE to `1` and run the application on varying input sizes with STARPU_SCHED set to `dmda` scheduler, so as to feed the performance model for a variety of inputs. The application can also provide the measurements explictly by using the function starpu_perfmodel_update_history(). The tools `starpu_perfmodel_display` and `starpu_perfmodel_plot` can be used to observe how much the performance model is calibrated (Performance Model Calibration); when their output look good, STARPU_CALIBRATE can be reset to `0` to let StarPU use the resulting performance model without recording new measures, and STARPU_SCHED can be set to `dmda` to benefit from the performance models. If the data input sizes vary a lot, it is really important to set STARPU_CALIBRATE to `0`, otherwise StarPU will continue adding the measures, and result with a very big performance model, which will take time a lot of time to load and save.

  For non-linear regression, since computing it is quite expensive, it is only done at termination of the application. This means that the first execution of the application will use only history-based performance model to perform scheduling, without using regression.

- Another type of model is STARPU_MULTIPLE_REGRESSION_BASED, which is based on multiple linear regression. In this model, the user defines both the relevant parameters and the equation for computing the task duration.

  are the parameters of the task, added at the task creation. These need to be extracted by the `cl_perf_func` function, which should be defined by the user. are the exponents defined by the user in `model->combinations` table. Finally, coefficients are computed automatically by the StarPU at the end of the execution, using least squares method of the `dgels_` LAPACK function.

  `examples/mlr/mlr.c` example provides more details on the usage of STARPU_MULTIPLE_REGRESSION_BASED models. The --enable-mlr configure option needs to be set to calibrate the model.

  Coefficients computation is done at the end of the execution, and the results are stored in standard codelet perfmodel files. Additional files containing the duration of task together with the value of each parameter are stored in `.starpu/sampling/codelets/tmp/` directory. These files are reused when STARPU_CALIBRATE environment variable is set to `1`, to recompute coefficients based on the current, but also on the previous executions. By default StarPU uses a lightweight dgels implementation, but the --enable-mlr-system-blas configure option can be used to make StarPU use a system-provided dgels BLAS.

  Additionally, when multiple linear regression models are not enabled through --enable-mlr or when the `model->combinations` are not defined, StarPU will still write output files into `.starpu/sampling/codelets/tmp/` to allow performing an analysis. This analysis typically aims at finding the most appropriate equation for the codelet and `tools/starpu_mlr_analysis` script provides an example of how to perform such study.

- Provided as an estimation from the application itself (model type STARPU_COMMON and field starpu_perfmodel::cost_function), see for instance `examples/common/blas_model.h` and `examples/common/blas_model.c`.

- Provided explicitly by the application (model type STARPU_PER_ARCH): either field starpu_perfmodel::arch_cost_function, or the fields `.per_arch[arch][nimpl].cost_function` have to be filled with pointers to functions which return the expected duration of the task in micro-seconds, one per architecture, see for instance `tests/datawizard/locality.c`

For STARPU_HISTORY_BASED, STARPU_REGRESSION_BASED, and STARPU_NL_REGRESSION_BASED, the dimensions of task data (both input and output) are used as an index by default. STARPU_HISTORY_BASED uses a CRC hash of the dimensions as an index to distinguish histories, and STARPU_REGRESSION_BASED and STARPU_NL_REGRESSION_BASED use the total size as an index for the regression.

The starpu_perfmodel::size_base and starpu_perfmodel::footprint fields however permit the application to override that, when for instance some of the data do not matter for task cost (e.g. mere reference table), or when using sparse structures (in which case it is the number of non-zeros which matter), or when there is some hidden parameter such as the number of iterations, or when the application actually has a very good idea of the complexity of the algorithm, and just not the speed of the processor, etc. The example in the directory `examples/pi` uses this to include the number of iterations in the base size. starpu_perfmodel::size_base should be used when the variance of the actual performance is known (i.e. bigger return value is longer execution time), and thus particularly useful for STARPU_REGRESSION_BASED or STARPU_NL_REGRESSION_BASED. starpu_perfmodel::footprint can be used when the variance of the actual performance is unknown (irregular performance behavior, etc.), and thus only useful for STARPU_HISTORY_BASED. starpu_task_data_footprint() can be used as a base and combined with other parameters through starpu_hash_crc32c_be() for instance.

StarPU will automatically determine when the performance model is calibrated, or rather, it will assume the performance model is calibrated until the application submits a task for which the performance can not be predicted. For STARPU_HISTORY_BASED, StarPU will require 10 (STARPU_CALIBRATE_MINIMUM) measurements for a given size before estimating that an average can be taken as estimation for further executions with the same size. For STARPU_REGRESSION_BASED and STARPU_NL_REGRESSION_BASED, StarPU will require 10 (STARPU_CALIBRATE_MINIMUM) measurements, and that the minimum measured data size is smaller than 90% of the maximum measured data size (i.e. the measurement interval is large enough for a regression to have a meaning). Calibration can also be forced by setting the STARPU_CALIBRATE environment variable to `1`, or even reset by setting it to `2`.

How to use schedulers which can benefit from such performance model is explained in Task Scheduling Policies.

The same can be done for task energy consumption estimation, by setting the field starpu_codelet::energy_model the same way as the field starpu_codelet::model. Note: for now, the application has to give to the energy consumption performance model a name which is different from the execution time performance model.

The application can request time estimations from the StarPU performance models by filling a task structure as usual without actually submitting it. The data handles can be created by calling any of the functions `starpu_*_data_register` with a `NULL` pointer and `-1` node and the desired data sizes, and need to be unregistered as usual. The functions starpu_task_expected_length() and starpu_task_expected_energy() can then be called to get an estimation of the task cost on a given arch. starpu_task_footprint() can also be used to get the footprint used for indexing history-based performance models. starpu_task_destroy() needs to be called to destroy the dummy task afterwards. See `tests/perfmodels/regression_based.c` for an example.

The application can also request an on-the-fly XML report of the performance model, by calling starpu_perfmodel_dump_xml() to print the report to a `FILE*`.