Motivation
The iterative-solver timing currently gives the total time spent in the solver, while MatVec is timed separately. However, the remaining solver cost may contain a significant contribution from vector updates, inner products, copies, and MPI communication, especially when the MatVec itself is accelerated on the GPU or shared between several systems.
It would therefore be useful to have an optional detailed breakdown of the linear-algebra operations performed by the iterative solvers.
A prototype implementation is available in the feature/timings branch.
Proposed profiling
The branch introduces a compile-time option:
When enabled, it profiles the linear-algebra operations used by the iterative solver.
For the CPU/MPI implementations, the profiling records for each linalg.c operation:
- number of calls;
- cumulative execution time over the complete solver run;
Operations executed internally by MatVec are excluded from this profiling to avoid double counting, since MatVec already has its own timing.
This makes it possible to distinguish, for example, the relative costs of:
- vector updates;
- inner products;
- vector copies;
- scaling operations;
- MPI reductions/communications;
- MatVec itself.
OCL_BLAS profiling
The same profiling mechanism is also implemented for OCL_BLAS.
Since OpenCL operations are asynchronous, simply timing the corresponding host calls would not provide meaningful device execution times. Instead, OpenCL events are collected from:
- clBLAS operations;
- device buffer copies;
- solver-specific OpenCL kernels.
The device execution times are then obtained from the event profiling information.
Importantly, the profiling is designed not to introduce additional synchronization points between the individual operations. The normal asynchronous execution of the solver is therefore preserved as much as possible.
As for the CPU implementation, timings are reported both for:
- the complete solution;
- the last complete solver iteration.
Motivation
The iterative-solver timing currently gives the total time spent in the solver, while MatVec is timed separately. However, the remaining solver cost may contain a significant contribution from vector updates, inner products, copies, and MPI communication, especially when the MatVec itself is accelerated on the GPU or shared between several systems.
It would therefore be useful to have an optional detailed breakdown of the linear-algebra operations performed by the iterative solvers.
A prototype implementation is available in the
feature/timingsbranch.Proposed profiling
The branch introduces a compile-time option:
When enabled, it profiles the linear-algebra operations used by the iterative solver.
For the CPU/MPI implementations, the profiling records for each
linalg.coperation:Operations executed internally by MatVec are excluded from this profiling to avoid double counting, since MatVec already has its own timing.
This makes it possible to distinguish, for example, the relative costs of:
OCL_BLAS profiling
The same profiling mechanism is also implemented for
OCL_BLAS.Since OpenCL operations are asynchronous, simply timing the corresponding host calls would not provide meaningful device execution times. Instead, OpenCL events are collected from:
The device execution times are then obtained from the event profiling information.
Importantly, the profiling is designed not to introduce additional synchronization points between the individual operations. The normal asynchronous execution of the solver is therefore preserved as much as possible.
As for the CPU implementation, timings are reported both for: