debugging/valgrind/example_rank_league
Valgrind
Example: OpenMPI with valgrind support
FAQ: Debugging applications in parallel: What kind of errors can Memchecker find?)
Set up valgrind environment
module purge export VALGRIND=true module add \ \ compiler/gnu \ mpi/openmpi devel/valgrind # or to update environment export VALGRIND=true module update
To check if valgrind tool memchecker is available in OpenMPI, simply run this command:
ompi_info | grep memchecker
Build
rank_league
benchmark with debug symbols and low optimization settingsmpicc -O1 -g rank_league.c -o rank_league
Examine
rank_leage
benchmark with valgrind memchecker. Use one output file per rankmpirun -np 4 bash -c \ 'valgrind --log-file=valgrind.out.${OMPI_COMM_WORLD_RANK} \ --suppressions="${MPI_ROOT}/share/openmpi/openmpi-valgrind.supp" \ --suppressions="/usr/share/hwloc/hwloc-valgrind.supp" \ rank_league '
-> many false positives -> massive increase in runtime
valgrind
-v
command line options show used suppressions files... --781867-- Reading suppressions file: /software/all/mpi/openmpi/4.0_gnu_10_valgrind/share/openmpi/openmpi-valgrind.supp --781867-- Reading suppressions file: /usr/share/hwloc/hwloc-valgrind.supp --781867-- Reading suppressions file: /software/all/devel/valgrind/3.19.0_gnu_10_openmpi_4.0/libexec/valgrind/default.supp ...
Example: Valgrind with MPI support
Set up valgrind environment
module purge export VALGRIND=true module add \ \ compiler/gnu \ mpi/openmpi devel/valgrind # or to update environment export VALGRIND=true module update
Build
rank_league
benchmark with debug symbols and low optimization settingsmpicc -O1 -g rank_league.c -o rank_league
mpirun -np 4 bash -c \ 'LD_PRELOAD="${VALGRIND_LIB_DIR}/valgrind/libmpiwrap-amd64-linux.so" \ MPIWRAP_DEBUG="warn" \ valgrind --log-file=valgrind.out.${OMPI_COMM_WORLD_RANK} \ rank_league 2>valgrind.stderr.${OMPI_COMM_WORLD_RANK}'
valgrind.stderr.0:
valgrind MPI wrappers 785165: Active for pid 785165 valgrind MPI wrappers 785165: Try MPIWRAP_DEBUG=help for possible options valgrind MPI wrappers 785165: warning: no wrapper for PMPI_Get_processor_name valgrind MPI wrappers 785165: warning: no wrapper for PMPI_Barrier
-> fewer false positives -> still massive increase in runtime
Valgrind MPI wrapper options
LD_PRELOAD="${VALGRIND_LIB_DIR}/valgrind/libmpiwrap-amd64-linux.so" \ MPIWRAP_DEBUG="help" \ valgrind rank_league
Valid options for the MPIWRAP_DEBUG environment variable are: quiet be silent except for errors verbose show wrapper entries/exits strict abort the program if a function with no wrapper is used warn give a warning if a function with no wrapper is used help display this message, then exit initkludge debugging hack; do not use ...