debugging/ltrace
Ltrace
Basic Usage
Build
stream
benchmarkmodule add compiler/intel/2022 icc -std=c11 -Ofast -xHost -ipo -qopenmp \ -o stream stream.c
Trace all linux function calls of benchmark stream
ltrace ./stream
Filter for
alloc
andfree
functions calls within the stream binary (ignoring these calls within libraries).OMP_NUM_THREADS=1 ltrace --demangle -e *alloc*@stream+free@stream stream >/dev/null
stream->aligned_alloc(64, 0x4c4b400, 0x14b547c66b00, 0x14b5491141c0) = 0x14b542a4d040 stream->aligned_alloc(64, 0x4c4b400, 0x14b542a4d040, 0x14b542a4d000) = 0x14b53de01040 stream->aligned_alloc(64, 0x4c4b400, 0x14b53de01040, 0x14b53de01000) = 0x14b5391b5040 stream->free(0x14b542a4d040) = <void> stream->free(0x14b53de01040) = <void> stream->free(0x14b5391b5040) = <void>
-> memory allocation and free for vectors a, b and c
Filter for
alloc
andfree
functions calls within the stream binary (ignoring these calls within libraries). Trace child processes to follow OpenMP Threads. Only count matching function calls.OMP_NUM_THREADS=2 ltrace -f --demangle -c -e *alloc*@stream+free@stream stream >/dev/null
% time seconds usecs/call calls function ------ ----------- ----------- --------- -------------------- 67.74 0.001951 325 6 free 25.42 0.000732 122 6 aligned_alloc 6.04 0.000174 174 1 exit_group 0.80 0.000023 23 1 exit ------ ----------- ----------- --------- -------------------- 100.00 0.002880 14 total
-> Each OpenMP Thread does its own memory allocation and free
Usage scenarios with OpenMPI
Build
rank_league
benchmarkmodule add \ \ compiler/intel/2022 mpi/openmpi/4.1mpicc -Ofast -xHost -ipo rank_league.c -o rank_league
Trace all MPI ranks. Output file per rank
mpirun -np 4 bash -c \ 'ltrace -o ltrace.out.${OMPI_COMM_WORLD_RANK} \ ./rank_league ll -h ltrace.out.*
-rw-r--r-- 1 bq0742 hk-project-scs 594K Jun 8 11:50 ltrace.out.0 -rw-r--r-- 1 bq0742 hk-project-scs 590K Jun 8 11:50 ltrace.out.1 -rw-r--r-- 1 bq0742 hk-project-scs 590K Jun 8 11:50 ltrace.out.2 -rw-r--r-- 1 bq0742 hk-project-scs 590K Jun 8 11:50 ltrace.out.3
Trace only first MPI rank
mpirun -np 4 bash -c \ 'if [[ ${OMPI_COMM_WORLD_RANK} -eq 0 ]]; then ltrace -o ltrace.out \ ./rank_league else ./rank_league fi' ll -h ltrace.out
-rw-r--r-- 1 bq0742 hk-project-scs 590K Jun 8 11:53 ltrace.out
Trace only first MPI rank. Count calls to MPI functions
mpirun -np 4 bash -c \ 'if [[ ${OMPI_COMM_WORLD_RANK} -eq 0 ]]; then ltrace -c -e *MPI* \ ./rank_league else ./rank_league fi'
% time seconds usecs/call calls function ------ ----------- ----------- --------- -------------------- 44.49 1.966504 1966504 1 MPI_Init 24.47 1.081684 1081684 1 exit_group 24.38 1.077691 1077691 1 MPI_Finalize 2.59 0.114608 143 800 MPI_Isend 2.11 0.093192 11649 8 MPI_Waitall 1.87 0.082807 103 800 MPI_Irecv 0.03 0.001316 329 4 MPI_Barrier 0.02 0.001008 126 8 MPI_Sendrecv 0.02 0.000903 112 8 MPI_Wtime 0.01 0.000330 110 3 MPI_Recv 0.00 0.000134 134 1 MPI_Get_processor_name 0.00 0.000125 125 1 MPI_Comm_size 0.00 0.000106 106 1 MPI_Comm_rank ------ ----------- ----------- --------- -------------------- 100.00 4.420408 1637 total