Rebase cuda-11.0 with kokkos-tools develop

.github/workflows/build-with-kokkos.yml

@@ -71,12 +71,14 @@ jobs:
                   exit -1
           esac
 
-      - name: Install CMake, OpenMPI and dtrace
+      - name: Install git, CMake, OpenMPI, PAPI and dtrace
         run: |
           apt --yes --no-install-recommends install \
+            git ca-certificates \
             cmake make \
             libopenmpi-dev \
-            systemtap-sdt-dev
+            systemtap-sdt-dev \
+            libpapi-dev
       - name: Compile and install Kokkos
         working-directory: kokkos
         run: |

.github/workflows/clang-format-check.yml

@@ -1,13 +1,11 @@
-name: Clang-Format Check
+name: Clang-Format check
 on: [push, pull_request]
 jobs:
   formatting-check:
-    name: Formatting Check
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v3
-    - name: Run clang-format style check for C/C++/Protobuf programs.
-      uses: jidicula/clang-format-action@v4.5.0
+    - name: Run clang-format style check.
+      uses: DoozyX/clang-format-lint-action@v0.16.2
       with:
-        clang-format-version: '8'
-        check-path: './'
+        clangFormatVersion: 8

CMakeLists.txt

@@ -45,6 +45,8 @@ option(KokkosTools_ENABLE_MPI     "Enable MPI support"              OFF)
 option(KokkosTools_ENABLE_CALIPER "Enable building Caliper library" OFF)
 option(KokkosTools_ENABLE_APEX    "Enable building Apex library"    OFF)
 option(KokkosTools_ENABLE_EXAMPLES "Build examples"                 OFF)
+option(KokkosTools_ENABLE_TESTS    "Build tests"                    OFF)
+
 # Advanced settings
 option(KokkosTools_REUSE_KOKKOS_COMPILER "Set the compiler and flags based on installed Kokkos settings" OFF)
 mark_as_advanced(KokkosTools_REUSE_KOKKOS_COMPILER)
@@ -95,7 +97,8 @@ endif()
 include(cmake/configure_variorum.cmake)
 
 set(KOKKOSTOOLS_HAS_CALIPER ${KokkosTools_ENABLE_CALIPER})
-set(KOKKOSTOOLS_HAS_NVTX ${Kokkos_ENABLE_CUDA}) # we assume that enabling CUDA for Kokkos program means nvtx should be available
+set(KOKKOSTOOLS_HAS_NVTX  ${Kokkos_ENABLE_CUDA}) # we assume that enabling CUDA for Kokkos program means nvtx should be available
+set(KOKKOSTOOLS_HAS_ROCTX ${Kokkos_ENABLE_HIP})  # we assume that enabling HIP for Kokkos program means roctx should be available
 
 if(DEFINED ENV{VTUNE_HOME})
   set(VTune_ROOT $ENV{VTUNE_HOME})
@@ -115,6 +118,9 @@ include_directories(${CMAKE_CURRENT_SOURCE_DIR}/profiling/all)
 set(COMMON_HEADERS_PATH ${CMAKE_CURRENT_BINARY_DIR}/common)
 include_directories(${COMMON_HEADERS_PATH})
 
+# Allow all tools to include any file.
+include_directories(${CMAKE_CURRENT_SOURCE_DIR}/common)
+
 set(SINGLELIB_PROFILERS "" CACHE STRING "" FORCE)
 
 # Export settings
@@ -263,6 +269,13 @@ if(KokkosTools_ENABLE_EXAMPLES)
   endif()
 endif()
 
+# Tests
+if(KokkosTools_ENABLE_TESTS)
+  enable_testing()
+  include(cmake/BuildGTest.cmake)
+  add_subdirectory(tests)
+endif()
+
 # Install exports
 install(TARGETS ${EXPORT_TARGETS} EXPORT ${EXPORT_NAME})
 install(EXPORT ${EXPORT_NAME}

CMakePresets.json

@@ -9,7 +9,9 @@
                 "CMAKE_CXX_STANDARD"                : "17",
                 "KokkosTools_ENABLE_EXAMPLES"       : "ON",
                 "KokkosTools_ENABLE_SINGLE"         : "ON",
-                "KokkosTools_ENABLE_MPI"            : "ON"
+                "KokkosTools_ENABLE_MPI"            : "ON",
+                "KokkosTools_ENABLE_PAPI"           : "ON",
+                "KokkosTools_ENABLE_TESTS"          : "ON"
             }
         },
         {

README.md

@@ -8,12 +8,38 @@ Note: `Kokkos` must be configured with `Kokkos_ENABLE_LIBDL=ON` to load profilin
 
 ## General Usage
 
-To use one of the tools you have to compile it, which will generate a dynamic library. Before executing the Kokkos application you then have to set the environment variable `KOKKOS_TOOLS_LIBS` to point to the dynamic library e.g. in the `bash` shell:
-```
-export KOKKOS_TOOLS_LIBS=${HOME}/kokkos-tools/src/tools/memory-events/kp_memory_event.so
-```
+To use one of the tools you have to compile it, which will generate a dynamic library. Before executing the Kokkos application you then have to set the environment variable `KOKKOS_TOOLS_LIBS` to point to the dynamic library.
+
+CMake and Makefiles are supported for building Kokkos Tools. The following provides instructions for both.
+
+## Using cmake 
+
+### Build
+
+1. Create a build directory in Kokkos Tools, e.g., type `mkdir myBuild; cd myBuild`
+2. To configure, type `cmake .. -DCMAKE_INSTALL_PREFIX=${YOUR_KOKKOS_TOOLS_INSTALL_DIR}`. There are more options but in most cases the defaults are sufficient.
+3. To compile, type `make`
+4. To install, type `make install`
+
+### Run 
+
+Given your installed tool shared library `lib<name_of_tool_shared_lib>.so` and an application executable called yourApplication.exe, type: 
+
+`export KOKKOS_TOOLS_LIBS=${YOUR_KOKKOS_TOOLS_INSTALL_DIR}/lib<name_of_tool_shared_lib>.so; ./yourApplication.exe`
+
+
+## Using make
+
+### Build
+
+To build some library `<name_of_tool_shared_lib>` with make, simply type `make` within that library's subdirectory `${YOUR_KOKKOS_TOOLS_LIB_SRC_DIR}` of Kokkos Tools. This generates the shared library within that subdirectory. 
+
+### Run 
+
+Given your installed tool shared library `<name_of_tool_shared_lib>.so` and an application executable called `yourApplication.exe`, type: 
+
+`export KOKKOS_TOOLS_LIBS=${YOUR_KOKKOS_TOOLS_LIB_SRC_DIR}/<name_of_tool_shared_lib>.so; ./yourApplication.exe`
 
-Many of the tools will produce an output file that uses the hostname as well as the process id as part of the filename. 
 
 ## Explicit Instrumentation
 
@@ -33,11 +59,14 @@ void foo() {
 The following provides an overview of the tools available in the set of Kokkos Tools. Click on each Kokkos Tools name to see more details about the tool via the Kokkos Tools Wiki. 
 
 ### Utilities
-
 + [**KernelFilter:**](https://github.com/kokkos/kokkos-tools/wiki/KernelFilter) 
 
     A tool which is used in conjunction with analysis tools, to restrict them to a subset of the application.
 
++ [**KernelSampler:**](https://github.com/kokkos/kokkos-tools/wiki/KernelSampler)
+
+   A tool to be used in conjunction with analysis tools to restrict the tooling to samples of Kokkos kernel invocations.
+
 ### Memory Analysis
 + [**MemoryHighWater:**](https://github.com/kokkos/kokkos-tools/wiki/MemoryHighWater)
 
@@ -85,30 +114,6 @@ The following provides an overview of the tools available in the set of Kokkos T
     accumulate statistics, and utilize various portable function calls for common needs w.r.t. timers,
     resource usage, etc. 
 
-# Building Kokkos Tools
-
-Use either CMake or Makefile to build Kokkos Tools. 
-
-## Using cmake
-
-1. create a build directory in Kokkos Tools, e.g., type `mkdir myBuild; cd myBuild` 
-2. To configure the Type `ccmake ..`  for any options you would like to enable/disable. 
-3. To compile, type `make`
-4. To install, type `make install`
-
-## Using make
-
-To build with make, simply type `make` within each subdirectory of Kokkos Tools. 
-
-
-Building using `make` is currently recommended. Eventually, the preferred method of building will be `cmake`.  
-
-# Running a Kokkos-based Application with a tool
-
-Given your tool shared library `<name_of_tool_shared_library>.so` (which contains kokkos profiling callback functions) and an application executable called yourApplication.exe, type: 
-
-`export KOKKOS_TOOLS_LIBS=${YOUR_KOKKOS_TOOLS_DIR}/<name_of_tool_shared_lib>; ./yourApplication.exe`  
-
 # Tutorial
 
 A tutorial on Kokkos Tools can be found here: https://github.com/kokkos/kokkos-tutorials/blob/main/LectureSeries/KokkosTutorial_07_Tools.pdf

build-all.sh

@@ -9,10 +9,9 @@ make -f $ROOT_DIR/profiling/memory-events/Makefile
 make -f $ROOT_DIR/profiling/memory-hwm/Makefile
 make -f $ROOT_DIR/profiling/memory-hwm-mpi/Makefile
 make -f $ROOT_DIR/profiling/memory-usage/Makefile
-make -f $ROOT_DIR/profiling/nvprof-connector/Makefile
-make -f $ROOT_DIR/profiling/nvprof-focused-connector/Makefile
+make -f $ROOT_DIR/profiling/nvtx-connector/Makefile
+make -f $ROOT_DIR/profiling/nvtx-focused-connector/Makefile
 make -f $ROOT_DIR/profiling/papi-connector/Makefile
-make -f $ROOT_DIR/profiling/simple-kernel-timer-json/Makefile
 make -f $ROOT_DIR/profiling/simple-kernel-timer/Makefile
 make -f $ROOT_DIR/profiling/space-time-stack/Makefile
 make -f $ROOT_DIR/profiling/systemtap-connector/Makefile

cmake/BuildGTest.cmake

@@ -0,0 +1,35 @@
+# Look for Google Test and enable it as a target.
+#
+# The main targets that will be available are:
+#       * GTest::gtest
+#       * GTest::gmock
+#
+# References:
+#   * https://github.com/google/googletest
+#   * https://matgomes.com/integrate-google-test-into-cmake/
+#   * https://google.github.io/googletest/quickstart-cmake.html
+#   * https://jeremimucha.com/2021/04/cmake-fetchcontent/
+
+include(FetchContent)
+
+# Declare the Google Test dependency
+FetchContent_Declare(
+    googletest
+    GIT_REPOSITORY https://github.com/google/googletest.git
+    GIT_TAG        v1.14.0
+)
+
+# If not yet populated, add Google Test to the build with the following options:
+#       * disable installation of Google Test
+#       * enable GMock
+# Note that we could have used FetchContent_MakeAvailable instead, but it would then
+# use the default configuration that would install Google Test.
+FetchContent_GetProperties(googletest)
+if (NOT googletest_POPULATED)
+    FetchContent_Populate(googletest)
+
+    set(BUILD_GMOCK   ON)
+    set(INSTALL_GTEST OFF)
+
+    add_subdirectory(${googletest_SOURCE_DIR} ${googletest_BINARY_DIR} EXCLUDE_FROM_ALL)
+endif()

cmake/configure_variorum.cmake

@@ -8,8 +8,9 @@ if(NOT DEFINED Variorum_ROOT)
   if(DEFINED ENV{VARIORUM_ROOT})
     set(Variorum_ROOT $ENV{VARIORUM_ROOT})
     set(MSG_NOTFOUND "check VARIORUM_ROOT environment variable ($ENV{VARIORUM_ROOT})")
+  else()
+    set(Variorum_ROOT ${VARIORUM_ROOT})
   endif()
-  set(Variorum_ROOT ${VARIORUM_ROOT})
 else()
   set(MSG_NOTFOUND "check Variorum_ROOT (${Variorum_ROOT})")
 endif()

common/kernel-filter/CMakeLists.txt

@@ -1 +1 @@
-kp_add_library(kp_kernel_filter ${KOKKOSTOOLS_LIBRARY_MODE} kp_kernel_filter.cpp)
+kp_add_library(kp_kernel_filter kp_kernel_filter.cpp)

common/kokkos-sampler/CMakeLists.txt

@@ -1 +1 @@
-add_library(kp_kokkos_sampler ${KOKKOSTOOLS_LIBRARY_MODE} kp_sampler_skip.cpp)
+kp_add_library(kp_kokkos_sampler kp_sampler_skip.cpp)

common/kokkos-sampler/Makefile

@@ -1,4 +1,4 @@
-CXX = clang++
+CXX = g++
 
 CXXFLAGS = -O3 -std=c++17 -g
 

common/kokkos-sampler/README.md

@@ -1,4 +1,11 @@
-This is a sampler utility that is intended to complement other tools in the Kokkos Tools set. This utility allows for sampling (rather than collecting) of profiling or debugging data gathered from a particular tool of the Kokkos Tools set. The Kokkos Tools user provides a sampling rate via the environment variable KOKKOS_TOOLS_SAMPLER_SKIP.  
+This is a sampler utility that is intended to complement other tools in the Kokkos Tools set. This utility allows for sampling (rather than collecting) of profiling or debugging data gathered from a particular tool of the Kokkos Tools set.
+
+To use this utility, a Kokkos Tools user provides a sampling probability by setting the environment variable `KOKKOS_TOOLS_SAMPLER_PROB` to a positive real number between 0.0 and 100.0. The user can alternatively set a sampling skip rate, i.e., the number of Kokkos kernel invocations to skip before the next sample is taken. The user does so by setting the environment variable `KOKKOS_TOOLS_SAMPLER_SKIP` to a non-negative integer.
+
+If both sampling probability and sampling skip rate are set by the user, this sampling utility only uses the sampling probability for sampling; the utility sets the sampling skip rate to 1, incorporating no pre-defined periodicity in sampling. If neither sampling probability nor the sampling skip rate are set by the user, then randomized sampling is done, with the sampler's probability being 10.0 percent. The sampler is periodic only if the sampling probability is not set by the user and the sampling skip rate is set by the user.
+
+For randomized sampling, the user can ensure reproducibility of this tool's output across multiple runs of a Kokkos application by setting `KOKKOS_TOOLS_RANDOM_SEED` to an integer value before all of the runs. If this environment variable is not set, the seed is based on the C time function.
 
 In order for the state of the sampled profiling and logging data in memory to be captured at the time of the utility's callback invocation, it might be important to enforce fences. However, this also means that there are more synchronization points compared with running the program without the tool.
-This fencing behavior can be controlled by setting the environment variable `KOKKOS_TOOLS_GLOBALFENCES`. A non-zero value implies global fences on invocation of the tool. The default is not to introduce extra fences.	
+This fencing behavior can be controlled by setting the environment variable `KOKKOS_TOOLS_GLOBALFENCES`. A non-zero value implies global fences on invocation of the tool. The default is not to introduce extra fences.
+