7. Investigate Performance Issues

7.1. Profiling Support on Linux for perf_event_paranoid Values

Following table describes profiling support on Linux for different perf_event_paranoid values:

7.2. Configure Profile

To perform a collect run, first you should configure the profile by specifying the profile configuration that identifies all the following information used to perform a collect measurement:

• Profile target

• Profile type: What profile data should be collected (CPU Profile, CPU Trace, GPU Trace, or Power Profile)

• Monitoring events: how the data should be collected

• Additional profile data (if needed): callstack samples, profile scheduling, and so on

7.2.1. Select Profile Target

To start a profile, either click the PROFILE page at the top navigation bar or Profile an Application? link in HOME page Welcome screen. The Start Profiling screen is displayed.

Select Profile Target is available in the Start Profiling window.

You can select the one of the following profile targets from the Select Profile Target drop-down:

• Application: Select this target when you want to launch an application and profile it (or launch and do a system-wide profile). The only compulsory option is a valid path to the executable. (By default, the path to the executable becomes the working directory unless you specify a path).

• System: Select this if you do not wish to launch any application but perform either a system-wide profile or profile specific set of cores.

• Process(es): Select this if you want to profile an application/process which is already running. This will bring up a process table which can be refreshed. Selecting any one of the processes from the table is mandatory to start profile.

Once profile target is selected and configured with valid data, the Next button will be enabled to go the next screen of Start Profiling.

7.2.2. Select Profile Type

Once profile target is selected and configured, click the Next button. The Select Profile Configuration screen is displayed as follows:

1. Select one of the following tabs:

   • Predefined Configs consists of all the predefined configurations, such as Time-based Profiling, Cache Analysis, and Assess Performance.

   • Live Power Profiling consists of options to perform real-time power profiling.

   • Custom Configs has options to perform Custom CPU Profile, CPU Tracing, and GPU Tracing.

2. Once you select a profile type, the left vertical pane within this window will list the options corresponding to the selected profile type. For CPU Profile type, all the available predefined sampling configurations will be listed.

   Modify event options are available only for the predefined configurations.

3. Click Advanced Options button to proceed to the Advanced Options screen and set the other options such as the Call Stack Options, Profile Scheduling, Sources, Symbols, and so on.

4. The details in the: Sample Data table are persistent and saved by the tool with a name (here, it is AMDuProf-EBP-ScimarkStable). You can define this name and navigate to PROFILE > Saved Configurations to reuse/select the same configuration later.

5. The Next and Previous buttons are available to navigate to various screen of the Start Profiling screen.

The CLI command is available at the bottom of this page, which displays the CLI version of the GUI option selected on the Select Profile Configuration page.

7.2.3. Advanced Options

Click the Advanced Options button in Select Profile Type screen, to set the advanced options for Windows/Linux and start the profiling.

Linux

Windows

You can set the following options on the Advanced Options screen and click Start Profile to begin profiling.

7.2.4. Start Profile

Once all the options are set correctly, click the Start Profile button to start the profile and collect the profile data. After the profile initialization the following screen is displayed.

The time elapsed during the data collection is displayed. When the profiling is in progress, click:

• Stop to stop the profiling.

• Cancel to cancel the profiling. It will take you back to Select Profile Target screen.

• Pause button to pause the profiling. The profile data will not be collected and you can click Resume to continue the profiling.

7.3. AMDuProf Overhead Estimation

The Overhead Estimation feature in AMDuProf provides insights into the overhead introduced during data collection and processing while profiling.

This capability helps users understand the impact of profiling on system performance and make informed decisions when selecting profiling configurations.

Supported and Unsupported Profiling Types

For unsupported configurations, a message stating no overhead data available for the selected configuration is displayed.

There are three categories of overheads:

• High Overhead

• Fair Overhead

• Low Overhead

There are two types of overheads:

• Collection Overhead: Refers to the time required to collect data during profiling. This time is influenced by the profiling configuration and system conditions.

• Processing Overhead: Refers to the time spent processing the collected data. This is impacted by various factors, including the presence of throttling events, sampling intervals, and sampling frequencies.

Before proceeding with the profiling experiment, AMDuProf provides a suggestion regarding the overhead based on the current configuration. This suggestion helps users decide whether to proceed with the current configuration or modify it based on the expected overhead. There are multiple factors influencing overheads.

1. Predefined Configurations: In predefined configurations, certain events such as overview and threading have a high overhead in both data collection and processing.

2. Throttling Events: If there are one or more throttling events present in either predefined or custom configurations, the processing overhead will be high, and the collection overhead will be Fair.

3. Sampling Intervals and Frequencies:

   1. Threshold for Sampling Intervals/Frequencies: A threshold exists for sampling intervals and frequencies.

   2. Higher Sampling Interval/Lower Sampling Frequency: Increases collection and processing overhead.

   3. Lower Sampling Interval/Higher Sampling Frequency: Decreases collection and processing overhead.

   This balance affects the efficiency of data collection and processing.

   In addition to overhead information, you get an estimate of the profiling time based on the configuration and overhead type.

   You can evaluate the overhead suggestion and decide whether to proceed with the current configuration or modify it for optimal performance. This guidance helps balance the desired profiling data and the overhead it may incur on the system.

   For configurations not supported by the overhead feature, a message stating that no overhead data available for the selected configuration will be displayed.

7.4. Analyze Profile Data

When the profiling stopped, the collected raw profile data will be processed automatically and you can analyze the profile data using the following UI sections to identify the potential performance bottlenecks:

The sections available depends on the profile type. The CPU Profile will have SUMMARY, ANALYZE, MEMORY, HPC, and SOURCES pages to analyze the data.

• SUMMARY page to look at overview of the hotspots for the profile session.

• ANALYZE page to examine the profile data at various granularities.

• SOURCES page to examine the data at source line and assembly level.

• MEMORY page to examine the cache-line data for potential false cache sharing.

• HPC page to examine the OpenMP tracing data for potential load imbalance issue and visualize MPI API trace, OS event trace.

7.4.1. Overview of Performance Hotspots

When the translation is complete, the SUMMARY page will be populated with the profile data and Hot Spots screen will be displayed. The SUMMARY page provides an overview of the hotspots for the profile session through various screens such as Hot Spots and Session Information.

In the Hot Spots screen, hotspots will be displayed for functions, modules, process, and threads. Processes and threads will be displayed only if there are more than one.

The following figure shows the Hot Spots screen:

In the above Hot Spots screen:

• The top 5 hottest functions, processes, modules and threads for the selected event are displayed.

• The Hot Functions pie chart is interactive in nature. You can click on any section and the corresponding function’s source will open in a separate tab in the SOURCES page.

• The hotspots are shown per event and the monitored event can be selected from drop-down in the top-right corner. You can change it to any other event to update the corresponding hotspot data.

• From the Select Summary View drop-down, select one of the following:

  • Hot Threads

  • Hot Processes

  • Hot Functions

  • Hot Modules

Based on the selection, one donut is displayed at a time.

7.4.1.1. Summary Overview

7.4.1.2. OS Trace Screen

7.4.1.3. GPU Trace Screen

7.4.1.4. MPI Trace Screen

7.4.1.5. CPU Profile

The CPU Profile screen is similar to the Summary - Hotspots Screen.

7.4.2. Thread Concurrency Graph

Click ANALYZE > Thread Concurrency to view the following graph to analyze the thread concurrency of the profiled application.

The thread concurrency graph displays the duration (in seconds) of the specific number of threads that were running simultaneously.

Bucketization approach is used for this graph. Instead of showing the Elapsed Time for each core, the weighted average based on the bucket size will be taken. The bucket size will be determined based on the cores and number of available pixels available. This is done to avoid the horizontal scrolling.

7.4.3. Function HotSpots

Click ANALYZE on the top horizontal navigation bar to go to Function Hotspots screen, which displays the hot functions across all the profiled processes and load modules as follows. You can view the following:

Process and thread wise breakdown of data is available if the entries are expanded in Function Hotspots View. The Functions table lists the hot functions. The IP samples are aggregated and attributed at the function-level granularity. On the table, you can do the following:

• Double-click on a function entry to navigate to the corresponding SOURCE view of that function.

• Right-click to view the following options: - Copy selected row(s) to copy the highlighted row to clipboard. - Copy all rows to copy all the rows to clipboard.

Filters and Options pane allows you filter the profile data as follows:

• You can click the Select View drop-down to control the counters that are displayed. The relevant counters and their derived metrics are grouped in predefined views.

• You can use the Value Type drop-down to display the counter values as follows: - Sample Count is the number of samples attributed to a function. - Event Count is the product of sample count and sampling interval. - Percentage is the percentage of samples collected for a function.

• You can use the System Modules option to either Exclude or Include the profile data attributed to system modules.

If callstack is enabled, the unique hot call-paths for the selected function is displayed in the Functions column.

Event Timeline is the line graph showing the number of aggregated sample values over the period of time. You can use it to identify the hot functions within a profile region. From the Select Metric drop-down you can select the event for which event timeline must be plotted.

All the entries will not be loaded for a profile. To load more than the default number of entries, click the vertical scroll bar on the right. When the entries are expanded, process and thread-wise breakdown of data is available.

7.4.4. Process and Functions

Click ANALYZE > Grouped Metrics to display the profile data table at various program unit granularities - Process, Load Modules, Threads, and Functions. This screen contains data in two different formats as follows:

The upper tree represents samples grouped by Process. You can expand the tree to view the child entries for each parent (that is for a process). The Load Modules and Threads are child entries for the selected process entry.

You can right-click to view the following options:

• Expand All Entries to list the modules and threads of all the processes.

• Collapse All Entries to list only the top-level entries.

• Copy selected row(s) to copy the highlighted row to clipboard.

• Copy all rows to copy all the rows to clipboard.

The lower Functions table contains samples attributed to corresponding functions. The function entries depend on what is selected in the upper tree. For more specific data, you can select a child entry from the upper tree and the corresponding function data will be updated in the lower tree. You can do any of the following:

• Double-click on a function entry to navigate to the corresponding SOURCE view.

• Right-click to view the following options:

  • Copy selected row(s) to copy the highlighted row to clipboard.

  • Copy all rows to copy all the rows to clipboard.

You can use the Filters and Options pane to filter the profile data displayed by various controls.

• The Select View controls the counters that are displayed. The relevant counters and their derived metrics are grouped in predefined views. You can select the views from the Select View drop-down.

• The Group By drop-down is used to group the data by Process, Module, and Thread. By default, the sample data is grouped-by Process.

• Click the ValueType drop-down to display the counter values as follows: - Sample Count is the number of samples attributed to a function. - Event Count is the product of sample count and sampling interval. - Percentage is the percentage of samples collected for a function.

You can use the System Modules option to Exclude or Include the profile data attributed to system modules.

Confidence level

The metrics that cannot be calculated reliably due to low number of samples collected for a program unit will be grayed out.

All entries will not be loaded for a profile. To load more than the default number of entries, click the vertical scroll bar on the right.

7.4.5. Source and Assembly

Double-click on any entry in the Functions table in the Metrics screen to load the source tab for the corresponding function in SOURCES page. If the GUI can find the path to the source file for that function, then it will try to open the file, failing which you will be prompted to locate it.

The following figure depicts the source and assembly screen.

The following sections are present in the SOURCES screen:

7.4.6. Top-down Callstack

Top-down Callstack view can be used to explore the call-sequence flow of the application to analyze the time spent in functions and its callees.

Click ANALYZE > Top-down Callstack to view it as follows:

Functions are displayed based on the parent to child entries depending on the inclusive samples values sorted.

Inclusive sample values for a function and its descendants.

Enabling Hide C++ std Library Calls option works only when C++ library calls are made. It will exclude such calls from the list and display the other child entries.

Context menu of collapse entries will close all the expanded entries. Expand entries will expand the child entries and the Open Source View option will display the corresponding source view.

7.4.7. Flame Graph

Flame graph is a visualization of sampled call-stack traces to quickly identify the hottest code execution paths. Click ANALYZE > Flame Graph to view it as follows:

The x-axis of the flame graph shows the call-stack profile and the y-axis shows the stack depth. It is not plotted based on passage of time. Each cell represents a stack frame and if a frame were present more often in the call-stack samples, the cell would be wider. This screen has the following options:

• Module-wise coloring of the cells.

• Click on a cell to zoom only that cell and its children. Use the Reset Zoom button visualize the entire graph.

• Right-click on a cell to view the following context options:

  • Copy Function Data to copy the function names and its metrics to clipboard.

  • Open Source View to navigate to the source tab of that function.

• Hover the mouse over a cell to display the tool-tip showing the inclusive and exclusive number of samples of that function.

Click the Zoom Graph button for a better zooming experience.

When you type a function name in the search box, a list of all the relevant matches will be displayed. Select the required function to highlight the cells corresponding to that function in the flame graph.

The Process drop-down lists all the processes for which call-stack samples are collected. Changing the process will plot the flame graph for that particular process.

For multi-threaded applications, the flame graph will be plotted for the cumulative data of all the threads by default.

The Threads drop-down lists all the threads for which call-stack samples are collected. Changing the thread will plot the flame graph for that thread.

The Select Metric drop-down lists all the metrics for which call-stack samples are collected. Changing the metric will plot the flame graph for that particular metric.

7.4.8. Call Graph

Click ANALYZE > Call Graph* to navigate to the call graph screen. This graph is constructed using the call-stack samples and offers a butterfly view to analyze the hot call-paths as follows:

The Function table lists all the functions with inclusive and exclusive samples. Click on function to display its Caller and Callee functions in a butterfly view. In addition the parents and children of the selected function in the Function table are displayed.

Options

• The Process drop-down lists all the processes for which call-stack samples are collected. Changing the process will show the call graph for that particular process.

• For multi-threaded applications, the call-graph will be plotted for the cumulative data of all the threads by default.

• The Threads drop-down lists all the threads for which call-stack samples are collected. Changing the thread will plot the call graph for that thread.

• The Select Metric drop-down lists the metrics for which call-stack samples are collected. Changing the counter will show the call graph for that particular counter.

7.4.9. All Thread Timeline

7.4.9.1. Timeline Analysis GUI in Linux

To configure threading analysis from the GUI:

1. Navigate to the Select Profile Configuration screen.

2. Select Predefined Configs from the tab.

3. Select Threading Analysis from the left vertical pane.

Profile data collected from CLI or GUI can be visualized in GUI by importing the session. On importing, the following section (Thread Timeline) is displayed on the ANALYZE page.

Time-series data is plotted in timelines per entity (thread, rank, device, and so on). Trace data (if collected) will only be plotted when you zoom into the timeline to address data size related scalability issues (trace data can have millions of records which will not be visually legible if plotted together). The entire view is broadly separated in three vertical parts, top data selectors, middle timelines, and bottom filters. You can use the timeline as follows:

• Hover the cursor over a timeline to view a vertical line containing the tool-tip for a specific entity, showing relevant details, and the current timestamp.

• If CPU profile data is collected, click and drag the mouse over the timeline to select a region across all timelines and brings up the Function Hotspot within the selected time range.

• Zoom-in/out horizontally into the timelines using one of the following - The mouse wheel - Pressing CTRL and +/- keys on the keyboard to zoom-in or zoom-out, respectively

The timeline section consists of:

1. Name of each thread in timeline with Thread ID.

2. Click Load More button which loads more threads. By default, only a small number of thread timelines are loaded to limit the resource consumption. This button enables loading the next set of thread timelines. The next set is determined by the entries in the table below the timeline.

3. Select the Data Source drop-down to enable selection of data to display on the timeline. Different types of data source are as follows:

   • CPU Utilization: Displays a timeline of CPU utilization (in percentage) per thread. To collect sufficient such data points, the total profile duration should be greater than or equal to 5 seconds. In Threading analysis configuration, this data is collected at fine-grained intervals in milliseconds, whereas for other configurations, it is collected at a per second interval.

   • Memory Consumption: Plots the timeline for the memory consumption (in MB) categorized as physical and virtual memory consumed. This is enabled only for the Threading Analysis configuration.

   • Context Switches: Plots the timeline for both voluntary context switches count (sleep, yield, and so on) or involuntary context switches count (OS scheduler triggered context switch). This is enabled only for the Threading Analysis configuration.

   • CPU Profile Samples: Plots the timeline for the CPU sample collected for the CPU events. The CPU samples are plotted as a heatmap (with colors ranging from dark-green (highest samples in a window) to yellow-green (lowest samples in a window) with gray regions depicting where there are no samples collected). The following events are supported:

     Table 7.7 Supported CPU Events#

     Events	Availability
     CPU Time	Time-based profiling is performed.
     Cycles not in Halt	PMC event CYCLES_NOT_IN_HALT is collected.
     Op Cycles	IBS op event is collected with count cycles unit mask.
     Retired Instructions	PMC event RETIRED_INSTRUCTIONS is collected.

   • GPU Related Counts: GPU Kernel Time, Copy Time and HIP API Time are also plotted to provide time spent in GPU kernels, GPU memory copy operations, and time spent inside HIP APIs, respectively.

   • Thread Trace: Plots the timeline based on OS trace data which can either originate from eBPF Tracing or User-mode Tracing. The trace data is categorized and aggregated at certain intervals to generate time-series plotted in timelines. The following categories are created:

     Table 7.8 Thread Time Categories#

     Category	Description
     Event Wait Time	Total time spent on IO Multiplexing (poll, select etc.), wait for process / thread to finish (wait, pthread_join etc.).
     I/O Sync Time	Total time spent on IO sync APIs (sync, fsync etc.).
     I/O Time	Total Time spent in I/O syscalls, that is, read, write, pread, pwrite, and so on.
     Pause Time	Total time spent on profile paused state. User can pause the profile collection with profile control APIs and GUI pause after starting the profile collection.
     Resource Wait Time	Total time spent on synchronization objects, reader and writer locks, thread barrier etc.
     Running Time	Total active processor time, which includes the time spent in IO operations and spin lock operations.
     Sleep Time	Total time spent on sleep and waiting for signal delivery system calls.
     Spin Time	Total time spent on spin lock.

4. The Select Trace Overlay drop-down enables selection of the type of trace data to display.

   • Thread State: Shows the current state of thread from eBPF or User-mode tracing. In the former, thread state is inferred from BPF data. In the latter, thread state is treated as Running if Running Time > 0, otherwise, Sleeping.

   • Thread Trace: Displays traces for the traced libpthread functions, such as pthread_mutex_lock, pthread_mutex_trylock, and so on.

   • Syscalls: Displays traces for traced syscall in the specific region of the timeline.

5. Trace Cutoff can be used to specify a duration in nanoseconds, which acts as a cutoff to load the trace data, that is, any traced function which takes less than the specified nanoseconds will not be displayed.

6. Click the Reset Zoom button to reset any zoom performed earlier.

7. Hover over any timeline to view the tool-tip containing the relevant data along with timestamp. If trace data is also present, the relevant traced functions with start time and duration.

8. Filter Threads/Ranks enables you to filter which thread’s (or rank’s) timelines must be displayed. By default, the timelines are sorted internally and the first 6 are loaded. However, from the table, you can select the required threads and clicking Apply Filter to apply the changes. If CPU profile data is collected, highlighting functions or modules is also possible. Each function is assigned a random color, which can be modified and highlighted in the timeline (implies there are samples from the function/module).

9. Each entry in the filter table has the necessary data, that is, name, parent object, and samples/trace times aggregated across the profile.

10. Click the Apply Filter button to apply a custom selection of entities or highlight entities in timeline. (If GPU acceleration is available, there is no need to click Apply as the changes are reflected instantaneously)

11. Click Deselect selected Items to deselect all the entries in the filtering table except the first one. This is useful when a custom selection is required but all timelines are already loaded.

12. At the bottom of the filtering pane, timeline legend is displayed, which helps in identifying how each type of data source or trace is mapped to which color.

13. The Show Core Transition button is disabled by default and works only when the CPU profiling data is collected. When enabled, a red line is displayed in each timeline to signify when a thread changes the core.

When you enable CPU profiling (along with other data sources), you can highlight functions and modules in the timeline across threads. The tool lists them in tables under the Filtering Threads option, ordered by CPU sample data. You can select multiple functions or modules to highlight, and they appear as overlays in the timeline view. You can also change the color for each function or module, and the overlay updates accordingly.

Highlight tasks if the profiled application is instrumented using the Instrumentation API. The timeline displays tasks across threads, and you can select them from the Highlight Tasks tab. This tab presents task data grouped by domain and sorted by the total time spent across all task instances. To focus the view, click the Show only associated Threads button to display only those threads that contain at least one instance of the selected tasks.

7.4.9.2. Region Selection

When you select a region in the timeline view by clicking and dragging with the mouse, uProf generates aggregate data for that region. Depending on the configuration and collected data, the following types of aggregate data is displayed:

1. Function Hotspot – visible only if CPU samples are collected

2. Flame Graph – visible only if CPU samples are collected

3. Wait Object Hotspot – visible only when using the Threading Analysis configuration

7.4.10. Per Thread Timeline

Per Thread timeline view focuses on showing all aspects of a specific thread based on the collected data. Hence it can show CPU profile samples, OS traces, GPU traces and System metrics at per thread level. The selection table at the bottom pane sorts them by the first event in the table, and threads can be switched from the table. Function/Module highlights work in the same way as All thread timeline.

Each per-thread timeline displays two types of flame graphs:

• A callstack flame graph: shows how the call stack evolves over time for a specific thread.

• A task flame graph: appears when the profiled application uses the Instrumentation API. This graph visualizes the evolution of tasks per thread. As the tasks can be nested, the flame graph may resemble a call stack when such nesting occurs.

The callstack flame graph is only available for the following profile configurations:

• Native

  • Hotspots + CSS

  • Overview

  • TBP + Function trace

  • EBP + Frequency + Function trace

  • IBS + Function trace

  • Custom config + Function trace

• Java

  • Hotspots + CSS

• Python

  • Hotspots + CSS

If GPU tracing data is also collected for applications using HIP APIs, this view also plots the GPU utilization, GPU Memory, and GPU power for all the GPUs to which kernels were scheduled from current thread. As a single thread can schedule kernels on multiple GPUs, one line is plotted for each such device, the device info being present in the tooltip.

7.4.11. Notes on GPU Acceleration

GPU acceleration is only available if OpenGL drivers exist on the system. This applies to both Windows and Linux. If not, the tool will automatically fallback to CPU implementation, which will not be as performant. (This is also the case when using remote X11 servers i.e. launching the Linux UI in Windows with MobaXTerm-like tools). The minimum expected OpenGL version is 3.1 on both Linux and Windows. While AMDuProf tries to detect the version automatically, should this fail due to unforeseen scenarios, where the tool falls back to CPU rendering, but OpenGL (>= 3.1) is still available, environment variables could be used to tweak the behavior, as listed in the following table.

7.4.12. IMIX View

IMIX view shows the summary of instruction-wise samples collected. This view is shown only for IBS profiling. Click ANALYZE > IMIX to navigate to the IMIX view.

7.4.13. Wait Object Hotspots

Wait Object Hotspots view shows the wait object related data in detail. Different groupings are also available for in depth analysis. It can be broken down by expanding in different levels. Click ANALYZE > Wait Object Hotspots to navigate to the Wait Object Hotspots view. For more information refer to Wait Time Analysis.

7.5. Hotspots Analysis

Hotspots Analysis is the starting point for algorithm analysis of an application. Use Hotspots Analysis to understand the application code flow and sections of code which has lot of execution time (CPU Time).

7.5.1. User Mode Sampling

User mode sampling embeds an agent library into application address space using LD_PRELOAD. The agent creates a per thread OS timer (default timer interval is 10 ms), interrupts the execution of a thread by generating SIGPROF or another runtime signal. Once thread receives the signal, the agent collects IP samples, and it’s callstack for each sample if callstack collection is enabled from signal handler. Collected data is stored in binary files for later processing.

7.5.2. Callstack Stitching

In OpenMP applications, if a parallel region is executed by multiple threads (master and worker threads), each worker thread will have its own calling sequence (call path or callstack) which logically starts when the master thread encountered the parallel region. During translation, AMDuProf will stitch the worker thread’s call path to master thread call path at the point where the parallel region started, if the worker threads are active in the parallel region. This allows runtimes from the worker threads to be attributed to the correct logical calling sequence of the program (i.e. calling sequence without OpenMP) so that uProf can produce accurate flame graphs.

By default, AMDuProf detects the compiler used to build the application by reading the .comment section and stitches the worker threads call path with master thread. If AMDuProf fails to stitch the call path with master thread, select OpenMP implementation type as omplib for GCC compiled applications and ompt for AOCC, ICC, and LLVM compiled applications.

7.5.3. Data Collection

7.5.3.1. Data Collection Using GUI

To launch the AMDuProf GUI, go to Home > Welcome page.

1. Click Profile an Application on the Welcome page.

2. Provide the application path, application options, working directory, and environment variables, if any. Click Next.

3. From Predefined Configs, select Hotspots.

4. Set the timer interval* and profiling signal.

5. From Advanced Options, select the OpenMP implementation type, callstack collection, and callstack unwind depth.

6. Click Start Profile to start the profiling.

7.5.3.2. Data Collection Using CLI

• Hotspots collection

  AMDuProfCLI collect --config hotspots -o <output-dir> <application>
  

• Hotspots with callstack collection

  AMDuProfCLI collect --config hotspots -g -o <output-dir> <application>
  

• Hotspots with timer interval of 100msec

  AMDuProfCLI collect --config hotspots --timer-interval 100 -o <output-dir> <application>
  

• Hotspots with profiling signal as SIGRTMIN

  AMDuProfCLI collect --config hotspots --profiling-signal 34 -o <output-dir> <application>
  

• Hotspots with callstack collection for GCC compiled OpenMP application

  AMDuProfCLI collect --config hotspots -g --openmp-impl omplib -o <output-dir> <application>
  

• Hotspots with callstack collection

  AMDuProfCLI collect --config hotspots -g --call-graph-depth 128 -o <output-dir> <application>
  

Once profile data collection completes, session directory will be generated. Use session directory to generate the csv report (or) to import the session in GUI. Refer AMDuProfCLI Collect Command Options.

Example

./AMDuProfCLI collect --config hotspots -g -o /tmp/ /tmp/ScimarkStable
Profiling started
………
Profiling (data collection) completed
Generated data files path: /tmp/AMDuProf-ScimarkStable-Hotspots_Sep-05-2024_21-43-08.

Here, the generated session directory is /tmp/AMDuProf-ScimarkStable-Hotspots_Sep-05-2024_21-43-08.

Hotspot Analysis Options

7.5.4. Analyze the Data

If data is collected using CLI, use Import Session to import the session into GUI to analyze data in GUI.

7.5.4.1. CLI Report

Use the following CLI report command to generate the profile report in .csv format by passing the session directory path generated in collection.

AMDuProfCLI report -i <session directory>

See table AMDuProfCLI Report Command Options for a list of all the supported options.

Example

./AMDuProfCLI report -i /tmp/AMDuProf-ScimarkStable-Hotspots_Sep-05-2024_21-43-08
Translation started
…
Report generation started
…
Report generation completed...
Generated report file: /tmp/AMDuProf-ScimarkStable-Hotspots_Sep-05-2024_21-43-08/report.csv

Use the Thread Concurrency Graph to analyze how efficiently the processor cores are utilized by the application. In other words, how much time specific no of threads are running on specific no of cores.

7.5.4.2. Identify the Hottest Function

Use Function HotSpots to get list of most CPU time (self-time and children time) consuming functions, expand the function to get its processes and further expand to get its threads. All the functions are sorted in descending order of CPU time.

Select a function to get all the call paths to this function and total CPU time consumed in every call path.

Double click on the function to analyze the instruction level sample attribution for that function using the Source View. See Source and Assembly.

7.5.4.3. Identify the Hot Code Paths

Use Flame Graph to identify hottest code paths of an application. The width of each functions indicates the percentage of CPU time of the function (it’s callees) to the total CPU time of selected process and thread.

Use Top-down Callstack to analyze any issues with call-sequence flow of the application and to analyze the total CPU time spent in functions and its callees.

7.5.5. Troubleshoot

• In Top-Down Callstack or Flame Graph, if ROOT node branches out to more than two nodes then increase callstack unwind depth with --call-graph-depth. Maximum depth supported is 1024.

• If worker threads parallel region callstack is not stitched with master thread, and if application is creating more than 1000000 parallel regions, increase this limit by setting the environment variable AMDUPROF_MAX_PR_INSTANCES.

$export AMDUPROF_MAX_PR_INSTANCES=2000000

7.5.6. Limitations

• MPI tracing and Openmp tracing are not supported with Hotspot analysis.

• In cluster environment, if two or more threads/processes have same thread id/process id then behavior is undefined.

• If the leaf function is highly optimized assembly code, then AMDuProf fails to find its callstack.

• This profile type cannot be combined with other pre-defined configurations.

• Analyzing 32-bit applications is not supported.

• Hotspots config does not report samples before the exec call for an application and samples belonging to vforked process for Hotspots.

• Linux

  • To report the proper callstack in case of system wide profiling, timer interval less than 10msec, attach process (or) thread profiling and launching static executable (application is built with -static) from uProf then make sure application, and its dependent libraries are compiled with frame pointers.

  • Callstack stitching is not supported in system wide profiling, attach process (or) thread profiling and if launched application is static executable (application is built with -static).

  • If launched application uses clone () system call directly to create a thread (or) process, then behavior is un-defined.

  • Behavior is undefined if --no-inherit (not profile the child processes) is used with hotspots analysis.

• Windows

  • To report the proper callstack compile the application and its dependent libraries with frame pointers.

7.6. Threading Analysis

Use Threading Analysis to identify how efficiently an application uses the processor cores, contention among the application threads due to synchronization, CPU utilization of the threads, Wait time analysis of the application threads.

Threading Analysis uses the User mode sampling and tracing approach. Threading analysis is supported only in Linux and if application is using libc and libpthread then these libraries should be linked dynamically.

Reference

• User Mode Sampling

• Callstack Stitching

7.6.1. User Mode Tracing

User Mode Tracing embeds an agent library into application address space using LD_PRELOAD. It interposes the pthread APIs and a few system calls, collects the start time and end time of an API, and a few other metrics.

System wide performance data and already running process/thread performance data collection is not supported with user mode sampling and tracing.

7.6.2. Data Collection

7.6.2.1. Data Collection Using GUI

To launch the AMDuProf GUI, go to Home > Welcome page.

1. Click Profile an Application on the Welcome page.

2. Provide application path, application options, working directory, and environment variables, if any. Click Next.

3. From Predefined Configs, select Threading Analysis.

4. Set the Timer Interval and Profiling Signal.

5. From Advanced Options, select the OpenMP implementation type, callstack collection, and callstack unwind depth.

6. Click Start Profile to start the profiling.

7.6.2.2. Data Collection Using CLI

• Threading collection

  AMDuProfCLI collect --config threading -o <output-dir> <application>
  

• Threading with callstack collection

  AMDuProfCLI collect --config threading -g -o <output-dir> <application>
  

• Threading with timer interval of 100 msec

  AMDuProfCLI collect --config threading --timer-interval 100 -o <output-dir> <application>
  

• Threading with profiling signal as SIGRTMIN

  AMDuProfCLI collect --config threading --profiling-signal 34 -o <output-dir> <application>
  

• Threading with callstack collection for GCC compiled OpenMP application

  AMDuProfCLI collect --config threading -g --openmp-impl omplib -o <output-dir> <application>
  

• Threading with callstack collection and unwind depth as 128

  AMDuProfCLI collect --config threading -g --call-graph-depth 128 -o <output-dir> <application>
  

• Threading with pthread APIs threshold of 1000000 nsec

  AMDuProfCLI collect --config threading --osrt-threshold pthread:1000000 -o <output-dir> <application>
  

• Threading to collect the callstack for system modules (standard C++ library to it’s implementation). By default, AMDuProf shows the callstack until the standard library.

  AMDuProfCLI collect --config threading -g --collect-sys-modules -o <output-dir> <application>
  

Once profile data collection completes, session directory will be generated. Use session directory to generate the .csv report (or) to import the session in GUI.

For a list of all the supported options, refer to AMDuProfCLI Collect Command Options.

Example

./AMDuProfCLI collect --config threading -g -o /tmp/ /tmp/ScimarkStable
Profiling started
…
Profiling (data collection) completed
Generated data files path: /tmp/AMDuProf-ScimarkStable-Threading_Sep-05-2024_21-48-42

Here, the generated session directory is /tmp/AMDuProf-ScimarkStable-Threading_Sep-05-2024_21-48-42.

System Call Tracing

By default, Threading analysis traces sleep and wait system calls. Configure system call tracing with threading analysis to trace all supported system calls which includes IO system calls, blocking system calls.

CLI command to collect the system call trace data with threading analysis.

AMDuProfCLI collect --config threading --trace osrt --osrt-event syscall -o <output-dir> <application>

Threading Analysis options

7.6.3. Analyze the Data

If data is collected using CLI, use Import Session to import the session into GUI to analyze data in GUI.

7.6.3.1. CLI Report

Use the following CLI report command to generate the profile report in .csv format by passing the session directory path generated in collection.

AMDuProfCLI report -i <session directory>

For a list of all the supported options, refer to AMDuProfCLI Report Command Options.

Example

./AMDuProfCLI report -i /tmp/AMDuProf-ScimarkStable-Threading_Sep-05-2024_21-48-42
Translation started
…
Report generation started
…
Report generation completed...
Generated report file: /tmp/AMDuProf-ScimarkStable-Threading_Sep-05-2024_21-48-42/report.csv

Reference

• Identify the Hottest Function

• Identify the Hot Code Paths

7.6.3.2. Threading Summary

Threading Summary provides high level performance snapshot of an application with respect to different timing details.

• Elapsed Time is wall clock time from application execution start time to end time.

• Total Time is total time of all the threads created by an application.

• CPU Time is total active processor time of all threads which includes the time spent in IO operations and spin lock operations. To optimize the code if no of threads are increased, then elapsed time may decrease whereas CPU time might increase.

  • IO Time is total time spent by all the threads in IO system calls except IO sync calls.

  • Spin Time is total time spent by all the threads in spin lock.

  When application calls IO system calls and blocking system calls (except IO sync calls), there is no guarantee that application will be blocked or not. So these times are added to total CPU time.

• Sleep Time is total time spent by all threads on sleep and waiting for signal delivery system calls.

• Event Wait Time is total time spent by all threads on IO Multiplexing (poll, select etc.), wait for process / thread to finish (wait, pthread_join etc.).

• Resource Wait Time is total time spent by all threads on synchronization objects, reader and writer locks, thread barrier etc.

• IO Sync Time is total time spent by all threads on IO sync APIs (sync, fsync etc.).

• Pause Time is total time spent by all threads on profile paused state. User can pause the profile collection with profile control APIs and GUI pause after starting the profile collection.

Refer Posix Thread APIs and libc System Call Wrapper APIs to know the APIs traced with threading analysis.

7.6.3.3. Wait Time Analysis

High wait time means application suffers with parallel performance, use Thread Summary to analyze per thread total run time, wait time and wait time percentage of the thread from total time of the thread. If a thread is optimized, then it’s wait time and percentage wait time might reduce when compare before and after optimization. It helps to identify whether a thread is using the core effectively or not.

Use Wait Object Summary to identify performance critical synchronization object which has more amount of wait time and wait count.

Syscall Summary provides the system call count, total time spent by the application on a system call. It helps to identify the system calls consuming most of the time and that can be optimized if the system calls are blocking or waiting in nature.

By default, Wait Object Hotspots ranks functions or processes according to their total wait time, which is the time spent blocked on locks, events, semaphores, I/O, and similar objects. You can also sort the results based on the wait count or the percentage of wait time relative to the total execution time. To focus your analysis on a specific period, you can select a time range from the top timeline. For any identified hotspot, you can expand the entry to view the associated wait object(s), the callsite, the full call stack, and—if debug information is available—the corresponding source file and line number.

There are four groupings available. User can select any one of those based on the requirements.

• Process → Function → Wait Object → Callsite → Callstack

• Wait Object → Process → Thread → Function → Callstack

• Task → Wait Object → Process → Thread → Function → Callstack

• Process → Wait Object → Callsite → Callstack

7.6.3.4. Timeline

Per Thread Timeline displays the selected thread’s state, CPU utilization, context switch count, running callstacks and many metrics over the time. When you select a time region, the view reveals the thread’s activity and presents an aggregated flamegraph that summarizes function calls within that interval.

7.6.4. Troubleshoot

• In Top-Down Callstack or Flame Graph, if the ROOT node branches out to more than two nodes then increase callstack unwind depth with --call-graph-depth. Maximum depth supported is 1024.

• If worker threads parallel region callstack is not stitched with master thread, and if application is creating more than 1000000 parallel regions, increase this limit by setting the environment variable AMDUPROF_MAX_PR_INSTANCES.

$export AMDUPROF_MAX_PR_INSTANCES=2000000

7.6.5. Limitations

• MPI tracing and OpenMP tracing are not supported with Threading config.

• System wide profiling data collection and already running process / thread profiling data collection is not supported.

• Threading Analysis is not supported if application is static executable (application is built with -static ) as LD_PRELOAD is ignored.

• If application uses clone system call directly to create a thread (or) process, then behavior is undefined.

• In cluster environment, if two or more threads/processes has same thread id/process id then behavior is undefined.

• If leaf function is highly optimized assembly code, then AMDuProf fails to find its callstack.

• Threading Analysis is not supported with AMD Zen1 and AMD Zen2 architectures.

• Analyzing the 32-bit applications is not supported.

• Behavior is undefined if --no-inherit (not profile the child processes) is used with threading analysis.

7.6.6. Reference

7.6.6.1. Posix Thread APIs

Resource Wait Time APIs

• pthread_mutex_lock

• pthread_mutex_trylock

• pthread_mutex_timedlock

• pthread_barrier_wait

• pthread_cond_wait

• pthread_cond_timedwait

• pthread_rwlock_rdlock

• pthread_rwlock_tryrdlock

• pthread_rwlock_timedrdlock

• pthread_rwlock_wrlock

• pthread_rwlock_trywrlock

• pthread_rwlock_timedwrlock

• sem_wait

• sem_trywait

• sem_timedwait

Event Wait Time APIs

• pthread_join

Spin Time APIs

• pthread_spin_lock

• pthread_spin_trylock

Other APIs

pthread_create pthread_exit pthread_cancel

7.6.6.2. libc System Call Wrapper APIs

List of libc APIs traced with threading config.

Sleep APIs

• sleep

• nanosleep

• clock_nanosleep

• usleep

• pause

• sigsuspend

• sigwait

• sigwaitinfo

• sigtimedwait

Event Wait Time APIs

• poll

• ppoll

• select

• pselect

• epoll_wait

• epoll_pwait

• wait

• waitpid

• waitid

• wait3

• wait4

Resource Wait Time APIs

• futex

• flock

• lock

IO Sync Time APIs

• fsync

• sync

• syncfs

• fdatasync

• sync_file_range

IO Time APIs

• create

• open

• openat

• read

• pread

• readv

• preadv

• preadv2

• write

• pwrite

• writev

• pwritev

• pwritev2

• lseek

• sendfile

• copy_file_range

• truncate

• ftruncate

• readahead

• close

Other APIs

• accept

• accept4

• recv

• recvfrom

• recvmsg

• recvmmsg

• send

• sendto

• sendmsg

• sendmmsg

• mq_send

• mq_timedsend

• mq_receive

• mq_timedreceive

• msgsnd

• msgrcv

• semget

• semop

• semtimedop

• semctl

• splice

• vmsplice

• msync

• fcntl

• ioctl

• epoll_create

• epoll_create1

• epoll_ctl

• socket

• bind

• listen

• connect

• socketpair

• mq_notify

• mq_getattr

• mq_setattr

• mq_close

• mq_unlink

• msgget

• msgctl

• pipe

• pipe2

• shmat

• shmctl

• shmget

• shmdt

• fork

• vfork

• alarm

• system

• kill

• killpg

• brk

• sbrk

• mlock

• munlock

• mlock2

• mlockall

• munlockall

• mmap

• munmap

• move_pages

• mprotect

• mremap

• process_vm_readv

• process_vm_writev

• acct

• chroot

• dup

• dup2

• dup3

• fallocate

• ioperm

• iopl

• mount

• prctl

• ptrace

• sigaction

• swapon

• swapoff

• tee

• umount

• umount2

• unshare

• vhangup

7.7. Overview Analysis

Use Overview Analysis to get high level performance snapshot of an application, identify hottest functions and it’s inclusive and exclusive elapsed times, CPU utilization of the threads, and Wait time analysis of the application threads.

Overview Analysis traces the functions defined in the application whose size is more than or equal to 128 bytes by default. It collects the start time, end time of the function, callees function time, and stores the data in raw file for further processing.

Overview Analysis traces GPU offloading which includes kernel launch, kernel execution and data transfer for GPU intensive applications.

Overview Analysis uses the User mode sampling and tracing approach, and it is supported only in Linux and if application is using libc and libpthread then these libraries should be linked dynamically.

Reference

• User Mode Sampling

• User Mode Tracing

7.7.1. Prerequisites

Here is the list of prerequisites.

• Linux kernel version 4.15 or later.

• To profile an application with Overview Analysis, run the script AMDuProfSetup.sh from AMDuProf installed directory with root access.

  $sudo ./AMDuProfSetup.sh
  

  Note

  When you install AMD uProf using the DEB package, the installer automatically executes the script.

• If the system has AMD Instinct accelerators, install the AMD ROCm™ v7.1.0 on the host system.

• After AMD ROCm™ 7.1.0 installation, make sure the symbolic link of /opt/rocm/ points to /opt/ rocm-7.1.0/.

  $ ln -s /opt/rocm-7.1.0/ /opt/rocm/
  

• Supported accelerators - AMD Instinct™ MI200 and MI300A.

7.7.2. Data Collection

7.7.2.1. Data Collection Using GUI

To launch the AMDuProf GUI, go to Home > Welcome page.

1. Click Profile an Application on the Welcome page.

2. Provide application path, application options, working directory, and environment variables, if any. Click Next.

3. From Predefined Configs, select Overview.

4. Set the Timer Interval and Profiling Signal.

5. Click Start Profile to start the profiling.

7.7.2.2. Data Collection Using CLI

• Overview collection

  AMDuProfCLI collect --config overview -o <output-dir> <application>
  

• Overview with timer interval of 100 msec

  AMDuProfCLI collect --config overview --timer-interval 100 -o <output-dir> <application>
  

• Overview with profiling signal as SIGRTMIN

  AMDuProfCLI collect --config overview --profiling-signal 34 -o <output-dir> <application>
  

• Overview with function tracing threshold of 1000000 nsec

  AMDuProfCLI collect --config overview --func-threshold 1000000 -o <output-dir> <application>
  

• Overview to trace the functions of size 32 bytes

  AMDuProfCLI collect --config overview --func-size 32 -o <output-dir> <application>
  

Once profile data collection completes, session directory will be generated. Use session directory to generate the csv report (or) to import the session in GUI.

For a list of all the supported options, refer to AMDuProfCLI Collect Command Options.

Example

./AMDuProfCLI collect --config overview -o /tmp/ /tmp/ScimarkStable
Profiling started
…
Profiling (data collection) completed
Generated data files path: /tmp/AMDuProf-ScimarkStable-Overview_Sep-05-2024_21-59-08

Here, the generated session directory is /tmp/AMDuProf-ScimarkStable-Overview_Sep-05-2024_21-59-08.

Overview Analysis options

7.7.3. Analyze the Data

If data is collected using CLI, use Import Session to import the session into GUI to analyze data in GUI.

7.7.3.1. CLI Report

Use the following CLI report command to generate the profile report in .csv format by passing the session directory path generated in collection.

AMDuProfCLI report -i <session directory>

For a list of all the supported options, refer to AMDuProfCLI Report Command Options.

Example

./AMDuProfCLI report -i /tmp/AMDuProf-ScimarkStable-Overview_Sep-05-2024_21-59-08
Translation started
…
Report generation started
…
Report generation completed...
Generated report file: /tmp/AMDuProf-ScimarkStable-Overview_Sep-05-2024_21-59-08/report.csv

Reference

• Identify the Hottest Function

• Threading Summary

• Wait Time Analysis

7.7.3.2. GPU Offload Analysis

Use GPU Kernel Summary (Analyze the Data) to identify hottest kernels launched to GPU and it’s count, total execution time on GPU cores.

Use Data Transfer Summary (Analyze the Data) to identify how much time spent in data transfer between host and device, how many times data transfer initiated.

Use Per Thread Timeline to analyze the following.

• GPU Usage, GPU Memory usage and GPU Power of your application over the profile duration.

• GPU Kernels executed over the profile duration.

• Data Transfer between host and device over the profile duration, it will help to identify the time spent in data copy.

• For each kernel execution and data copy, refer the HIP trace to check which hip API is causing this activity on GPU. Refer the callstack section for CPU callstack at that time.

From this analysis, we can identify whether application is CPU bound or GPU bound. If application is GPU bound, then use GPU Profiling for further analysis and optimization in kernel execution.

7.7.3.3. Function Trace Analysis

Use the Function Count Summary to identify how many times a function is executed and it’s inclusive and exclusive times.

7.7.3.4. Timeline

Use Per Thread Timeline to analyze the following from CPU.

• CPU Utilization, Context switch count, Memory consumption and thread state over the profile duration.

• To optimize the code, identify the section where thread state is waiting and check what is the cpu utilization at that time and callstack. (Which call path caused this thread to wait).

• Select a region where the CPU utilization is less, timeline provides the list of functions executed within that region and its CPU time.

Here callstack data is collected using Function tracing, it is more accurate when compared with sampling data reported with Hotspots and Threading Analysis. Refer Function Tracing to trace custom functions with overview analysis.

7.7.3.5. Limitations

• MPI tracing and OpenMP tracing are not supported with Overview Analysis.

• System wide profiling data collection and already running process / thread profiling data collection is not supported.

• Overview Analysis is not supported if application is static executable (application is built with -static) as LD_PRELOAD is ignored.

• Callstack data collection is not supported with Overview Analysis.

• If application uses clone system call directly to create a thread (or) process, then behavior is undefined.

• In cluster environment, if two or more threads/processes has same thread id/process id then behavior is undefined.

• Overview Analysis is not supported with AMD Zen1 and AMD Zen2 architectures.

• Profile Control APIs are not supported with Overview Analysis.

• Function tracing will be skipped for non-ELF executables in Overview Analysis.

• Analyzing the 32-bit applications is not supported.

• Behavior is undefined if --no-inherit (not profile the child processes) is used with overview analysis.

7.8. Time-based Profiling

In this analysis, the profile data is periodically collected based on the specified OS timer interval. It is used to identify the hotspots of the profiled applications that are consuming the most time. These hotspots are good candidates for further investigation and optimization.

7.8.1. Configuring and Starting Profile

To configure and start a profile:

1. Click PROFILE > Start Profiling to navigate to the Select Profile Target screen.

2. Select the required profile target and click Next. The Select Profiling screen is displayed.

3. From the Select Profiling screen, select the Predefined Configs tab.

   

4. Select Time-based Sampling in the left vertical pane.

5. Click Advanced Options to enable call-stack, set symbol paths (if the debug files are in different locations) and other options. See Advanced Options for more information on this screen.

6. Set all options and click Start Profile to begin profiling.

   After the profile initialization the profile data collection screen is displayed.

7.8.2. Analyzing Profile Data

Complete the following steps to analyze the profile data:

When the profiling stops, the collected raw profile data will be processed automatically and the Hot Spots screen of the Summary page is displayed. The hotspots are shown for the Timer samples. See Overview of Performance Hotspots for more information.

1. Click ANALYZE on the top horizontal navigation bar to go to the Function HotSpots screen. See Function HotSpots for more information on this screen.

2. Click ANALYZE > Metrics to display the profile data table at various granularities: Process, Load Modules, Threads, and Functions. Refer the section Process and Functions for more information on this screen.

3. Double-click any entry on the Functions table in the Metrics screen to load the source tab for that function in the SOURCES page. Refer the section Source and Assembly for more information on this screen.

7.9. Micro Architecture Analysis

Micro Architecture Analysis profiling follows a statistical sampling-based approach to collect profile data to identify the performance bottlenecks in the application. Use this analysis type to understand the micro architectural bottlenecks in the application runtime.

7.9.1. Overview

AMD uProf CPU profiler follows a statistical sampling-based approach to collect profile data to identify the performance bottlenecks in the application. A few high-level features to understand the CPU profiler capabilities are listed here:

• Profile data is collected using one of the following approaches:

  • Event Based Profiling (EBP) — sampling based on Core PMC events to identify micro- architecture related performance issues in the profiled applications.

  • Instruction based Sampling (IBS) — precise instruction-based sampling.

• Call-stack Sampling

• Secondary Profile Data

  • Thread concurrency (Windows only, requires admin privilege)

  • Thread names (Windows and Linux only)

• Profile Scope

  • Launch App— launch an application and profile that process and its children.

  • System-wide — profile all the running processes and/or kernel.

  • Attach Process — Attach to an existing application (Native applications only)

• Profile mode

  • User/Kernel — profile data is collected when the application is running in User and/or Kernel mode.

• Supported Languages:

  • C, C++

  • Java

  • .NET (5.0, 6.0, and Framework)

  • FORTRAN

  • Assembly applications

• Supported Software Components

  • User-space applications

  • Dynamically linked/loaded modules

  • Drivers

  • OS kernel modules

• Profile data is attributed at various granularities

  • Process, Thread, Load Module, Function, Source line, or Disassembly

  • C++ and Java in-lined functions

  Note

  uProf requires debug information from the compiler for correlating the profile data to functions and source lines.

• Data and Report Files

  • Collected profile data initially stored to raw data files.

  • Processed profile data is stored to database files used for generating the CLI report or GUI visualization.

  • Profile report is saved to a comma-separated-value (CSV) format file that can be viewed using any spreadsheet viewer.

• AMDuProfCLI, the command-line-interface can be used to configure a profile run, collect the profile data, and generate the profile report.

  • Collect command to configure and collect the profile data.

  • Report command to process the profile data and to generate the profile report.

  • Profile command to collect the performance profile data, analyze it, and generate the profile report.

• AMDuProf GUI can be used to:

  • Configure a profile run.

  • Start the profile run to collect the performance data.

  • Analyze the performance data to identify potential bottlenecks.

• AMDuProf GUI has various UI elements to analyze and view the profile data at various granularities:

  • Hot spots summary

  • Session Information

  • Thread concurrency graph (Windows only and requires admin privileges)

  • Process and function analysis

  • Source and disassembly analysis

  • Top-down and bottom-up call path — visualizations to explore the function call flow of an application for analyzing the time spent on functions and its callees.

  • Flame Graph — callstack visualizer as a flame graph

  • Call Graph — call stack and caller/callee visualizer in table format

  • HPC — to analyze OpenMP and MPI profile data

  • Timeline Visualizer — timeline views for MPI API trace and OS event trace information

  • Cache Analysis — to analyze the hot cache lines that are false shared

• Profile Control API

  • Selectively enable and disable profiling from the target application by instrumenting it, to limit the scope of the profiling.

7.9.1.1. Predefined Sampling Configuration

The Predefined Sampling Configuration provides a convenient way to select a useful set of sampling events for profile analysis. The following table lists all such configurations:

7.9.1.2. Predefined Core PMC Events

Listed here are some of the Core Performance events of AMD Zen processors.

Core CPU Metrics

7.9.2. Analysis with Event-based Profiling

In this profile, the CPU Profiler uses the PMCs to monitor the various micro-architectural events supported by the AMD x86-based processor. It helps to identify the CPU and memory related performance issues in the profiled applications. The CPU Profiler provides several predefined EBP profile configurations. To analyze an aspect of the profiled application (or system), a specific set of relevant events are grouped and monitored together. The CPU Profiler provides a list of predefined event configurations, such as Assess Performance and Investigate Branching. You can select any of these predefined configurations to profile and analyze the runtime characteristics of your application. You also can create their custom configurations of events to profile.

In this profile mode, a delay called skid occurs between the time the sampling interrupt occurs and the time the sampled instruction address is collected. Due to this delay, samples may be recorded near but not exactly at the instruction that caused the interrupt. This can lead to an inaccurate distribution of samples, where events are sometimes attributed to neighboring instructions rather than the actual source instruction.

7.9.2.1. Data Collection

7.9.2.1.1. Data Collection Using GUI

To launch the AMDuProf GUI, go to Home > Welcome page.

1. Click Profile an Application on the Welcome page.

2. Provide application path, application options, working directory, and environment variables, if any. Click Next.

3. From Predefined Configs, select any Event Based configuration. For example: Assess Performance, Investigate CPI, Investigate Branching, etc.

4. Alternatively, use Custom Config to configure events individually.

5. From Advanced Options, select the appropriate options.

6. Click Start Profile to start the profiling.

7.9.2.1.2. Data Collection Using CLI

• Event based collection using Assess config

  AMDuProfCLI collect --config assess -o <output-dir> <application>
  

• Event based collection using Investigate Branch config with callstack

  AMDuProfCLI collect --config branch -g -o <output-dir> <application>
  

• Event based collection using individual predefined events

  AMDuProfCLI collect -e cycles-not-in-halt -e retired-inst -o <output-dir> <application>
  

• Event based collection using individual hardware event IDs

  AMDuProfCLI collect -e event=pmcx76,interval=250000 -e event=pmcxc0,user=1,os=0,interval=250000 -o <output-dir> <application>
  

Once profile data collection completes, session directory will be generated. Use session directory to generate the csv report (or) to import the session in GUI.

For a list of all the supported options, refer to AMDuProfCLI Collect Command Options.

Example

./AMDuProfCLI collect --config assess-g -o /tmp/ /tmp/ScimarkStable
Profiling started
………
Profiling (data collection) completed
Generated data files path: /tmp/AMDuProf-ScimarkStable-EBP_Sep-05-2024_21-43-08

Here, the generated session directory is /tmp/AMDuProf-ScimarkStable-EBP_Sep-05-2024_21-43-08.

7.9.2.1.3. Microarchitecture Analysis Options

7.9.2.2. Analyze Data

If data is collected using CLI, use Import Session to import the session into GUI to analyze data in GUI.

7.9.2.2.1. CLI Report

Use the following CLI report command to generate the profile report in .csv format by passing the session directory path generated in collection.

AMDuProfCLI report -i <session directory>

For a list of all the supported options, refer to AMDuProfCLI Report Command Options.

Example

./AMDuProfCLI report -i /tmp/AMDuProf-ScimarkStable-EBP_Sep-05-2024_21-43-08
Translation started
…
Report generation started
…
Report generation completed...
Generated report file: /tmp/AMDuProf-ScimarkStable-EBP_Sep-05-2024_21-43-08/report.csv

Use the Thread Concurrency Graph to analyze how efficiently the processor cores are utilized by the application. In other words, how much time specific number of threads are running on specific no of cores.

Select a function to get all the call paths to this function from different threads, each call path provides the number of samples in that path. Double-click on the function to analyze the instruction level sample attribution for that function using Source View.

7.9.2.2.2. Identify the Hot Code Paths

Use Flame Graph to identify hottest code paths of an application. The width of each function indicates the percentage of event samples of the function (it’s callees) to the total number of samples of selected process and thread for a specific event.

Use Top-Down Callstack to analyze any issues with call-sequence flow of the application and to analyze the bottlenecks in functions and its callees.

7.9.2.2.3. Advisory

Confidence Threshold

The metric with low number of samples collected for a program unit either due to multiplexing or statical sampling will be grayed out. A few points to remember are:

• This is applicable to SW Timer and Core PMC based metrics.

• This confidence threshold value can be set through Preferences section in SETTINGS page.

Issue Threshold

Highlight the CPI metric cells exceeding the specific threshold value (>1.0). Those cells will be highlighted in pink to show them as potential performance problem as follows:

7.9.2.3. Limitations

• CPU profiling expects the profiled application executable binaries must not be compressed or obfuscated by any software protector tools. For example: VMProtect.

• In the case of AMD EPYC™ 1st generation B1 parts, only one PMC register is used at a time for Core PMC event-based profiling (EBP).

7.9.3. Analysis with Instruction Based Sampling

In this profile, the CPU Profiler uses the IBS HW supported by the AMD x86-based processor to observe the effect of instructions on the processor and on the memory subsystem. In IBS, HW events are linked with the instruction that caused them. Also, HW events used by the CPU Profiler to derive various metrics, such as data cache latency.

7.9.3.1. Data Collection

7.9.3.1.1. Data Collection Using GUI

To launch the AMDuProf GUI, go to Home > Welcome page.

1. Click Profile an Application on the Welcome page.

2. Provide application path, application options, working directory, and environment variables, if any. Click Next.

3. From Predefined Configs, select Instruction Based Sampling configuration.

   Alternatively, use Custom Config to configure IBS_FETCH, IBS_ALL_OPS events individually.

4. From Advanced Options, select the appropriate options.

5. Click Start Profile to start the profiling.

7.9.3.1.2. Data Collection Using CLI

• Collection using Instruction Based Sampling config

  AMDuProfCLI collect --config ibs -o <output-dir> <application>
  

• Collection using Instruction Based Sampling config with callstack

  AMDuProfCLI collect --config ibs -g -o <output-dir> <application>
  

• Collection using individual IBS events

  AMDuProfCLI collect -e event=IBS_ALL_OPS,interval=250000
  -e event=IBS_FETCH,user=1,os=0,interval=250000 -o <output-dir> <application>
  

Once profile data collection completes, session directory will be generated. Use session directory to generate the csv report (or) to import the session in GUI.

For a list of all the supported options, refer to AMDuProfCLI Collect Command Options.

Example

./AMDuProfCLI collect --config ibs -g -o /tmp/ /tmp/ScimarkStable
Profiling started
………
Profiling (data collection) completed
Generated data files path: /tmp/AMDuProf-ScimarkStable-IBS_Sep-05-2024_21-43-08

Here, the generated session directory is /tmp/AMDuProf-ScimarkStable-EBP_Sep-05-2024_21-43-08.

7.9.3.2. Analyze Data

If data is collected using CLI, use Import Session to import the session into GUI to analyze data in GUI.

7.9.3.2.1. CLI Report

Use the following CLI report command to generate the profile report in .csv format by passing the session directory path generated in collection.

AMDuProfCLI report -i <session directory>

For a list of all the supported options, refer to AMDuProfCLI Report Command Options.

Example

./AMDuProfCLI report -i /tmp/AMDuProf-ScimarkStable-IBS_Sep-05-2024_21-43-08
Translation started
…
Report generation started
…
Report generation completed...

Generated report file is stored as``/tmp/AMDuProf-ScimarkStable-IBS_Sep-05-2024_21-43-08/report.csv``

Use the Thread Concurrency Graph to analyze how efficiently the processor cores are utilized by the application. In other words, how much time specific no of threads are running on specific no of cores.

Use Function Hotspots to list the functions and the number of samples for the configured events. Expand the function to get its processes and further expand to get its threads.

Select a function to get all the call paths to this function from different threads, each call path provides the number of samples in that path. Double-click on the function to analyze the instruction level sample attribution for that function using Source View.

7.9.3.2.2. Identify the Hot Code Paths

Use Flame Graph to identify hottest code paths of an application. The width of each function indicates the percentage of event samples of the function (it’s callees) to the total number of samples of selected process and thread for a specific event.

Use Top-Down Callstack to analyze any issues with call-sequence flow of the application and to analyze the bottlenecks in functions and its callees.

7.9.3.2.3. ASCII Dump of IBS Samples

For some scenarios, it would be useful to analyze the ASCII dump of IBS OP profile samples. To do so, complete the following steps:

Where:

– interval denotes sampling interval – loadstore denotes collect only the load & store ops (Windows only option) – ibsop-count-control=1 represents count dispatched micro-ops (0 for count clock cycles) - -data-buffer-count 1024 represents the number of per-core data buffers to allocate (Windows only option)

To collect the IBS OP samples:

1. Once the raw file is generated, run the following command to translate and get the ASCII dump of IBS OP samples:

   C:\> AMDuProfCLI.exe translate --ascii event-dump -i C:\temp\AMDuProf-IBS_<timestamp>\
   

   The CSV file that containing ASCII dump of the IBS OP samples is generated:

   C:\temp\AMDuProf-IBS_<timestamp>\IbsOpDump.csv
   

2. During collection the following control knobs are available:

   -e event=ibs-op,interval=100000,loadstore,ibsop-count-control=1
   

In case, there are too many missing records, try the following:

• Increase the sampling interval.

• Increase the data buffer count.

• Reduce the number of cores profiled.

7.9.3.2.4. IBS Derived Events

AMD uProf translates the IBS information produced by the hardware into derived event sample counts that resemble EBP sample counts. All the IBS-derived events contain IBS in the event name and abbreviation. Although IBS-derived events and sample counts look similar to the EBP events and sample counts, the source and sampling basis for the IBS event information are different.

Arithmetic calculation should never be performed between IBS derived event sample counts and EBP event sample counts. It is not meaningful to directly compare the number of samples taken for events that represent the same hardware condition. For example, fewer IBS DC miss samples is not necessarily better than a larger quantity of EBP DC miss samples.

Following table shows the IBS fetch events:

Here is a list of IBS fetch metrics.

Here is a list of IBS op events.

Here is a list of IBS op metrics for AMD Zen3, Zen4 and AMD Zen5 platforms.

7.9.3.2.5. Limitations

CPU profiling in AMD uProf has the following limitations:

• CPU profiling expects the profiled application executable binaries must not be compressed or obfuscated by any software protector tools. For example, VMProtect.

• In case of AMD EPYC™ 1st generation B1 parts, only one PMC register is used at a time for Core PMC event-based profiling (EBP).

IMIX has the following limitations:

• The IMIX view or report is supported only for IBS profile type.

• If any module/binary has less than 10 samples, it is not shown in the IMIX report. Extremely less number of samples are not useful for IMIX analysis.

• Linux kernel module .ko files are not shown in the IMIX view or report.

7.9.3.3. Cache Analysis

The Cache Analysis uses IBS OP samples to detect the hot false sharing cache lines in multi- threaded and multi-process with shared memory applications.

At a high-level, this feature will report:

• The cache lines where there is a potential false sharing

• Offsets where those accesses occur, readers and writers to those offsets

• PID, TID, Function Name, Source File, and Line Number for those reader and writers

• Load latency for the loads to those cache lines

7.9.3.3.1. Supported Metrics

The following IBS OP derived metrics are used to generate false cache sharing report.

7.9.3.3.2. Cache Analysis Using GUI

Configuring and Starting Profile

To perform cache analysis, complete the following steps:

1.Select the profile target. 2.Select Cache Analysis profile type in Predefined Configs tab. 3.Start the profile.

Analyzing the Report

After the profile completion, navigate to Cache Analysis page in MEMORY tab to analyze the profile data. This page shows the cache-lines and it offsets with the associated metric values:

The Cache Analysis screen has the following options:

• Group By drop-down decides how the cache-line samples are grouped in the detailed table. It has the option Cache Line Offset.

• ValueType drop-down allows you to show the value in sample count.

7.9.3.3.3. Cache Analysis Using CLI

The CLI has a config type called memory to cache the analysis data. Run the following command to collect the profile data:

$ AMDuProfCLI collect --config memory -o /tmp/cache_analysis <target app>

This command will launch the program and collect the profile data required to generate the cache analysis report. The raw profile data file is created in /tmp/cache_analysis/AMDuProf- IBS_<timestamp>/ directory.

Report Generation and Analysis

Use the following CLI command to generate the cache analysis report.

$ AMDuProfCLI report -i /tmp/cache_analysis/AMDuProf-IBS_<timestamp>/

This will generate a CSV report in /tmp/cache_analysis/AMDuProf- IBS_<timestamp>/report.csv and it will have the following sections.

• SHARED DATA CACHELINE SUMMARY: Lists the summary values of all the metrics.

• SHARED DATA CACHELINE REPORT: Lists the cache lines and the associated summary values of the metrics.

• SHARED DATA CACHELINE DETAIL REPORT: Lists the following:

  • The cache lines having a potential false sharing

  • Offsets where those accesses occur, readers and writers to those offsets

  • PID, TID, Function Name, Source File, and Line Number for those reader and writers

  • Load latency for the loads to those cache lines

  • Supported metrics

The following figure shows the Cache Analysis summary sections.

The following figure shows the Cache Analysis detailed report.

Use any of the listed metric options with the following command (for example, --sort-by event=ldst-count) to change the sorting by order during the report generation.

--sort-by event=<METRIC>

7.9.3.4. Branch Analysis

AMD Zen4 processors support Last Branch Record (LBR) CPU feature that is useful for branch analysis. Use uProf CLI to collect and generate the branch analysis report.

7.9.3.4.1. Prerequisites

PMC event must be enabled for LBR sample collection. If no PMC event is passed, PMCX0C0 event is enabled during LBR sample collection.

7.9.3.4.2. Configuration

CLI

1. Collect the LBR info.

   $ AMDuProfCLI collect --branch-filter -o /tmp/ ./ScimarkStable/scimark2_64static
   

2. Generate branch analysis report.

   $ AMDuProfCLI report --detail -i /tmp/AMDuProf-scimark2_64static-Custom_mmm-dd-yyyy_hh-mm-ss
   

7.9.3.4.3. Analyze the Data

The report generated contains a section for branch analysis. Here is a sample screenshot of the Branch Analysis Summary.

7.9.3.4.4. Limitations

Branch analysis has the following limitations:

• Branch analysis is supported only on Linux platform.

• Branch analysis is not supported for Java apps.

The branch analysis summary table comprises of the following columns:

7.9.3.5. Virtualization Support

7.9.3.5.1. Profiling of Guest VM from Guest VM

Time based profiling can be performed on all the supported Host and Guest VMs, whereas the hardware counter profiling is completely dependent on the vPMUs exposed by the hypervisor.

7.9.3.5.2. Profiling of Guest VM from Host System (KVM Hypervisor)

This feature supports profiling of KVM guest OS kernel and kernel modules (*.ko) from the host. The following features are supported:

• Collection of PMU samples on guest OS

• Profiling of guest OS and/or host OS

• System wide profiling to profile KVM-guest and other running processes.

The following features are not supported: - Call stack - Attach to process - Launch application

7.9.3.5.3. Preparing Host system to Profile Guest Kernel Modules

Before beginning the profiling on the guest OS, the following files must be copied on the host machine to facilitate symbol resolution for the guest VMs:

1. Copy /proc/kallsyms and /proc/modules from the guest OS to the host machine.

2. Copy guest vmlinux and kernel sources in a folder on the host system.

These files should belong to the guest VM whose PID is provided as an argument to --guest-kvm option.

7.9.3.5.4. AMD uProf CLI with Profiling Options

AMD uProf CLI contains the following options to support the guest OS profiling from the host OS:

$ ./AMDuProfCLI collect [--kvm-guest <pid>] [--guest-kallsyms <path>] [--guest-modules <path>]
[--guest-search-path <path>] ....

The following table lists the Collect command options applicable for profiling options.

Examples

Get the kvm guest OS PID.

$ ps aux | grep kvm

Collecting pmcx76 event data for 10 secs (for guest kallsyms and guest kernel modules).

$ ./AMDuProfCLI collect -e event=pmcx76,interval=250000 -o /tmp/cpuprof-76-guest-only -d 10 -
-kvm-guest 2444 --guest-kallsyms /home/amd/guest/guest-kallsyms --guest-modules /home/amd/ guest/guest-module

Generate report from the collected data.

Collecting pmcx76 event data for 10 secs (for guest kallsyms).

$ ./AMDuProfCLI collect -e event=pmcx76,interval=250000 -o /tmp/cpuprof-76-guest-only -d 10 -
-kvm-guest 2444 --guest-kallsyms /home/amd/guest/guest-kallsyms

Generate report from the collected data.

$ ./AMDuProfCLI report -i /tmp/cpuprof-76-guest-only/AMDuProf-SWP-EBP_Nov-08-2021_15-00-33

Collecting system-wide samples for pmcx76 event data for 10 secs (for guest kallsyms and guest kernel modules).

$ ./AMDuProfCLI collect -e event=pmcx76,interval=250000 -o /tmp/cpuprof-76-guest-only -d 10 -
-kvm-guest 2444 --guest-kallsyms /home/amd/guest/guest-kallsyms --guest-modules /home/amd/ guest/guest-module -a

Generate report from the collected data.

$ ./AMDuProfCLI report -i /tmp/cpuprof-76-guest-only/AMDuProf-SWP-EBP_Nov-08-2021_15-00-33

Collecting system-wide samples for pmcx76 event data for 10 secs (for guest kallsyms).

$ ./AMDuProfCLI collect -e event=pmcx76,interval=250000 -o /tmp/cpuprof-76-guest-only -d 10 -
-kvm-guest 2444 --guest-kallsyms /home/amd/guest/guest-kallsyms -a$ ./AMDuProfCLI collect -e event=pmcx76,interval=250000 -o        /tmp/cpuprof-76-guest-only -d 10 -
-kvm-guest 2444 --guest-kallsyms /home/amd/guest/guest-kallsyms -a

Generate report from the collected dataGenerate report from the collected data.

$ ./AMDuProfCLI report -i /tmp/cpuprof-76-guest-only/AMDuProf-SWP-EBP_Nov-08-2021_15-00-33

7.10. Parallelism - OpenMP Analysis

The OpenMP API uses the fork-join model of parallel execution. The program starts with a single master thread to run the serial code. When a parallel region is encountered, multiple threads perform the implicit or explicit tasks defined by the OpenMP directives. At the end of that parallel region, the threads join at the barrier and only the master thread continues to execute.

When the threads execute the parallel region code, they should utilize all the available CPU cores and the CPU utilization should be maximized. But the threads wait without doing anything useful due to several reasons:

• Idle: Thread being in an inactive state, waiting for an event to occur.

• Sync: If locks are used inside the parallel region, threads can wait on synchronization locks to acquire the shared resource. Also, this includes thread waiting at implicit barrier.

• Overhead: The thread management overhead

The OpenMP analysis helps to trace the activities performed by OpenMP threads, their states, and provides the thread state timeline for parallel regions to analyze the performance issues. Use the Parallel Strong Scaling Metrics (Parallel Strong Scaling Metrics (MPI + OpenMP)) to quantify and decompose scalability losses in OpenMP (and MPI) applications.

Parallel Region Aggregation

A parallel region can be executed multiple times during runtime. Reporting all the instances separately might result in a lengthy report making it difficult to analyze the data. AMD uProf aggregates data of multiple instances of the same parallel region and shows it as a single entry for better analysis.

Support Matrix

The following table shows the support matrix:

Prerequisite

Compile the OpenMP application using a supported compiler (on a supported platform) with the required compiler options to enable OpenMP.

7.10.1. Data Collection Using GUI

Complete the following steps to start profiling:

1. Click Profile an Application on the Welcome page.

2. Provide application path, application options, working directory, and environment variables, if any. Click Next.

3. Select at least one supported predefined Configuration such as TBP/EBP/IBS along with any desired configuration and click Advanced Options.

4. In the OpenMP Tracing Options pane, turn on the Enable OpenMP Tracing option.

   

5. Select the Select OpenMP Trace Implementation type. Choose:

   1. ompt (default option) for tracing of OpenMP libraries supporting OMPT interface (example: LLVM, AOCC, ICC).

   2. omplib for tracing GCC OpenMP library.

6. If you have selected ompt, next Select OpenMP Tracing Mode. Choose:

   1. full for tracing all the OpenMP events.

   2. basic for basic tracing, where synchronization related OpenMP events are not traced to reduce the disk space usage.

7. Click Start Profile to start the profiling.

7.10.2. Data Collection Using CLI

Command to collect basic trace info of an OpenMP application supporting OMPT interface:

$ AMDuProfCLI collect --trace openmp --openmp-impl ompt --openmp-scope basic -o /tmp/myapp_perf <openmp-app>

Command to profile an OpenMP application compiled with GCC OpenMP library:

$ AMDuProfCLI collect --trace openmp --openmp-impl omplib -o /tmp/myapp_perf <openmp-app>

Use the --openmp-impl option to provide OpenMP implementation type: ompt for tracing of OpenMP libraries supporting OMPT interface (example: LLVM, AOCC, ICC), omplib for tracing GCC OpenMP library. If --openmp-impl is not specified, the default selection is ompt.

Use --openmp-scope option to provide tracing scope: full for tracing all the OpenMP events, basic for basic tracing, where synchronization related OpenMP events are not traced to reduce the disk space usage. If --openmp-scope is not specified, the default selection is basic.

While performing the regular profiling, add option –trace openmp –openmp-impl <ompt | omplib> to enable OpenMP profiling. This command will launch the program and collect the profile data required to generate the OpenMP analysis report.

Once profile data collection is complete, a session directory will be generated. Use session directory to generate the csv report (or) to import the session in GUI.

For a list of all the supported options, refer to AMDuProfCLI Collect Command Options.

7.10.3. Analyze the Data

If data is collected using CLI, then use Import Session to import the session into GUI to analyze data in GUI. OpenMP trace data can be collected in Linux and the session can be imported to GUI or CLI on Windows.

Analyzing the GUI Views

After the session is opened, navigate to the HPC page to analyze the OpenMP tracing data. You can use the left side vertical pane on this page to navigate through the following views:

Overview shows the quick details about the runtime. The following image shows the Overview page.

• Overview has the following fields:

  • Profile Time: Total profile duration.

  • Time Inside Parallel Region: Total time spent inside parallel regions.

  • Time Outside Parallel Region: Total time spent outside parallel regions.

  • Parallel Time %: Percentage of total time spent inside parallel regions.

  • Total threads created: Total number of threads created.

  • CPU Time: Total time spent utilizing the CPU.

    • Work Time: Time spent working in the parallel region.

    • Sync Time: Total time spent at barrier, lock contention and other synchronization activities.

      • Barrier/Wait Time: Time spent waiting at explicit or implicit barrier by the threads.

      • Lock Contention Time: Time spent by threads waiting on locks or ordered parallel loops.

      • Other Sync Time: Time spent in synchronization activities other than waiting at barriers or for locks.

    • Overhead Time: Total time spent in atomic, reduction and other thread management operations.

      • Reduction Time: Time spent on reduction operations.

      • Atomic Time: Time spent on performing atomic operations.

      • Other Overhead Time: Time spent on other thread management overhead.

    • Idle Time: Time spent in an inactive state, waiting for an event to occur.

• OpenMP Parallel Regions shows the summary of all the parallel regions. This tab is useful to quickly understand which parallel region might be load imbalanced, i.e., a region with less total work time with respect to its total elapsed time. Double-click on the region names to open the region-wise thread details page.

• OpenMP Parallel Regions has the following columns:

  • Avg Idle Time (secs): Average time spent by the parallel region threads in an inactive state, waiting for an event to occur. It is computed as the difference between the total elapsed time of a parallel region and the sum of Avg Sync Time, Avg Overhead Time and Avg Work Time of that parallel region.

  • Avg Sync Time (secs): Average time spent by the parallel region threads waiting on the synchronization locks to acquire the shared resource & waiting in barriers. It is computed as the sum of sync time of all the threads within the parallel region divided by the thread count of that parallel region.

  • Avg Overhead Time (secs): Average time spent in atomic, reduction and other thread management operations. It is computed as the sum of overhead time of all the threads within the parallel region divided by the thread count of that parallel region.

  • Avg Work Time (secs): Average time spent by the parallel region threads working. It is computed as the sum of work time of all the threads within the parallel region divided by the thread count of that parallel region.

  • Total Elapsed Time (secs): Time spent in the parallel region.

  • Thread Count: Number of threads in the parallel region.

  • Instance Count: Number of instances of the parallel region executed.

• Region-Wise Thread Details page helps in understanding how each thread spent its time in the parallel region. If a thread spends too much time on non-work activity, the parallel region should be optimized further to improve the work time of each thread in that region. It has the following columns:

  • Thread No.: Serial number of the thread.

  • Thread Id: Thread identifier.

  • Total Idle Time (secs): Time spent by the thread in an inactive state, waiting for an event to occur.

  • Total Sync Time (secs): Time spent by the thread waiting on the synchronization locks to acquire the shared resource.

  • Total Overhead Time (secs): Time spent by the thread in atomic, reduction and other thread management operations.

  • Total Work Time (secs): Time spent by the thread working.

7.10.4. Generate Profile Report

You can generate a CSV report using the AMDuProfCLI report command. Any additional option is not required for the OpenMP report generation. AMD uProf checks for the availability of any OpenMP profiling data and includes it in the report, if available.

The following command will generate a CSV report in /tmp/myapp_perf/<SESSION-DIR>/ report.csv:

$ ./AMDuProfCLI report -i /tmp/myapp_perf/<SESSION-DIR>

An example of the OpenMP report section in the CSV file is given here:

Analyzing the OpenMP Report

Openmp report includes the following sections:

• OpenMP OVERVIEW has the following fields:

  • Profile Time (s): Total profile duration.

  • Time Inside Parallel Region (s): Total time spent inside parallel regions.

  • Time Outside Parallel Region (s): Total time spent outside parallel regions.

  • Parallel Region Time %: Percentage of total time spent inside parallel regions.

  • Total threads created: Total number of threads created.

  • CPU Time (seconds): Total time spent utilizing the CPU.

    • Work Time: Time spent working in the parallel region.

      • Sync Time: Total time spent at barrier, lock contention and other synchronization activities.

      • Barrier/Wait Time: Time spent waiting at explicit or implicit barrier by the threads.

      • Lock Contention Time: Time spent by threads waiting on locks or ordered parallel loops.

    • Other Sync Time: Time spent in synchronization activities other than waiting at barriers or for locks.

      • Overhead Time: Total time spent in atomic, reduction and other thread management operations.

      • Reduction Time: Time spent on reduction operations.

      • Atomic Time: Time spent on performing atomic operations.

      • Other Overhead Time: Time spent on other thread management overhead.

    • Idle Time (seconds): Time spent in an inactive state, waiting for an event to occur.

• OpenMP PARALLEL-REGION METRIC helps in understanding the imbalanced region, that is, a region with less total work time with respect to its total elapsed time. It has the following columns:

  • Avg Idle Time (s): Average time spent by the parallel region threads in an inactive state, waiting for an event to occur. It is computed as the difference between the total elapsed time of a parallel region and the sum of Avg Sync Time, Avg Overhead Time and Avg Work Time of that parallel region.

  • Avg Sync Time (s): Average time spent by the parallel region threads waiting on the synchronization locks to acquire the shared resource. It is computed as the sum of sync time of all the threads within the parallel region divided by the thread count of that parallel region.

  • Avg Overhead Time (s): Average time spent in atomic, reduction and other thread management operations. It is computed as the sum of overhead time of all the threads within the parallel region divided by the thread count of that parallel region.

  • Avg Work Time (s): Average time spent by the parallel region threads working. It is computed as the sum of work time of all the threads within the parallel region divided by the thread count of that parallel region.

  • Total Elapsed Time (s): Time spent in the parallel region.

  • Thread Count: Number of threads in the parallel region.

  • Instance Count: Number of instances of the parallel region executed.

• OpenMP THREAD METRIC helps in understanding how each thread spent its time in the parallel region. If a thread spends too much time on non-work activity, the parallel region should be optimized further to improve the work time of each thread in that region. It has the following columns:

  • ThreadNum: Serial number of the thread.

  • ThreadId: Thread identifier.

  • Total Idle Time (s): Time spent by the thread in an inactive state, waiting for an event to occur.

  • Total Sync Time (s): Time spent by the thread waiting on the synchronization locks to acquire the shared resource.

  • Total Overhead Time (s): Time spent by the thread in atomic, reduction and other thread management operations.

  • Total Work Time (s): Time spent by the thread working.

7.10.5. Environment Variables

AMDUPROF_MAX_PR_INSTANCES – Set the max number of parallel regions to be traced. The default value is 2000000.

7.10.6. Limitations

The following features are not supported in this release:

• OpenMP profiling with system-wide profiling scope.

• Nested parallel regions.

• GPU offloading and related constructs.

• Callstack for individual OpenMP threads.

• OpenMP profiling on Windows and FreeBSD platforms.

• Profiling applications with static linkage of OpenMP libraries.

• omp_control_tool routine to start/pause/end OpenMP tracing is currently not supported for GCC compiled applications.

• While tracing LLVM/AOCC compiled applications, calling omp_control_tool routine to start/pause/end OpenMP tracing might not work as expected if it is the very first OpenMP routine in the application. It is recommended to call one of these entry points (omp_get_max_threads() , omp_get_num_procs()) or #pragma omp parallel etc. before calling omp_control_tool routine.

• Attaching to running OpenMP application.

• Profiling applications compiled with GCC OpenMP library and using multiple non-OpenMP threads.

• Tracing of target, teams and taskloop constructs are not supported in case of applications compiled with GCC OpenMP library.

7.11. Parallelism - MPI Trace Analysis

MPI trace analysis can be used to analyze, and compute the message passing load imbalance among the ranks of a MPI application running on a cluster. It supports OpenMPI, MPICH, and their derivatives.

The supported thread models are MPI_THREAD_SINGLE, MPI_THREAD_FUNNLED, and MPI_THREAD_SERIALIZED. The profile reports are generated for Point-to-Point and Collective API activity summary.

Fortran bindings are configured and built while compiling the MPI implementations. You can enable/ disable the Fortran bindings based on your need for Fortran language support.

Refer the following options to disable/enable the Fortran bindings:

• OpenMPI

  --enable-mpi-fortran[=VALUE]
  --disable-mpi-fortran
  

  By default, OpenMPI will attempt to build all the 3 Fortran bindings: mpif.h, mpi module, and mpi_f08 module.

• MPICH

  --disable-fortran
  

  By default, the Fortran bindings are enabled. You can use this option to disable it.

MPI Trace Support Matrix