feat(rocprofiler-sdk): share per-agent ATT resources across contexts by minseobshin11 · Pull Request #8644 · ROCm/rocm-systems

minseobshin11 · 2026-07-15T20:27:09Z

Motivation

ATT allocates heavyweight resources per context and per agent, even though only one thread trace can use an agent at a time:

A separate GPU output-buffer set per context wastes device memory. At ~1 GiB each, a few hundred contexts exceed a single device's HBM.
A separate HSA submission queue per context exhausts the agent's queue budget. HSA_AGENT_INFO_QUEUES_MAX is 128 on gfx942 and is shared with the application's own queues; exceeding it aborts.

This PR gives each GPU agent a single resource owner. Contexts targeting the same agent reuse its output slots, submission queue, and CPU staging buffers.

Technical Details

Adds shared_trace_resources.{hpp,cpp} with one AgentTraceResources per agent, owning the submission queue, CPU staging buffers, and GPU output slots.
Tracers, AQLProfile packets, and producer/consumer workers hold the owner by shared_ptr, so a buffer cannot be released while anything still references it.
Collects each context's buffer and staging sizes before the first owner is constructed. The output buffer is allocated from the AQLProfile allocation callback during the first tracer's construction, so its size must be final by then.
Maps AQLProfile output allocations by packet-local slot index, so slot i resolves to the same buffer for every context on an agent. Host-visible allocations stay per packet.
Passes each trace's own buffer size to the producer thread, since shared staging buffers are sized to the largest context on the agent.
Records the owning context on its first trace and fails loudly if a second context tries to trace the same agent concurrently.
Waits for a trace's stop packet to retire, and drains in-flight dispatches before removing the interception callback, so resources are released only once the GPU is done.

JIRA ID

AIPROFSDK-102

Test Plan

~400 contexts with varying ATT parameters, alternating 1 GiB and 512 MiB buffers, in both device and dispatch mode — over 3x the 128-queue limit.
Validate decoded shader data: PCs resolve to loaded code objects, chunks come only from selected shader engines, and on gfx10+ waves come from the selected SIMD.
Shared-resource, producer/consumer, standard ATT, pre-HSA-init, large-buffer, and triple-buffer tests.
Hardware: AMD Instinct MI300X (gfx942).

Test Result

400-context device and dispatch: 5/5 repeated runs passed.
thread-trace-packet-test: 9/9 passed.
thread-trace-producer-consumer-test: 10/10 passed.
ATT integration suite: all 12 enabled tests passed (2 disabled upstream).

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

therock-pr-bot · 2026-07-15T20:28:53Z

❌ PR Check — Action Required

Check	Status	Details
🌿 Branch Name	✅ Pass	—
📝 PR Title/Description	❌ Fail	Error: PR description must reference a JIRA ID, ISSUE ID, or a GitHub closing keyword. Expected: include a `JIRA ID` / `ISSUE ID` line (separator `:` or `-`, or omitted; value may be a JIRA key, a number with/without `#`, or a link), OR a closing keyword + issue reference. Accepted examples: • `JIRA ID : TESTAUTO-6039` • `JIRA ID - #330` • `JIRA ID #330` • `JIRA ID` (on separate line) `ROCM-25757` • `ISSUE ID : TESTUTO-3334` • `ISSUE ID #3334` • `ISSUE ID - TESTAUTO-3433` • `ISSUE ID` (on separate line) `AIRUNTIME-2352` • `ISSUE ID : https://github.com/<org_name>/<repo_name>/issues/1234` • `Closes #10` • `Fixes octo-org/octo-repo#100` • `Resolves: #123` • `#123` • `https://github.com/<org_name>/<repo_name>/issues/123` Current: no valid JIRA/ISSUE/closing-keyword reference found
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🔎 pre-commit	⏳ Pending	⏳ Still running…
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

⚠️ 1 policy check(s) failed. Please address the issues above before this PR can be Reviewed.

🚫 Please fix the failed policies

❌ PR Title/Description

The Not ready to Review label was added to this PR. Once all policies pass, the label is removed automatically.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

github-actions · 2026-07-15T21:32:24Z

Code Coverage Report

Tests Only

Samples Only

Tests + Samples

ApoKalipse-V · 2026-07-20T10:16:30Z

 }

+void
+DispatchThreadTracer::register_shared_queue_sizes()


This is duplicated. Maybe join with above function?

ApoKalipse-V · 2026-07-20T10:16:42Z

 }

+void
+DeviceThreadTracer::register_shared_queue_sizes()


Same deduplication

ApoKalipse-V · 2026-07-20T10:17:01Z

    {
-        if(ctx->device_thread_trace) ctx->device_thread_trace->register_shared_buffer_sizes();
-        if(ctx->dispatch_thread_trace) ctx->dispatch_thread_trace->register_shared_buffer_sizes();
+        if(ctx->device_thread_trace)


You can revert this change if they are joined

true-- changed

ApoKalipse-V · 2026-07-20T10:18:54Z

+    // Max triple-buffer staging size requested by any context on this agent; the
+    // shared queue is created with this size so every context's flips fit.
+    uint64_t        max_triple_buffer_size = 0;
+    att_queue_ptr_t queue                  = {};


Maybe join with the buffer struct? They have the same semantics.
(not mandatory)

left them separate because their contents diverged after changes

ApoKalipse-V · 2026-07-20T10:20:45Z

+// Keep each context active across a few dispatches so it reliably captures one.
+constexpr uint64_t CAPTURE_WINDOW = 2;
+
+// Repro mode for the AILIKFD-39 teardown hang: start one context and never stop it,


Why are we reproing it here?

ApoKalipse-V · 2026-07-20T10:28:36Z

+    if(leave_active())
+    {
+        // Deliberately do NOT stop the trace context: leave SQTT active so rocprofiler
+        // teardown destroys the still-active per-agent tracers one-by-one (AILIKFD-39).


same comment
Also, this is unlikely to work in dispatch mode

ApoKalipse-V · 2026-07-20T10:31:11Z

+
+    // Not every target_cu is guaranteed to have active waves, so require that MOST
+    // configs produced data rather than all.
+    assert(captured * 4 >= configured * 3 && "most configs should capture data");


This is not not going to fail even if the profiler gave you the same header-only buffer every time. You need to verify the waves are (mostly) valid in the capture window and that they are all in the correct SIMD/CUs you selected in the configuration.

Note that SIMD selection is different for gfx >= 10, you need to take SEL%4 while for gfx9 you take the bitmask.
Then in the validation: Verify you don't get wave_t from simd/cus you didnt ask for, and that you do get (most) of them for the ones you did. To verify waves are valid, you can check the instructions have a valid PC address (except for TRAP and for the first few in case of gfx1250).

fixed so that the test now decodes every captured trace and asserts real content

I believe gfx9 emits all the wave records for all four SIMDs regardless of the requested simd_select - is there a way to validate on gfx10+

ApoKalipse-V · 2026-07-20T10:33:03Z

We also need to cover device thread trace

added more tests to cover

therock-pr-bot · 2026-08-10T17:16:21Z

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

❌ PR Title/Description

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

github-actions Bot added the project: rocprofiler-sdk label Jul 15, 2026

minseobshin11 mentioned this pull request Jul 15, 2026

[ATT] Improved buffer management: per-agent shared trace buffers #8272

Draft

1 task

systems-assistant Bot added the organization: ROCm label Jul 15, 2026

minseobshin11 changed the base branch from develop to users/mishin/att-buffer-management July 16, 2026 18:23

ApoKalipse-V requested changes Jul 20, 2026

View reviewed changes

minseobshin11 force-pushed the users/mishin/att-buffer-management branch from 6fd101b to 00a0b6f Compare July 20, 2026 22:54

minseobshin11 force-pushed the users/mishin/att-shared-queue branch from 6a85b04 to a442878 Compare July 20, 2026 22:54

minseobshin11 force-pushed the users/mishin/att-buffer-management branch from 00a0b6f to 7178390 Compare July 20, 2026 23:49

minseobshin11 force-pushed the users/mishin/att-shared-queue branch 2 times, most recently from 8d51e5a to 3ae7f3c Compare July 21, 2026 16:21

minseobshin11 force-pushed the users/mishin/att-shared-queue branch from 3ae7f3c to eacd73b Compare July 21, 2026 16:27

minseobshin11 removed documentation Improvements or additions to documentation project: amdsmi project: rocprofiler-systems labels Jul 21, 2026

minseobshin11 changed the base branch from users/mishin/att-buffer-management to develop July 26, 2026 18:50

minseobshin11 added 3 commits July 27, 2026 14:07

shorten test timeouts

584d5a0

refactor

4e35568

add comments

04650a1

minseobshin11 added 2 commits August 10, 2026 17:17

rescope to resource unification

b8425cc

nit

10511c6

Uh oh!

Conversation

minseobshin11 commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

JIRA ID

Test Plan

Test Result

Submission Checklist

Uh oh!

therock-pr-bot Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ PR Check — Action Required

Uh oh!

github-actions Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage Report

Code Coverage Report

Tests Only

Samples Only

Tests + Samples

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ApoKalipse-V Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

therock-pr-bot Bot commented Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

minseobshin11 commented Jul 15, 2026 •

edited

Loading

therock-pr-bot Bot commented Jul 15, 2026 •

edited

Loading

github-actions Bot commented Jul 15, 2026 •

edited

Loading

ApoKalipse-V Jul 20, 2026 •

edited

Loading