[xpupti] Support more than 1 device in ptiMetricsScopeConfigure - #1479
[xpupti] Support more than 1 device in ptiMetricsScopeConfigure#1479aostrowski-hbn wants to merge 3 commits into
Conversation
|
@pytorchbot label ciflow/xpu |
|
The ciflow label(s) ciflow/xpu will be added, but CI won't be triggered until the workflows are approved (scroll to the bottom of this page). Please ping one of the reviewers if you do not have access to approve and run workflows. |
|
The following ciflow label(s) have been added but CI has not been triggered yet because the workflows are awaiting approval:
Once a maintainer approves the workflows (scroll to the bottom of the PR page), the corresponding CI jobs will be triggered automatically. Please ping one of the reviewers if you do not have access to approve and run workflows. |
Empty list means all devices (PTI auto-detect); a comma-separated list selects an explicit device-index subset, parsed with a C++20 views::filter|transform pipeline. PYTORCHDGQ-8150. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
7ee6615 to
8f18fc8
Compare
Remove the hardcoded device_count=1 in enableScopeProfiler. On PTI >= 0.18 (PTI-363) default to auto-detect (all devices) and honor an explicit XPUPTI_PROFILER_DEVICES subset; on older PTI keep single-device behavior and reject a >1-device request with a clear error. Add selectDeviceHandles(std::span<const pti_device_handle_t>, std::span<const int>): a C++20 ranges gather (find_if bounds check + transform) mapping requested indices to device handles. Host-only gtests cover the config key and the helper. PYTORCHDGQ-8150. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add PerKernelScopeExplicitDevice0 driving the selectDeviceHandles -> ptiMetricsScopeConfigure(handles,count) branch on real hardware; parameterize RunTest with an optional XPUPTI_PROFILER_DEVICES value. PYTORCHDGQ-8150. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
8f18fc8 to
f2275e5
Compare
|
|
||
| #if PTI_VERSION_AT_LEAST(0, 18) | ||
| if (requestedDevices.empty()) { | ||
| // Default: profile every available device (PTI auto-detect mode). |
There was a problem hiding this comment.
This comment was inaccurate. When requested devices is 0, it will profile whichever devices the workload actually uses.
Note: you may also want to update the PR body.
| xpuptiProfilerMaxScopes_); | ||
| xpuptiProfilerMaxScopes_, | ||
| xpuptiProfilerDevices_.empty() | ||
| ? std::string("all") |
There was a problem hiding this comment.
"auto" may be more accurate than "all" here - For example, on a host where the workload only touches 4 of 16 cards, only 4 get profiled, not "all."
|
One comment to the Testing description: "Auto-detect initializes 16 per-device metrics handlers" is true because the 16-tile PVC test workload dispatched to all 16 tiles — not because auto-detect unconditionally profiles all devices. |
|
Nit: The PTI implementation validates uniformity for any multi-device configure: So on a mixed-GPU host, both explicit multi-device selection and auto-detect (if the workload touches >1 non-uniform device) will fail inside |
| TEST_F(XpuptiScopeProfilerConfigTest, DevicesParsedList) { | ||
| KN::Config cfg; | ||
| EXPECT_TRUE(cfg.parse("XPUPTI_PROFILER_METRICS = metric1")); | ||
| EXPECT_TRUE(cfg.parse("XPUPTI_PROFILER_DEVICES = 0, 2, 3")); |
There was a problem hiding this comment.
nit: would be nice to add a test for testing empty "tokens", for example:
EXPECT_TRUE(cfg.parse("XPUPTI_PROFILER_DEVICES = 0, ,2,"));
Trailing comma + doubled comma: empty tokens must be skipped, not parsed as index 0 or rejected as invalid integers.
Additionally, what if there are duplicated indices: XPUPTI_PROFILER_DEVICES=0,0,2?
2 participants
Enables the XPU Scope Profiler to collect hardware metrics on more than one XPU. Previously
enableScopeProfilerhardcodeddevice_count=1in theptiMetricsScopeConfigurecall, so metrics were only ever collected on a single card even on multi-GPU hosts.What changed
devices=nullptr, count=0): profiles all available devices.XPUPTI_PROFILER_DEVICES=0,2,3selects an explicit device subset via a newselectDeviceHandleshelper (C++20std::span+std::rangesgather with bounds validation).PTI_VERSION_AT_LEAST(0, 18): multi-device support inptiMetricsScopeConfigurelanded in PTI 0.18. On older PTI, single-device behavior is preserved and a request for >1 device is rejected with a clear error.Device attribution needs no change: each scope record is already correlated to its kernel activity (and thus device/resource) via
_kernel_id.Testing
Verified on a 16-tile Intel Data Center GPU Max (PVC) host built against PTI 0.18:
XpuptiScopeProfilerConfigTest): 8/8 pass — config-key parsing +selectDeviceHandlesbounds validation.XpuptiScopeProfilerTest): 3/3 pass, including a newPerKernelScopeExplicitDevice0exercising the explicit-subset path.