Added changes to support runtime version selection of schema in rocpdsna-part1 by anujshuk-amd · Pull Request #4762 · ROCm/rocm-systems

anujshuk-amd · 2026-04-07T09:16:59Z

Motivation

Rocpdsna currently only supports only the V3 (3.0.0) schema. As the profiling ecosystem evolves, a newer V4 (4.0.0) schema is needed to support with runtime selction of schema version. Changes supports...

Timestamp normalization: V4 replaces direct start/end BIGINT columns with foreign keys to a rocpd_timestamp table, enabling deduplication and cross-table timestamp correlation.

Track-based context: V4 consolidates per-row nid/pid/tid/agent_id/queue_id/stream_id columns into a single track_id FK, reducing redundancy and simplifying queries.

Normalized event metadata: V4 moves embedded JSONB call_stack and line_info out of rocpd_event into dedicated rocpd_call_stack and rocpd_line_info tables with proper FK relationships.

New V4-only tables : rocpd_info_category, rocpd_info_address_range, rocpd_info_source_code, and rocpd_info_pc support richer profiling data.

Runtime schema selection: Users can choose V3 or V4 at runtime without recompilation.
This PR implements the V4 schema writers, adds comprehensive test coverage for both schemas, and cleans up the codebase.

Technical Details

V4 Schema Writer Implementation : Implemented region_writer.hpp, kernel_dispatch_writer.hpp, memory_copy_writer.hpp, memory_alloc_writer.hpp, pmc_event_writer.hpp under source/writers/schema_v4/
Insert Statement Layer

schema_v3/insert_statements.hpp: updated V3 insert SQL to align with the versioned backend schema under backends/schema/3.0.0/
schema_v4/insert_statements.hpp: new V4 insert SQL covering all 27 V4 tables, including normalized timestamp, track, call stack, line info, and category tables

Entity Registry & Utilities updated
Test Coverage: Added new test cases for v3 and v4 schema validation.

The rocpdsna depends on #347 PR, the latest version of rocprofiler-sdk schema changes.

JIRA ID

TBA

Test Plan

Added new tests to support new changes.

Test Result

All rocpd tests and unit tests are passing, and the benchmark is running without any error.

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

nileshnegi · 2026-04-10T14:05:31Z

@anujshuk-amd please ensure projects/rccl/ext-src/rocSHMEM‎ is not affected by this PR

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR adds runtime-selectable writer support for rocpd schema versions (v3 and new v4), along with accompanying infrastructure (policies, context, query builders, and build/test/benchmark wiring) to support the new normalized v4 schema layout.

Changes:

Introduces schema-versioned writer policies and v4 writer implementations (region/kernel/memory/pmc), plus shared writer_context and policy trait validation.
Adds query-builder utilities (SELECT/INSERT fluent builders) and expands build system support (queries lib, tests, benchmarks, bundled deps).
Adds/updates v3/v4 schema backend SQL assets (views/indexes/templates) and various common utilities (entity registry/traits/logger).

Reviewed changes

Copilot reviewed 113 out of 157 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
projects/rocpdsna/tests/benchmarks/utility.hpp	Replaces old insert-query helper with file size formatting/stat helpers for benchmarks
projects/rocpdsna/tests/benchmarks/CMakeLists.txt	Adds benchmark executable target and dependencies
projects/rocpdsna/tests/CMakeLists.txt	Adds benchmark subdirectory gated by `ROCPDSNA_BUILD_BENCHMARKS`
projects/rocpdsna/source/writers/writer_policy_traits.hpp	Adds compile-time policy validation (required type aliases + CRTP interface checks)
projects/rocpdsna/source/writers/writer_context.hpp	Introduces shared writer dependency container (`backend`, registry, key providers, validator, uuid)
projects/rocpdsna/source/writers/schema_v4/writer_policy.hpp	Defines v4 writer policy wiring (tag + stmts + writer types)
projects/rocpdsna/source/writers/schema_v4/region_writer.hpp	Implements v4 region inserts using track/timestamp/event normalization
projects/rocpdsna/source/writers/schema_v4/pmc_event_writer.hpp	Implements v4 PMC event inserts and optional event/sample correlation
projects/rocpdsna/source/writers/schema_v4/memory_copy_writer.hpp	Implements v4 memory copy inserts using track/timestamp normalization
projects/rocpdsna/source/writers/schema_v4/memory_alloc_writer.hpp	Implements v4 memory alloc inserts plus type/level validation
projects/rocpdsna/source/writers/schema_v4/kernel_dispatch_writer.hpp	Implements v4 kernel dispatch inserts using track/timestamp normalization
projects/rocpdsna/source/writers/schema_v3/writer_policy.hpp	Defines v3 writer policy wiring
projects/rocpdsna/source/writers/schema_v3/region_writer.hpp	Implements v3 region inserts (legacy nid/pid/tid + raw timestamps)
projects/rocpdsna/source/writers/schema_v3/pmc_event_writer.hpp	Implements v3 PMC event inserts (legacy event insert)
projects/rocpdsna/source/writers/schema_v3/memory_copy_writer.hpp	Implements v3 memory copy inserts (legacy topology columns)
projects/rocpdsna/source/writers/schema_v3/memory_alloc_writer.hpp	Implements v3 memory alloc inserts (legacy topology columns)
projects/rocpdsna/source/writers/schema_v3/kernel_dispatch_writer.hpp	Implements v3 kernel dispatch inserts (legacy topology columns)
projects/rocpdsna/source/writers/schema_v3/common_insert_operations.hpp	Adds v3 common insert ops (strings/events/samples/args)
projects/rocpdsna/source/writers/region_writer.hpp	Adds schema-tag templated region writer forward declaration
projects/rocpdsna/source/writers/pmc_event_writer.hpp	Adds schema-tag templated PMC writer forward declaration
projects/rocpdsna/source/writers/memory_copy_writer.hpp	Adds schema-tag templated memory copy writer forward declaration
projects/rocpdsna/source/writers/memory_alloc_writer.hpp	Adds schema-tag templated memory alloc writer forward declaration
projects/rocpdsna/source/writers/kernel_dispatch_writer.hpp	Adds schema-tag templated kernel dispatch writer forward declaration
projects/rocpdsna/source/writers/interfaces/region_writer_interface.hpp	Adds CRTP interface for region insert API
projects/rocpdsna/source/writers/interfaces/pmc_event_writer_interface.hpp	Adds CRTP interface for PMC event insert API
projects/rocpdsna/source/writers/interfaces/memory_copy_writer_interface.hpp	Adds CRTP interface for memory copy insert API
projects/rocpdsna/source/writers/interfaces/memory_alloc_writer_interface.hpp	Adds CRTP interface for memory alloc insert API
projects/rocpdsna/source/writers/interfaces/kernel_dispatch_writer_interface.hpp	Adds CRTP interface for kernel dispatch insert API
projects/rocpdsna/source/writers/interfaces/info_registration_writer_interface.hpp	Adds CRTP interface for info registration (v3+ and v4-only methods)
projects/rocpdsna/source/writers/interfaces/api_writer_base.hpp	Adds CRTP base helper (`self()`)
projects/rocpdsna/source/writers/info_registration_writer.hpp	Adds schema-tag templated info writer forward declaration
projects/rocpdsna/source/writers/common_insert_operations.hpp	Adds schema-tag templated common insert ops forward declaration
projects/rocpdsna/source/writer_impl.cpp	Implements runtime schema dispatch to v3 vs v4 writer implementations
projects/rocpdsna/source/writer.cpp	Wires public `writer_t` API to pimpl implementation
projects/rocpdsna/source/storage_impl.hpp	Introduces `storage_t::impl` and schema-version storage state
projects/rocpdsna/source/storage_impl.cpp	Implements storage backend creation and schema-version retrieval
projects/rocpdsna/source/storage.cpp	Exposes `storage_t` public API + schema version getter
projects/rocpdsna/source/reader_impl.hpp	Adds reader implementation scaffolding and cached utilities
projects/rocpdsna/source/reader.cpp	Wires public `reader_t` API to pimpl implementation
projects/rocpdsna/source/queries/select/table_select_query.hpp	Adds fluent SELECT query entry-point wrapper
projects/rocpdsna/source/queries/select/select_query_builders.hpp	Adds select builder chain declarations enforcing clause order
projects/rocpdsna/source/queries/query_common.hpp	Adds shared query enums (sort/join)
projects/rocpdsna/source/queries/query_builder_base.hpp	Adds base builder with stream helpers and common column appends
projects/rocpdsna/source/queries/query_builder_base.cpp	Implements base builder stream helpers
projects/rocpdsna/source/queries/insert/table_insert_query.hpp	Adds fluent INSERT query entry-point wrapper
projects/rocpdsna/source/queries/insert/insert_query_builders.hpp	Adds insert column/value builders
projects/rocpdsna/source/queries/CMakeLists.txt	Builds `rocpdsna_queries` static library
projects/rocpdsna/source/primary_key_providers.hpp	Adds centralized PK generators incl. v4-specific tables
projects/rocpdsna/source/json_serializers.hpp	Declares JSON serialization/deserialization helpers for callstack/source context
projects/rocpdsna/source/insert_validator.hpp	Declares insert validation & FK resolution helpers
projects/rocpdsna/source/entity_utility.hpp	Adds thread-safe registry utility incl. optimized string_view lookup specialization
projects/rocpdsna/source/entity_registry.hpp	Adds entity registry containers for v3+ and v4-only entities
projects/rocpdsna/source/data_storage/schema_version.hpp	Adds schema tag types for compile-time specialization (v3/v4)
projects/rocpdsna/source/data_storage/backends/schema/4.0.0/summary_views.sql	Adds/updates v4 summary views
projects/rocpdsna/source/data_storage/backends/schema/4.0.0/rocpd_views.sql	Adds/updates v4 logical views mapping uuid-suffixed tables
projects/rocpdsna/source/data_storage/backends/schema/4.0.0/rocpd_shema.in	Adds v4 schema header-generation template
projects/rocpdsna/source/data_storage/backends/schema/4.0.0/rocpd_metadata.sql	Adds v4 metadata insertion script
projects/rocpdsna/source/data_storage/backends/schema/4.0.0/rocpd_indexes.sql	Adds v4 index creation script
projects/rocpdsna/source/data_storage/backends/schema/3.0.0/rocpd_shema.in	Updates v3 schema template namespace layout
projects/rocpdsna/source/data_storage/CMakeLists.txt	Adjusts storage sources inclusion (sqlite backend compilation wiring)
projects/rocpdsna/source/common/types.hpp	Adds internal “owned” key types + hashers used by registries
projects/rocpdsna/source/common/traits.hpp	Adds generic traits used by query builders and utilities
projects/rocpdsna/source/common/string_conversions.hpp	Adds key extraction + to_string/to_error_string helpers
projects/rocpdsna/source/common/logger.hpp	Adds logging implementation wrapper around spdlog
projects/rocpdsna/source/common/directory.hpp	Adds directory helper utilities (dirname/mkdirp)
projects/rocpdsna/source/common/debug.hpp	Adds logging macros gated by `ROCPDSNA_ENABLE_LOGGING`
projects/rocpdsna/source/autoincrementer.hpp	Adds atomic autoincrement helper for PKs
projects/rocpdsna/source/CMakeLists.txt	Adds core sources + subdirectories (queries/data_storage) and links queries lib
projects/rocpdsna/include/writer.hpp	Adds/updates public writer API including v4-only registration methods
projects/rocpdsna/include/storage_types.hpp	Introduces `version_t` struct and comparisons for schema versioning
projects/rocpdsna/include/storage.hpp	Adds public storage API including schema version getter
projects/rocpdsna/include/shared_types.hpp	Adds shared callstack/source context ADTs referenced by writer/reader
projects/rocpdsna/docs/graphs/writer_insert_flow.dot	Adds writer insert flow documentation diagram
projects/rocpdsna/docs/graphs/writer_context_dependencies.dot	Adds writer context dependency injection diagram
projects/rocpdsna/docs/graphs/writer_architecture.dot	Adds layered writer architecture diagram
projects/rocpdsna/docs/graphs/system_layers.dot	Adds system layering diagram
projects/rocpdsna/docs/graphs/schema_statement_correspondence.dot	Adds statements↔schema correspondence diagram
projects/rocpdsna/docs/graphs/public_api_ownership.dot	Adds public API ownership/lifetime diagram
projects/rocpdsna/docs/graphs/backend_abstraction.dot	Adds backend abstraction diagram
projects/rocpdsna/cmake/sqlite3.cmake	Adds bundled/system SQLite3 selection & build logic
projects/rocpdsna/cmake/spdlog.cmake	Adds bundled/system spdlog selection & fetch logic
projects/rocpdsna/cmake/rocprofiler-sdk-rocpd.cmake	Adds schema file generation and optional rocprofiler-sdk-rocpd integration
projects/rocpdsna/cmake/rocpdsna-config.cmake.in	Adds install-time package config template
projects/rocpdsna/cmake/rocm.cmake	Adds ROCm prefix-path helper
projects/rocpdsna/cmake/nlohmann_json.cmake	Adds bundled/system nlohmann_json selection & fetch logic
projects/rocpdsna/cmake/gtest.cmake	Adds bundled/system GoogleTest selection & fetch logic
projects/rocpdsna/cmake/coverage.cmake	Adds code coverage build targets
projects/rocpdsna/cmake/benchmark.cmake	Adds bundled/system Google Benchmark selection & fetch logic
projects/rocpdsna/cmake/Findrocpdsna.cmake	Adds find-module for consumers (imported targets, vars)
projects/rocpdsna/README.md	Adds project-level README with build/install instructions
projects/rocpdsna/.gitignore	Adds project-specific ignore list
projects/rocpdsna/.clang-tidy	Adds clang-tidy config for the project
projects/rocpdsna/.clang-format	Adds clang-format configuration

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+        // Default to v3 (backward compatible)
+        m_impl = std::make_unique<writer_impl_polymorphic<writer_policy_v3>>(ctx);
+    }


+/**
+ * @brief Constructor with runtime schema selection
+ *
+ * Queries the storage instance for its configured schema version
+ * and instantiates the appropriate writer implementation (v3 or v4).
+ * Supports LATEST marker which resolves to the most recent schema.
+ */
+writer_t::impl::impl(std::unique_ptr<rocpdsna::storage_t> storage)
+: m_storage(std::move(storage))
+, m_version(m_storage->get_storage_version())
+{
+    // Resolve LATEST marker to actual version
+    auto actual_version = m_version;


@@ -21,37 +21,47 @@
 // SOFTWARE.

 #pragma once


+#include <cstddef>
+#include <functional>
+#include <string>
+
+#include "rocpdsna/writer_types.hpp"


+
+#include <memory>
+#include <optional>


+CREATE INDEX `rocpd_timestamp{{uuid}}_track_id_idx` ON `rocpd_timestamp{{uuid}}` ("track_id");
+
+-- CREATE INDEX `rocpd_kernel_dispatch{{uuid}}_guid_pid_tid_idx` ON `rocpd_kernel_dispatch{{uuid}}` ("guid", "pid", "tid");
+CREATE INDEX `rocpd_memory_copy{{uuid}}_guid_pid_tid_idx` ON `rocpd_memory_copy{{uuid}}` ("guid", "pid", "tid");


@@ -0,0 +1,83 @@
+---
+# Clang-Tidy configuration for rocstorage


+
+WarningsAsErrors: ''
+
+HeaderFilterRegex: '.*/rocstorage/(include|source)/.*\.hpp$'


…alues.

mradosav-amd added 6 commits March 25, 2026 12:35

Add rocsna project

f3969c3

Add integration of rocpdsna in rocprofiler-systems

5dc2b09

Switch to use string_view instead const char* in writer api

6cf1dca

First phase of aligning rocpd implementation

02ad893

Change agent_type to be optional for AI nic needs

6b87cf9

Add support for static linking of rocpd-sna

6c0b2e1

github-actions Bot added documentation Improvements or additions to documentation project: rocprofiler-systems project: rccl labels Apr 7, 2026

systems-assistant Bot added the organization: ROCm label Apr 7, 2026

anujshuk-amd self-assigned this Apr 13, 2026

anujshuk-amd requested review from adjordje-amd and mradosav-amd April 13, 2026 13:33

anujshuk-amd force-pushed the users/anujshuk-amd/version-rocpd-sna-develop branch 3 times, most recently from 4dd4ef2 to 840a644 Compare April 14, 2026 10:47

anujshuk-amd added 2 commits April 14, 2026 06:51

Add runtime schema version selection and V4 schema support in rocpdsna

8035e59

Revert accidental rocSHMEM submodule change

3b565c9

anujshuk-amd force-pushed the users/anujshuk-amd/version-rocpd-sna-develop branch from 840a644 to 3b565c9 Compare April 14, 2026 10:53

anujshuk-amd marked this pull request as ready for review April 14, 2026 14:12

anujshuk-amd requested a review from jrmadsen as a code owner April 14, 2026 14:12

Copilot AI review requested due to automatic review settings April 14, 2026 14:12

anujshuk-amd requested a review from a team as a code owner April 14, 2026 14:12

Copilot AI reviewed Apr 14, 2026

View reviewed changes

Fixed build issue with current sdk and updated code to take extdata v…

c0d9165

…alues.

anujshuk-amd closed this May 7, 2026

Navigation Menu

Uh oh!

Added changes to support runtime version selection of schema in rocpdsna-part1 - #4762

Added changes to support runtime version selection of schema in rocpdsna-part1#4762
anujshuk-amd wants to merge 9 commits into
developfrom
users/anujshuk-amd/version-rocpd-sna-develop

anujshuk-amd commented Apr 7, 2026 •

edited

Loading

Uh oh!

nileshnegi commented Apr 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -0,0 +1,83 @@
		---
		# Clang-Tidy configuration for rocstorage


		WarningsAsErrors: ''

		HeaderFilterRegex: './rocstorage/(include\|source)/.\.hpp$'


		#include <memory>
		#include <optional>

Uh oh!

Conversation

anujshuk-amd commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

JIRA ID

Test Plan

Test Result

Submission Checklist

Uh oh!

nileshnegi commented Apr 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

anujshuk-amd commented Apr 7, 2026 •

edited

Loading