feat(hubs): add Azure Resource Manager ingestion - #2277
Draft
Brett Wilson (MSBrett) wants to merge 9 commits into
Draft
feat(hubs): add Azure Resource Manager ingestion#2277Brett Wilson (MSBrett) wants to merge 9 commits into
Brett Wilson (MSBrett) wants to merge 9 commits into
Conversation
Add generic ARM query ingestion with quota and Savings Plan definitions, normalized KQL surfaces, catalog queries, documentation, and regression coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Brett Wilson (MSBrett)
had a problem deploying
to
ftk-pr
August 24, 2026 00:28 — with 
GitHub Actions
Error
GitHub Actions
Error
microsoft-github-policy-service
Bot
added
the
Needs: Review 👀
Contributor
|
Hubs + ADX (managed) deployment failed. View logs. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds reusable Azure Resource Manager ingestion to FinOps hubs, supporting savings-plan recommendations and quota/capacity datasets.
Changes:
- Adds manifest-driven ARM GET ingestion and deployment wiring.
- Adds normalized quota and savings-plan schemas and KQL queries.
- Extends portal configuration, documentation, and regression tests.
Reviewed changes
Copilot reviewed 45 out of 46 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
src/templates/finops-hub/test/main.test.bicep |
Adds feature deployment combinations. |
.../Recommendations/schemas/recommendations_1.1.json |
Defines savings-plan translation. |
.../Recommendations/queries/Recommendations-Microsoft-SavingsPlan-P3Y.json |
Adds three-year recommendations. |
.../Recommendations/queries/Recommendations-Microsoft-SavingsPlan-P1Y.json |
Adds one-year recommendations. |
.../Recommendations/app.bicep |
Registers new recommendation assets. |
.../Quota/schemas/quota_1.0-usage.json |
Defines usage translation. |
.../Quota/schemas/quota_1.0-sql.json |
Defines SQL quota translation. |
.../Quota/schemas/quota_1.0-disk.json |
Defines disk translation. |
.../Quota/schemas/quota_1.0-capacity-reservation.json |
Defines capacity translation. |
.../Quota/queries/Quota-Microsoft-StorageUsage.json |
Adds Storage usage ingestion. |
.../Quota/queries/Quota-Microsoft-SqlSubscriptionUsage.json |
Adds SQL usage ingestion. |
.../Quota/queries/Quota-Microsoft-PremiumSSDv2Disk.json |
Adds disk inventory ingestion. |
.../Quota/queries/Quota-Microsoft-ComputeUsage.json |
Adds compute usage ingestion. |
.../Quota/queries/Quota-Microsoft-CognitiveServicesUsage.json |
Adds AI quota ingestion. |
.../Quota/queries/Quota-Microsoft-CapacityReservation.json |
Adds capacity inventory ingestion. |
.../Quota/queries/Quota-Microsoft-AppServiceUsage.json |
Adds App Service usage ingestion. |
.../Quota/app.bicep |
Deploys quota queries and schemas. |
.../IngestionQueries/app.bicep |
Integrates ARM staging and finalization. |
.../AzureResourceManager/README.md |
Documents the ARM engine. |
.../AzureResourceManager/metadata.bicep |
Defines ARM app metadata. |
.../AzureResourceManager/app.bicep |
Implements ARM request pipelines. |
.../AzureResourceGraph/app.bicep |
Extends the shared engine contract. |
.../Analytics/scripts/IngestionSetup_v1_2.kql |
Normalizes savings-plan recommendations. |
.../Analytics/scripts/IngestionSetup_v1_0.kql |
Adds quota transforms and tables. |
.../Analytics/scripts/IngestionSetup_RawTables.kql |
Adds the quota raw table. |
.../Analytics/scripts/HubSetup_v1_0.kql |
Adds the versioned quota function. |
.../Analytics/scripts/HubSetup_Latest.kql |
Adds the latest quota alias. |
src/templates/finops-hub/modules/hub.bicep |
Wires ARM and quota apps. |
.../fx/hub-deploymentScript.bicep |
Forces deployment-script refreshes. |
src/templates/finops-hub/main.bicep |
Exposes quota configuration. |
src/templates/finops-hub/createUiDefinition.json |
Adds portal controls and guidance. |
src/scripts/Build-HubIngestionQueries.ps1 |
Generates manifests for both apps. |
src/queries/INDEX.md |
Catalogs the new queries. |
src/queries/finops-hub-database-guide.md |
Documents the quota contract. |
src/queries/catalog/savings-plan-recommendation-breakdown.kql |
Summarizes savings-plan recommendations. |
src/queries/catalog/quota-storage-usage.kql |
Queries Storage quota state. |
src/queries/catalog/quota-sql-subscription-usage.kql |
Queries SQL quota state. |
src/queries/catalog/quota-premium-ssd-v2-disks.kql |
Queries Premium SSD v2 inventory. |
src/queries/catalog/quota-headroom.kql |
Calculates quota headroom. |
src/queries/catalog/quota-current-usage.kql |
Queries current quota state. |
src/queries/catalog/quota-compute-usage.kql |
Queries compute quota state. |
src/queries/catalog/quota-cognitive-services-usage.kql |
Queries AI quota state. |
src/queries/catalog/quota-capacity-reservations.kql |
Queries capacity reservations. |
src/queries/catalog/quota-app-service-usage.kql |
Queries App Service quota state. |
src/powershell/Tests/Unit/HubsIngestionQueries.Tests.ps1 |
Extends ingestion regression coverage. |
.gitignore |
Ignores Copilot tracking artifacts. |
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
Adds an eighth view to the FinOps hub dashboard canvas extension, porting the AI and emerging workloads page from the Azure Data Explorer dashboard into the extension's own query and rendering pipeline. Query layer (kusto.mjs): - getAi() supplies 16 named queries across estate composition, token and model economics, workload detail, allocation, and rate posture. - Month-over-month comparisons anchor to the last closed month. The newest month in a window is normally still ingesting, so comparing it against a full month reported a spend collapse that was not real. - Token direction uses word-boundary regexes rather than substring matching, so meters such as "Cd Wr Std" classify correctly instead of falling into an Other bucket, and a future "Throughput" meter cannot be read as output. - Owner tags fold case-insensitively while retaining the highest-cost casing for display, so one owner cannot appear as several rows. - Follows repo KQL guidance: no bare joins, and contains reserved for genuinely fused substrings. Rendering (public/): - Six KPI cards, two new chart primitives, and fifteen panels. - Formatters gain fmtRate, fmtQty, fmtShare, moneyColumn and rateColumn. Column helpers hold one precision per column, because a reader compares cells down a column rather than against each value's own magnitude, and mark sub-unit values rather than rounding real spend to zero. - Axis ceilings round up to readable steps and never clip the series. - Every table is wrapped so it scrolls within its panel. An unwrapped table previously forced horizontal scrolling on the whole page at narrow widths. - The active tab is scrolled into view, including on deep links. - Panels are paired by content height so no card renders mostly empty. Also fixes two latent defects reachable from existing views: table swatches were not displayed, and threshold colours were defined only under .kpi so they had no effect inside tables. Verified against a live hub across 1.2M rows, with no horizontal overflow at 1600, 1280, 950, 768 and 480 CSS pixels. Test suite covers the new behaviour and passes 34/34. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Quota answered only the demand side of capacity: how much of my limit have I used. It could not answer the supply side: can a deployment land in this region and zone at all. Two Azure Resource Manager quota queries close that gap. - Quota-Microsoft-AvailabilityZoneMapping records each subscription's logical-to-physical availability zone mapping, so zonal capacity is comparable across subscriptions. - Quota-Microsoft-ComputeResourceSku records VM SKU family, zone, and the region and zone restrictions that block placement regardless of quota. Both queries get a schema, a raw table, an ingestion transform, a Hub function, and a catalog query for rehydration. Fix rehydration filters that returned the wrong rows. Each catalog query now filters on x_SourceType, ResourceType, and ResourceName together instead of x_SourceType alone. Premium SSD v2 and disk quota come from the Compute provider but describe storage, so they belong to the Storage view, not the Compute view. Compute usage now keeps only core and VM family metrics. Raise the ingestion query schedule from daily to every four hours. Quota and restriction state changes faster than a day. Publish a manifest only when the query's own parquet file is present. Checking folder existence alone published manifests for queries that wrote nothing, because a sibling query had already created the folder. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A repository-scoped Copilot canvas that reads a FinOps hub cluster directly, so a capacity manager can inspect quota and capacity without building a report first. It carries seven capacity views: App Service, Azure AI, Compute, Azure SQL, Storage, capacity reservations, and Premium SSD v2. The Compute view adds an estate-wide VM family by region heatmap. Supply and demand are separate channels, because they are separate questions: - Supply is a bar on the left edge of the cell. Red means a deployment cannot land here, whether the whole region is restricted or every zone is. Amber means some zones are restricted. Green means open. - Demand is the colour of the percentage itself, measured against a selectable high-water mark that defaults to 70 percent. Colour never carries the alarm alone. Each cell also states its condition in words, so the heatmap reads without colour vision. At estate scale the matrix is a lookup grid, not a picture. It keeps frozen row and column headers, always-visible scrollbars, a family search, region chips, and four condition lenses. Add npm run test-dashboard and run it in the development workflow. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
graphify-out/ is generated locally and must not enter the repository. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Brings in the AI and emerging workloads view. Both branches had first committed the dashboard independently while it was untracked, so git saw every dashboard file as an add/add conflict with no common ancestor. Resolved as a real three-way merge against checkpoint 3370dbac5, the last shared state of the untracked working copy. Six of the eight files split cleanly: the capacity work and the AI work touched disjoint regions. Two needed a decision: - extension.mjs: both sides rewrote the canvas description. Kept the new view and kept "capacity views" rather than "capacity evidence views". - The test file: both sides appended to the same tail. Rebuilt as the union, 31 capacity tests plus 12 AI tests, with no name collisions. The merged AI code reintroduced the word "evidence", which this dashboard no longer uses. Renamed the "Evidence" table column to "Detail", reworded the rate-posture panel, and reworded four comments, so the vocabulary test passes on the union rather than by suspending the rule. Verified live against the hub: nine tabs, the AI view rendering all 15 panels with data and no horizontal overflow, and the Compute heatmap still reporting 135 region-restricted and 73 all-zones-restricted cells with red supply bars. 43 of 43 tests pass and the design detector is clean. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- add filter-aware family and subscription detail tabs - page subscription results with search and focus restoration - align Kusto filters and preserve unknown headroom states - add contract and accessibility regression coverage 🗺️ - Generated by Copilot Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Brett Wilson (MSBrett)
temporarily deployed
to
ftk-pr
August 24, 2026 19:55 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 60 out of 63 changed files in this pull request and generated 8 comments.
Suppressed comments (3)
src/templates/finops-hub/modules/Microsoft.FinOpsHubs/AzureResourceManager/app.bicep:1005
- Native
AbsoluteUrlpagination follows the response'snextLinkwithout checking its authority. A compromised or malformed ARM response can redirect the REST linked service to another host while it is authenticated with the managed identity, which violates the linked requirement to keep continuation requests on the current Resource Manager authority. Validate every continuation URL before issuing it; native unvalidated paging is not safe for this engine.
src/templates/finops-hub/modules/Microsoft.FinOpsHubs/AzureResourceManager/app.bicep:1030 - The request URL is built directly from manifest-controlled
queryScopeandqueryvalues and sent through the authenticated ARM linked service. There is no runtime rejection of absolute/scheme-relative URLs, fragments, backslashes, malformed resource IDs, or unsupported scopes, so a modified query manifest can escape the intended ARM-relative request contract. Add validation before this dataset is invoked and attach authentication only after the value is proven ARM-relative.
src/templates/finops-hub/modules/Microsoft.FinOpsHubs/AzureResourceManager/app.bicep:400 - This prefix check accepts every Microsoft.Billing resource ID, including unsupported nested resources such as invoice sections. The Savings Plan queries are only compatible with billing accounts and billing profiles, so one unsupported configured scope can make the parallel query pipeline fail. Validate the exact resource-ID shape and dispatch only scopes whose derived resource type is declared compatible by the query.
Comment on lines
+215
to
216
| batchCount: app.hub.options.privateRouting ? 4 : 30 | ||
| isSequential: false |
Comment on lines
+1682
to
+1686
| | extend x_SourceVersion = coalesce(x_SourceVersion, case( | ||
| x_SourceType =~ 'SavingsPlan', '2026-06-01', | ||
| ProviderName == 'Microsoft', '2023-05-01', | ||
| '' | ||
| )) |
Comment on lines
+14
to
+15
| - '.github/extensions/ftk-local-dashboard/**' | ||
| - 'package.json' |
| > **Note:** Refer to the [FinOps hub database documentation](./finops-hub-database-guide.md) for table and column definitions. | ||
|
|
||
| This catalog contains 37 scenario-specific FinOps Hub KQL queries used by the FinOps Toolkit agents and the Azure SRE Agent recipe. | ||
| This catalog contains 47 scenario-specific FinOps hub KQL queries used by the FinOps Toolkit agents and the Azure SRE Agent recipe. |
Comment on lines
+643
to
+649
| | `AppServiceUsage` | `1.0-usage` | App Service regional quota usage | Synthetic `ResourceId` | | ||
| | `CapacityReservation` | `1.0-capacity-reservation` | Capacity reservation group inventory | Native ARM `ResourceId` | | ||
| | `CognitiveServicesUsage` | `1.0-usage` | Azure AI services regional quota usage | Synthetic `ResourceId` | | ||
| | `ComputeUsage` | `1.0-usage` | Compute regional quota usage | Synthetic `ResourceId` | | ||
| | `PremiumSSDv2Disk` | `1.0-disk` | Premium SSD v2 disk inventory | Native ARM `ResourceId` | | ||
| | `SqlSubscriptionUsage` | `1.0-sql` | Azure SQL regional quota usage | Native ARM `ResourceId` | | ||
| | `StorageUsage` | `1.0-usage` | Storage regional quota usage | Synthetic `ResourceId` | |
Comment on lines
918
to
+919
| "options": { | ||
| "text": "The Data Factory managed identity requires Reader role on the management groups or subscriptions you want to scan. After deployment, grant the hub's managed identity Reader access to the desired scopes." | ||
| "text": "The Data Factory managed identity requires the Reader role on the management groups or subscriptions to scan. After deployment, grant the hub's managed identity Reader access to each scope." |
Comment on lines
+32
to
+33
| - Added a repository-scoped FinOps hub dashboard canvas with separate capacity views for App Service, Azure AI, Compute, Azure SQL, Storage, capacity reservations, and Premium SSD v2. The Compute view adds an estate-wide VM family by region heatmap that reads supply restrictions and quota utilization as separate signals ([#2277](https://github.com/microsoft/finops-toolkit/pull/2277)). | ||
| - Added an AI and emerging workloads view to the FinOps hub dashboard canvas, covering estate composition, token and model economics, workload detail, allocation coverage, and commitment and rate posture. Month-over-month comparisons anchor to the last closed month so a partially ingested month cannot report a false decline ([#2277](https://github.com/microsoft/finops-toolkit/pull/2277)). |
Comment on lines
+2837
to
+2838
| const kustoBundleUrl = `https://cdn.jsdelivr.net/npm/@kusto/monaco-kusto@${KUSTO_MONACO_VERSION}/+esm`; | ||
| const kustoBundleSrc = await fetch(kustoBundleUrl).then((r) => r.text()); |
Contributor
|
Hubs + ADX (managed) deployed to |
- validate ARM scopes and continuations using one stored response per page - filter configured scopes and propagate source metadata - preserve quota and recommendation account identities - enforce deployment order and least-privilege workflow access 🔒 - Generated by Copilot Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
|
Hubs + ADX (managed) deployed to |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🛠️ Description
This PR added a reusable Azure Resource Manager ingestion path for GET-only APIs and connected its output to the existing FinOps hub ingestion flow.
Managed Azure Resource Manager ingestion
Quota and recommendation data
Quota()functions and query-catalog entries for current usage, headroom, source-specific quota views, and savings plan recommendation breakdowns.Documentation and tests
.copilot-tracking/workflow artifacts.This work provides shared ingestion groundwork for MACC consumption. MACC Lots and MACC Events ingestion are not included in this PR. Quota and capacity support extends beyond the current iteration described in #1721.
Refs #1721
Related to #2120 and #2129
📷 Screenshots
Not applicable.
📋 Checklist
🔬 How did you test this change?
The full unit suite passed with 2,453 tests and no failures. The full lint suite passed with 3,439 tests and no failures. App Service ingestion completed through Data Factory and Azure Data Explorer, and all seven quota catalog projections returned the expected schema and key behavior.
📦 Deploy to test?
🙋♀️ Do any of the following that apply?
📑 Did you update
docs/changelog.md?A changelog entry is pending before this draft is ready for review.
📖 Did you update documentation?
Public documentation remains pending before this draft is ready for review.