Skip to content

Commit 260f4aa

Browse files
author
kjjanaki
committed
feat(mcp): EMR Spark Troubleshooting MCP integration for AWS DevOps Agent
Adds mcp/emr-spark-troubleshooting-mcp-connect/ — a single CloudFormation template that registers the AWS-managed Apache Spark Troubleshooting MCP server (SageMaker Unified Studio) with AWS DevOps Agent using SigV4 auth. What deploys ------------ - IAM role trusted by aidevops.amazonaws.com with SourceAccount / SourceArn conditions scoped to arn:aws:aidevops:<region>:<account>: service/* (matches the console-suggested scope). - Conditional inline policies for MCP invoke, EMR on EC2, EMR Serverless, Glue, scoped S3 read on caller-supplied Spark artifacts buckets, and optional CloudWatch Logs KMS decrypt. IAM shape mirrors the AWS canonical spark-troubleshooting-mcp-setup template, tightened so S3 access is scoped to caller-supplied bucket names instead of Resource: '*'. - AWS::DevOpsAgent::Service (native CFN) with MCPServerSigV4 details — registers the MCP endpoint as a capability provider. Attaching the registered Service to an Agent Space is a one-time manual step performed once via the DevOps Agent console — this is intentional and out of scope for the template. No VPC, VPC endpoint, or Private Connection ------------------------------------------- Earlier drafts of this template deployed a VPC endpoint for sagemaker-unified-studio-mcp and an AWS::DevOpsAgent::PrivateConnection. Both were removed after PR review confirmed AWS DevOps Agent reaches SigV4-authenticated MCPs over the AWS backbone using each MCP's publicly-resolvable endpoint. The DevOps Agent API also explicitly rejects PrivateConnectionName on SigV4 Services. Any VPC endpoint / Private Connection alongside a SigV4 MCP is inert for the DevOps Agent → MCP data path. Region support -------------- CFN Rules block fail-fasts outside the 11 regions where both AWS DevOps Agent and the SageMaker Unified Studio MCP service are available. Tested ------ - cfn-lint clean. - Stack CREATE_COMPLETE in ~30 seconds, DELETE_COMPLETE in ~20 seconds. - Verified Service registered with the correct SigV4 details and role ARN via aws cloudformation describe-stacks.
1 parent 3a5dd7f commit 260f4aa

4 files changed

Lines changed: 522 additions & 0 deletions

File tree

‎llms.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,7 @@ Tools can be used with these AWS DevOps Agent types:
5656
- [VPC DNS Diagnostics MCP](mcp/aws-vpc-dns-diagnostics-mcp/README.md): Observes live DNS resolution from inside an affected subnet and simulates proposed DNS control-plane changes before they are applied
5757
- [EKS Node Diagnostics MCP](mcp/aws-eks-node-diagnostics-mcp/README.md): Collects and analyzes diagnostic logs from EKS worker nodes via SSM Automation (kubelet, containerd, CNI, iptables, dmesg, and more) not accessible through the Kubernetes API or CloudWatch
5858
- [ECS Instance Log MCP](mcp/ecs-instance-log-mcp/README.md): Collects and analyzes diagnostic logs from ECS container instances via SSM Automation (ECS agent, Docker/containerd, system logs, networking, GPU) not accessible through the ECS API or CloudWatch
59+
- [EMR Spark Troubleshooting MCP — DevOps Agent Integration](mcp/emr-spark-troubleshooting-mcp-connect/README.md): Registers the AWS-managed Apache Spark Troubleshooting MCP endpoint (SageMaker Unified Studio) with AWS DevOps Agent using SigV4; creates the IAM role and the Service (capability provider) only — no MCP server code is deployed
5960

6061
## Key Concepts
6162

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Changelog
2+
3+
All notable changes to this integration are documented here.
4+
5+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
6+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7+
8+
## [1.0.0] - 2026-09-23
9+
10+
### Added
11+
12+
- Initial release. Single CloudFormation template
13+
(`cloudformation/integration.yaml`) that registers the AWS-managed
14+
Apache Spark Troubleshooting MCP endpoint (SageMaker Unified Studio) with
15+
AWS DevOps Agent.
16+
- Native use of the CloudFormation resource type
17+
`AWS::DevOpsAgent::Service` — no Lambda-backed custom resource is
18+
required for the DevOps Agent side of the registration.
19+
- SigV4 IAM role trusted by `aidevops.amazonaws.com`, with
20+
`aws:SourceAccount` and `aws:SourceArn` conditions (scoped to
21+
`arn:aws:aidevops:<region>:<account>:service/*` per the console
22+
guidance), and conditional inline policies for MCP invoke, EMR on EC2,
23+
EMR Serverless, Glue, S3 access on caller-supplied Spark artifacts
24+
buckets, and optional CloudWatch Logs KMS decrypt.
25+
- IAM shape mirrors the AWS canonical
26+
`spark-troubleshooting-mcp-setup.yaml` template with S3 access
27+
tightened to caller-supplied bucket names.
28+
- Region fail-fast (`Rules` block) covering the 11 regions where both AWS
29+
DevOps Agent and the SageMaker Unified Studio MCP service are
30+
available.
31+
32+
### Notes
33+
34+
- **Only the IAM role and the DevOps Agent Service registration are
35+
deployed.** No VPC, subnets, VPC endpoint, or DevOps Agent Private
36+
Connection are created. Investigation during PR review (see
37+
[`aws/tools-for-devops-agent#103`](https://github.com/aws/tools-for-devops-agent/pull/103))
38+
confirmed that AWS DevOps Agent reaches SigV4-authenticated MCPs over
39+
the AWS backbone using the MCP's publicly-resolvable endpoint. The
40+
DevOps Agent API also explicitly rejects `PrivateConnectionName` on
41+
SigV4 Services. Any VPC endpoint / Private Connection deployed
42+
alongside a SigV4 MCP is inert for the DevOps Agent → MCP data path.
43+
- Attaching the registered Service to an Agent Space is a one-time manual
44+
step performed via the DevOps Agent console — out of scope for the
45+
template.
Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
# EMR Spark Troubleshooting MCP — DevOps Agent Integration
2+
3+
One-click CloudFormation deploy that registers the AWS-managed **Apache Spark Troubleshooting Agent for Amazon EMR** (a SageMaker Unified Studio MCP server) with **AWS DevOps Agent**. **No MCP server code is deployed** — the MCP is hosted by AWS. This template creates the SigV4 IAM role AWS DevOps Agent uses to invoke the MCP, and the AWS DevOps Agent Service (capability provider) that wraps the registration.
4+
5+
> [!IMPORTANT]
6+
> This is sample code. Test in a non-production environment first, review IAM permissions against your organization's policies, and validate the deployment before production use.
7+
8+
## What this deploys
9+
10+
| Resource | Why |
11+
|---|---|
12+
| `AWS::IAM::Role` trusted by `aidevops.amazonaws.com` | The SigV4 identity DevOps Agent uses to call the MCP. `SourceAccount` + `SourceArn` conditions confine it to your account. |
13+
| Inline policies on that role | MCP invoke; scoped S3 read on caller-supplied Spark artifacts buckets; conditional per-toggle reads of EMR / EMR Serverless / Glue and their CloudWatch Log groups; optional CloudWatch KMS decrypt. |
14+
| `AWS::DevOpsAgent::Service` (MCPServerSigV4) | Registers the MCP endpoint as a capability provider under SigV4 auth. |
15+
16+
**Not deployed**: agent spaces, capability-provider attachments, S3 buckets, demo workloads. Attaching the registered Service to your Agent Space is a one-time manual step (see [Step 3](#step-3-attach-the-service-to-your-agent-space)).
17+
18+
## Region support
19+
20+
Deploy in a region where **both** AWS DevOps Agent and the SageMaker Unified Studio MCP service are available:
21+
22+
`us-east-1`, `us-west-2`, `ap-northeast-1`, `eu-west-1`, `ap-southeast-1`, `ap-southeast-2`, `ca-central-1`, `sa-east-1`, `eu-central-1`, `eu-west-2`, `ap-south-1`
23+
24+
The template fails fast in any other region via a `Rules` block.
25+
26+
## Prerequisites
27+
28+
- An AWS account
29+
- An **AWS DevOps Agent space** already created in the target region (see the [DevOps Agent Getting Started guide](https://docs.aws.amazon.com/devopsagent/latest/userguide/getting-started-with-aws-devops-agent-creating-an-agent-space.html))
30+
- AWS CLI v2 with credentials that can create IAM roles and DevOps Agent resources
31+
- One or more S3 bucket names where your Spark event logs, driver / executor stdout / stderr, or application code live — you'll pass these as `SparkArtifactsBucketNames`
32+
33+
## Step 1 — Deploy
34+
35+
```bash
36+
aws cloudformation deploy \
37+
--stack-name spark-troubleshooting-mcp-connect \
38+
--template-file cloudformation/integration.yaml \
39+
--capabilities CAPABILITY_NAMED_IAM \
40+
--parameter-overrides \
41+
SparkArtifactsBucketNames=my-spark-logs,my-spark-code
42+
```
43+
44+
Deploy takes about 30 seconds.
45+
46+
## Step 2 — Capture the outputs
47+
48+
```bash
49+
aws cloudformation describe-stacks \
50+
--stack-name spark-troubleshooting-mcp-connect \
51+
--query 'Stacks[0].Outputs'
52+
```
53+
54+
Note the `ServiceArn` — you'll reference it in Step 3.
55+
56+
## Step 3 — Attach the Service to your Agent Space
57+
58+
The template registers the Service (capability provider), but **does not** attach it to your Agent Space. In the AWS DevOps Agent console:
59+
60+
1. Open your Agent Space.
61+
2. Under **Capability Providers**, choose **Add**.
62+
3. Select the Service named `spark-troubleshooting` (or whatever value you passed for `ServiceName`).
63+
4. Save.
64+
65+
See the [DevOps Agent MCP registration guide](https://docs.aws.amazon.com/devopsagent/latest/userguide/configuring-integrations-and-knowledge-connecting-mcp-servers.html) for details.
66+
67+
## Parameters
68+
69+
| Parameter | Required | Default | Notes |
70+
|---|---|---|---|
71+
| `TroubleshootingRoleName` | | `SparkTroubleshootingRole` | IAM role name |
72+
| `ServiceName` | | `spark-troubleshooting` | Capability provider name; unique per account per region |
73+
| `EnableEMREC2` | | `true` | Attach EMR-on-EC2 read permissions |
74+
| `EnableEMRServerless` | | `true` | Attach EMR Serverless read permissions |
75+
| `EnableGlue` | | `true` | Attach Glue read permissions |
76+
| `SparkArtifactsBucketNames` | ✅ | | Comma-delimited S3 bucket **names** (not ARNs) — where Spark logs / code live |
77+
| `CloudWatchKmsKeyArn` | | `''` | Optional — KMS key ARN if your log groups are CMK-encrypted |
78+
79+
## IAM permissions summary
80+
81+
Inline policies attached to the role (conditional on the toggles above):
82+
83+
| Policy | Scope |
84+
|---|---|
85+
| `MCPInvokeAccess` | `sagemaker-unified-studio-mcp:InvokeMcp`, `CallReadOnlyTool`, `CallPrivilegedTool` (service actions have no ARN form) |
86+
| `SparkArtifactsS3Access` | `s3:GetObject`, `s3:GetObjectVersion`, `s3:ListBucket`, `s3:GetBucketLocation` — scoped to the buckets you name |
87+
| `EMREC2Access` | EMR-on-EC2 describes + persistent-app-UI actions |
88+
| `EMRServerlessAccess` | EMR Serverless describes + CloudWatch Logs read on `/aws/emr-serverless/*` |
89+
| `GlueAccess` | Glue job reads + CloudWatch Logs read on `/aws-glue/*` + Spark web UI actions + `iam:PassRole` scoped to `glue.amazonaws.com` |
90+
| `CloudWatchKmsDecrypt` | `kms:Decrypt`, `DescribeKey` on the KMS key you supply |
91+
92+
The role's trust policy allows only `aidevops.amazonaws.com`, conditioned on `aws:SourceAccount` matching the deployer account and `aws:SourceArn` matching `arn:aws:aidevops:<region>:<account>:service/*` (matches the console-suggested scope).
93+
94+
## Cleanup
95+
96+
If you attached the Service to an Agent Space in Step 3, **detach it first** in the DevOps Agent console — otherwise stack delete fails because the Service is still in use.
97+
98+
```bash
99+
aws cloudformation delete-stack --stack-name spark-troubleshooting-mcp-connect
100+
aws cloudformation wait stack-delete-complete --stack-name spark-troubleshooting-mcp-connect
101+
```
102+
103+
## References
104+
105+
- [Apache Spark Troubleshooting Agent for Amazon EMR — setup](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/spark-troubleshooting-agent-setup.html)
106+
- [Connecting MCP servers to AWS DevOps Agent](https://docs.aws.amazon.com/devopsagent/latest/userguide/configuring-integrations-and-knowledge-connecting-mcp-servers.html)
107+
- [AWS Big Data Blog — Accelerate Apache Spark debugging on Amazon EMR with AWS DevOps Agent](https://aws.amazon.com/blogs/big-data/accelerate-apache-spark-debugging-on-amazon-emr-with-aws-devops-agent/)

0 commit comments

Comments
 (0)