From 709f6192cdb6bd224c05b6365cedea3866f0dbdd Mon Sep 17 00:00:00 2001 From: Evgeni Sokolov Date: Fri, 12 Jun 2026 15:44:53 +0200 Subject: [PATCH 1/2] Add prompt caching and latest-model support to latency benchmarking tool Enhances model-latency-benchmarking/ (refs #713): - Add opt-in, per-scenario prompt caching (prompt_caching, cache_ttl, cached_context) with a global PROMPT_CACHING default. Inserts a Converse cachePoint after the cached context; extended 1h TTL is applied only for Anthropic models, others use the default duration. - Capture cache metrics (cache_read_input_tokens, cache_write_input_tokens, Cache_Hit_Rate) and split TTFT into cached vs uncached in the analysis, preserving all existing columns and aggregates. - Add a boto3 minimum-version gate and confirm Converse streaming usage. - Fix inference config so configured TEMPERATURE/TOP_P/TOP_K are honored instead of hardcoded; send a single sampling parameter (INFERENCE_SAMPLING) since several models reject temperature and topP together. - Refresh the sample dataset to current models (Claude Opus/Sonnet/Haiku 4.5, Claude 3.7 Sonnet, Amazon Nova Pro) and add a dedicated caching demo dataset with cached/uncached pairs for Claude Haiku 4.5 and Amazon Nova Pro. - Update readme with the new fields, metrics, prerequisites, and caching notes. Existing datasets without the new fields run unchanged. Change is limited to the model-latency-benchmarking/ folder. --- ...aching-demo-prompts-for-benchmarking.jsonl | 4 + .../dummy-prompts-for-benchmarking.jsonl | 10 +- .../latency-benchmarking-tool.ipynb | 486 ++++++++++++++++-- model-latency-benchmarking/readme.md | 109 +++- 4 files changed, 531 insertions(+), 78 deletions(-) create mode 100644 model-latency-benchmarking/caching-demo-prompts-for-benchmarking.jsonl diff --git a/model-latency-benchmarking/caching-demo-prompts-for-benchmarking.jsonl b/model-latency-benchmarking/caching-demo-prompts-for-benchmarking.jsonl new file mode 100644 index 000000000..7dd029372 --- /dev/null +++ b/model-latency-benchmarking/caching-demo-prompts-for-benchmarking.jsonl @@ -0,0 +1,4 @@ +{"text_prompt": "Using the reference document above, summarize the six pillars of the AWS Well-Architected Framework and explain how prompt caching reduces latency and input-token cost for workloads that reuse a large static context.", "expected_output_tokens": 200, "task_type": "Text-Generation", "model_id": "us.anthropic.claude-haiku-4-5-20251001-v1:0", "region": "us-east-1", "inference_profile": "standard", "prompt_caching": true, "cache_ttl": "5m", "cached_context": "## AWS Well-Architected Framework Overview\nThe AWS Well-Architected Framework helps cloud architects build secure, high-performing, resilient, and efficient infrastructure for a variety of applications and workloads. The framework is organized around six pillars: operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability. Each pillar contains a set of design principles and best practices that teams can apply when designing and operating systems in the cloud. The framework is not a checklist to be completed once; it is a continuous mechanism for reviewing workloads, identifying risks, and prioritizing remediation. Architects use the framework during design reviews, before major launches, and at regular intervals after a workload reaches production so that decisions stay aligned with evolving business goals and customer needs.\n\n## Operational Excellence Pillar\nThe operational excellence pillar focuses on running and monitoring systems to deliver business value and continually improving supporting processes and procedures. Key topics include organizing teams around business outcomes, implementing observability across the stack, automating responses to events, and performing frequent, small, reversible changes. Teams that practice operational excellence define runbooks for routine operations and playbooks for unexpected events. They instrument their applications with structured logs, metrics, and traces, and they review operational metrics in regular operational readiness reviews. Game days and failure injection exercises help teams validate that their procedures work under realistic conditions before a genuine incident occurs. The goal is to learn from every operational event and feed those lessons back into design and process.\n\n## Security Pillar\nThe security pillar describes how to protect data, systems, and assets to take advantage of cloud technologies to improve security. Core practices include implementing a strong identity foundation, applying least privilege, enabling traceability, applying security at all layers, automating security best practices, protecting data in transit and at rest, keeping people away from raw data, and preparing for security events. Identity and access management is central: every principal should have only the permissions it needs, and those permissions should be reviewed regularly. Encryption keys should be managed with a dedicated key management service, and sensitive material should never be embedded in source code or configuration files. Detective controls such as centralized logging and continuous monitoring allow teams to detect anomalies and respond before they escalate into incidents.\n\n## Reliability Pillar\nThe reliability pillar ensures a workload performs its intended function correctly and consistently when it is expected to. This includes the ability to operate and test the workload through its total lifecycle. Reliability practices cover foundations such as managing service quotas and network topology, workload architecture that isolates faults, change management that uses automation to deploy and roll back safely, and failure management that anticipates, withstands, and recovers from disruptions. Designing for reliability means assuming that components will fail and building systems that degrade gracefully. Techniques such as health checks, automatic failover across availability zones, backups with tested restores, and well-defined recovery time and recovery point objectives are foundational. Reliability is verified through testing, not assumed from design.\n\n## Performance Efficiency Pillar\nThe performance efficiency pillar focuses on using computing resources efficiently to meet system requirements and maintaining that efficiency as demand changes and technologies evolve. It encourages teams to democratize advanced technologies, go global in minutes, use serverless architectures, experiment more often, and consider mechanical sympathy when selecting resources. Choosing the right instance families, storage classes, and database engines for a workload can dramatically change both performance and cost. Caching is a recurring theme: caching static content at the edge, caching database query results, and caching expensive computations all reduce latency and load. Teams should benchmark candidate configurations with representative traffic rather than relying on intuition, and they should re-evaluate choices as new instance types and managed services become available.\n\n## Cost Optimization Pillar\nThe cost optimization pillar helps teams avoid unnecessary costs. It includes understanding spending over time, selecting the most cost-effective resources, matching supply with demand, and continuously analyzing and attributing expenditure. Practices include adopting a consumption model so you pay only for what you use, using managed services to reduce the cost of ownership, and applying commitment-based discounts where usage is predictable. Tagging resources consistently allows costs to be attributed to teams, products, and environments, which in turn drives accountability. Right-sizing compute, deleting idle resources, and moving infrequently accessed data to cheaper storage tiers are common, high-impact optimizations. Cost optimization is a continuous activity that pairs well with operational metrics and forecasting.\n\n## Sustainability Pillar\nThe sustainability pillar focuses on minimizing the environmental impact of running cloud workloads. It encourages teams to understand their impact, establish sustainability goals, maximize utilization, anticipate and adopt new more efficient hardware and software offerings, use managed services, and reduce the downstream impact of cloud workloads. Maximizing utilization means running fewer, busier resources rather than many idle ones. Choosing regions and scheduling flexible workloads thoughtfully can also reduce impact. The pillar treats sustainability as a shared responsibility between the cloud provider, which improves the efficiency of the underlying infrastructure, and the customer, which improves the efficiency of the workloads that run on it.\n\n## Prompt Caching on Amazon Bedrock\nPrompt caching is an Amazon Bedrock feature that lets applications cache a static prompt prefix so that repeated requests reusing that prefix can skip reprocessing it. This reduces time to first token and lowers input-token cost for workloads that send a large, stable context on every request, such as a system prompt, a long set of instructions, a knowledge document, or few-shot examples. To use prompt caching with the Converse API, the request places the static content first and then inserts a cache checkpoint block immediately after the content that should be cached. On a cache hit, the response usage block reports cache read input tokens; on the write that populates the cache, it reports cache write input tokens. Each model has a minimum number of tokens that the cached prefix must contain before caching is triggered, and the cache entry has a time to live after which it expires. Prompt caching is available for on-demand invocation and is most effective when the same large prefix is reused many times within the cache lifetime.\n\n## Latency Benchmarking Methodology\nBenchmarking model latency requires careful, repeatable methodology. Time to first token, sometimes called time to first byte, measures how quickly the model begins returning output and is heavily influenced by input length and any cached prefix. Time to last token measures the full response duration. Output tokens per second is computed from the output token count divided by the total streaming time. To produce trustworthy numbers, run many invocations per scenario, report percentiles such as the median, the ninetieth percentile, and the standard deviation rather than a single average, and hold inference parameters constant across compared scenarios. When comparing cached and uncached behavior, keep the model, region, prompt, and inference configuration identical so that the cache is the only variable. Warm the cache with an initial request before measuring cache hit latency, because the first request pays the write cost while later requests enjoy the read benefit.\n\n## Inference Parameters and Their Effects\nTemperature, top-p, and top-k shape how a model samples its next token. Temperature scales the probability distribution: lower values make output more deterministic and focused, while higher values make it more diverse and creative. Top-p, also called nucleus sampling, restricts sampling to the smallest set of tokens whose cumulative probability exceeds the configured threshold. Top-k restricts sampling to the k most likely tokens. For reproducible benchmarks, these parameters must be set explicitly and held constant, because letting them vary or default silently makes results impossible to compare. Not every model supports every parameter; top-k in particular is passed through an additional model request fields channel and should be omitted for models that do not accept it, to avoid validation errors.\n\n## AWS Well-Architected Framework Overview (Detailed Guidance, Part 2)\nThe AWS Well-Architected Framework helps cloud architects build secure, high-performing, resilient, and efficient infrastructure for a variety of applications and workloads. The framework is organized around six pillars: operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability. Each pillar contains a set of design principles and best practices that teams can apply when designing and operating systems in the cloud. The framework is not a checklist to be completed once; it is a continuous mechanism for reviewing workloads, identifying risks, and prioritizing remediation. Architects use the framework during design reviews, before major launches, and at regular intervals after a workload reaches production so that decisions stay aligned with evolving business goals and customer needs.\n\n## Operational Excellence Pillar (Detailed Guidance, Part 2)\nThe operational excellence pillar focuses on running and monitoring systems to deliver business value and continually improving supporting processes and procedures. Key topics include organizing teams around business outcomes, implementing observability across the stack, automating responses to events, and performing frequent, small, reversible changes. Teams that practice operational excellence define runbooks for routine operations and playbooks for unexpected events. They instrument their applications with structured logs, metrics, and traces, and they review operational metrics in regular operational readiness reviews. Game days and failure injection exercises help teams validate that their procedures work under realistic conditions before a genuine incident occurs. The goal is to learn from every operational event and feed those lessons back into design and process.\n\n## Security Pillar (Detailed Guidance, Part 2)\nThe security pillar describes how to protect data, systems, and assets to take advantage of cloud technologies to improve security. Core practices include implementing a strong identity foundation, applying least privilege, enabling traceability, applying security at all layers, automating security best practices, protecting data in transit and at rest, keeping people away from raw data, and preparing for security events. Identity and access management is central: every principal should have only the permissions it needs, and those permissions should be reviewed regularly. Encryption keys should be managed with a dedicated key management service, and sensitive material should never be embedded in source code or configuration files. Detective controls such as centralized logging and continuous monitoring allow teams to detect anomalies and respond before they escalate into incidents.\n\n## Reliability Pillar (Detailed Guidance, Part 2)\nThe reliability pillar ensures a workload performs its intended function correctly and consistently when it is expected to. This includes the ability to operate and test the workload through its total lifecycle. Reliability practices cover foundations such as managing service quotas and network topology, workload architecture that isolates faults, change management that uses automation to deploy and roll back safely, and failure management that anticipates, withstands, and recovers from disruptions. Designing for reliability means assuming that components will fail and building systems that degrade gracefully. Techniques such as health checks, automatic failover across availability zones, backups with tested restores, and well-defined recovery time and recovery point objectives are foundational. Reliability is verified through testing, not assumed from design.\n\n## Performance Efficiency Pillar (Detailed Guidance, Part 2)\nThe performance efficiency pillar focuses on using computing resources efficiently to meet system requirements and maintaining that efficiency as demand changes and technologies evolve. It encourages teams to democratize advanced technologies, go global in minutes, use serverless architectures, experiment more often, and consider mechanical sympathy when selecting resources. Choosing the right instance families, storage classes, and database engines for a workload can dramatically change both performance and cost. Caching is a recurring theme: caching static content at the edge, caching database query results, and caching expensive computations all reduce latency and load. Teams should benchmark candidate configurations with representative traffic rather than relying on intuition, and they should re-evaluate choices as new instance types and managed services become available.\n\n## Cost Optimization Pillar (Detailed Guidance, Part 2)\nThe cost optimization pillar helps teams avoid unnecessary costs. It includes understanding spending over time, selecting the most cost-effective resources, matching supply with demand, and continuously analyzing and attributing expenditure. Practices include adopting a consumption model so you pay only for what you use, using managed services to reduce the cost of ownership, and applying commitment-based discounts where usage is predictable. Tagging resources consistently allows costs to be attributed to teams, products, and environments, which in turn drives accountability. Right-sizing compute, deleting idle resources, and moving infrequently accessed data to cheaper storage tiers are common, high-impact optimizations. Cost optimization is a continuous activity that pairs well with operational metrics and forecasting.\n\n## Sustainability Pillar (Detailed Guidance, Part 2)\nThe sustainability pillar focuses on minimizing the environmental impact of running cloud workloads. It encourages teams to understand their impact, establish sustainability goals, maximize utilization, anticipate and adopt new more efficient hardware and software offerings, use managed services, and reduce the downstream impact of cloud workloads. Maximizing utilization means running fewer, busier resources rather than many idle ones. Choosing regions and scheduling flexible workloads thoughtfully can also reduce impact. The pillar treats sustainability as a shared responsibility between the cloud provider, which improves the efficiency of the underlying infrastructure, and the customer, which improves the efficiency of the workloads that run on it.\n\n## Prompt Caching on Amazon Bedrock (Detailed Guidance, Part 2)\nPrompt caching is an Amazon Bedrock feature that lets applications cache a static prompt prefix so that repeated requests reusing that prefix can skip reprocessing it. This reduces time to first token and lowers input-token cost for workloads that send a large, stable context on every request, such as a system prompt, a long set of instructions, a knowledge document, or few-shot examples. To use prompt caching with the Converse API, the request places the static content first and then inserts a cache checkpoint block immediately after the content that should be cached. On a cache hit, the response usage block reports cache read input tokens; on the write that populates the cache, it reports cache write input tokens. Each model has a minimum number of tokens that the cached prefix must contain before caching is triggered, and the cache entry has a time to live after which it expires. Prompt caching is available for on-demand invocation and is most effective when the same large prefix is reused many times within the cache lifetime.\n\n## Latency Benchmarking Methodology (Detailed Guidance, Part 2)\nBenchmarking model latency requires careful, repeatable methodology. Time to first token, sometimes called time to first byte, measures how quickly the model begins returning output and is heavily influenced by input length and any cached prefix. Time to last token measures the full response duration. Output tokens per second is computed from the output token count divided by the total streaming time. To produce trustworthy numbers, run many invocations per scenario, report percentiles such as the median, the ninetieth percentile, and the standard deviation rather than a single average, and hold inference parameters constant across compared scenarios. When comparing cached and uncached behavior, keep the model, region, prompt, and inference configuration identical so that the cache is the only variable. Warm the cache with an initial request before measuring cache hit latency, because the first request pays the write cost while later requests enjoy the read benefit.\n\n## Inference Parameters and Their Effects (Detailed Guidance, Part 2)\nTemperature, top-p, and top-k shape how a model samples its next token. Temperature scales the probability distribution: lower values make output more deterministic and focused, while higher values make it more diverse and creative. Top-p, also called nucleus sampling, restricts sampling to the smallest set of tokens whose cumulative probability exceeds the configured threshold. Top-k restricts sampling to the k most likely tokens. For reproducible benchmarks, these parameters must be set explicitly and held constant, because letting them vary or default silently makes results impossible to compare. Not every model supports every parameter; top-k in particular is passed through an additional model request fields channel and should be omitted for models that do not accept it, to avoid validation errors.\n\n## AWS Well-Architected Framework Overview (Detailed Guidance, Part 3)\nThe AWS Well-Architected Framework helps cloud architects build secure, high-performing, resilient, and efficient infrastructure for a variety of applications and workloads. The framework is organized around six pillars: operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability. Each pillar contains a set of design principles and best practices that teams can apply when designing and operating systems in the cloud. The framework is not a checklist to be completed once; it is a continuous mechanism for reviewing workloads, identifying risks, and prioritizing remediation. Architects use the framework during design reviews, before major launches, and at regular intervals after a workload reaches production so that decisions stay aligned with evolving business goals and customer needs.\n\n## Operational Excellence Pillar (Detailed Guidance, Part 3)\nThe operational excellence pillar focuses on running and monitoring systems to deliver business value and continually improving supporting processes and procedures. Key topics include organizing teams around business outcomes, implementing observability across the stack, automating responses to events, and performing frequent, small, reversible changes. Teams that practice operational excellence define runbooks for routine operations and playbooks for unexpected events. They instrument their applications with structured logs, metrics, and traces, and they review operational metrics in regular operational readiness reviews. Game days and failure injection exercises help teams validate that their procedures work under realistic conditions before a genuine incident occurs. The goal is to learn from every operational event and feed those lessons back into design and process.\n\n## Security Pillar (Detailed Guidance, Part 3)\nThe security pillar describes how to protect data, systems, and assets to take advantage of cloud technologies to improve security. Core practices include implementing a strong identity foundation, applying least privilege, enabling traceability, applying security at all layers, automating security best practices, protecting data in transit and at rest, keeping people away from raw data, and preparing for security events. Identity and access management is central: every principal should have only the permissions it needs, and those permissions should be reviewed regularly. Encryption keys should be managed with a dedicated key management service, and sensitive material should never be embedded in source code or configuration files. Detective controls such as centralized logging and continuous monitoring allow teams to detect anomalies and respond before they escalate into incidents.\n\n## Reliability Pillar (Detailed Guidance, Part 3)\nThe reliability pillar ensures a workload performs its intended function correctly and consistently when it is expected to. This includes the ability to operate and test the workload through its total lifecycle. Reliability practices cover foundations such as managing service quotas and network topology, workload architecture that isolates faults, change management that uses automation to deploy and roll back safely, and failure management that anticipates, withstands, and recovers from disruptions. Designing for reliability means assuming that components will fail and building systems that degrade gracefully. Techniques such as health checks, automatic failover across availability zones, backups with tested restores, and well-defined recovery time and recovery point objectives are foundational. Reliability is verified through testing, not assumed from design.\n\n## Performance Efficiency Pillar (Detailed Guidance, Part 3)\nThe performance efficiency pillar focuses on using computing resources efficiently to meet system requirements and maintaining that efficiency as demand changes and technologies evolve. It encourages teams to democratize advanced technologies, go global in minutes, use serverless architectures, experiment more often, and consider mechanical sympathy when selecting resources. Choosing the right instance families, storage classes, and database engines for a workload can dramatically change both performance and cost. Caching is a recurring theme: caching static content at the edge, caching database query results, and caching expensive computations all reduce latency and load. Teams should benchmark candidate configurations with representative traffic rather than relying on intuition, and they should re-evaluate choices as new instance types and managed services become available.\n\n## Cost Optimization Pillar (Detailed Guidance, Part 3)\nThe cost optimization pillar helps teams avoid unnecessary costs. It includes understanding spending over time, selecting the most cost-effective resources, matching supply with demand, and continuously analyzing and attributing expenditure. Practices include adopting a consumption model so you pay only for what you use, using managed services to reduce the cost of ownership, and applying commitment-based discounts where usage is predictable. Tagging resources consistently allows costs to be attributed to teams, products, and environments, which in turn drives accountability. Right-sizing compute, deleting idle resources, and moving infrequently accessed data to cheaper storage tiers are common, high-impact optimizations. Cost optimization is a continuous activity that pairs well with operational metrics and forecasting.\n\n## Sustainability Pillar (Detailed Guidance, Part 3)\nThe sustainability pillar focuses on minimizing the environmental impact of running cloud workloads. It encourages teams to understand their impact, establish sustainability goals, maximize utilization, anticipate and adopt new more efficient hardware and software offerings, use managed services, and reduce the downstream impact of cloud workloads. Maximizing utilization means running fewer, busier resources rather than many idle ones. Choosing regions and scheduling flexible workloads thoughtfully can also reduce impact. The pillar treats sustainability as a shared responsibility between the cloud provider, which improves the efficiency of the underlying infrastructure, and the customer, which improves the efficiency of the workloads that run on it.\n\n## Prompt Caching on Amazon Bedrock (Detailed Guidance, Part 3)\nPrompt caching is an Amazon Bedrock feature that lets applications cache a static prompt prefix so that repeated requests reusing that prefix can skip reprocessing it. This reduces time to first token and lowers input-token cost for workloads that send a large, stable context on every request, such as a system prompt, a long set of instructions, a knowledge document, or few-shot examples. To use prompt caching with the Converse API, the request places the static content first and then inserts a cache checkpoint block immediately after the content that should be cached. On a cache hit, the response usage block reports cache read input tokens; on the write that populates the cache, it reports cache write input tokens. Each model has a minimum number of tokens that the cached prefix must contain before caching is triggered, and the cache entry has a time to live after which it expires. Prompt caching is available for on-demand invocation and is most effective when the same large prefix is reused many times within the cache lifetime.\n\n## Latency Benchmarking Methodology (Detailed Guidance, Part 3)\nBenchmarking model latency requires careful, repeatable methodology. Time to first token, sometimes called time to first byte, measures how quickly the model begins returning output and is heavily influenced by input length and any cached prefix. Time to last token measures the full response duration. Output tokens per second is computed from the output token count divided by the total streaming time. To produce trustworthy numbers, run many invocations per scenario, report percentiles such as the median, the ninetieth percentile, and the standard deviation rather than a single average, and hold inference parameters constant across compared scenarios. When comparing cached and uncached behavior, keep the model, region, prompt, and inference configuration identical so that the cache is the only variable. Warm the cache with an initial request before measuring cache hit latency, because the first request pays the write cost while later requests enjoy the read benefit.\n\n## Inference Parameters and Their Effects (Detailed Guidance, Part 3)\nTemperature, top-p, and top-k shape how a model samples its next token. Temperature scales the probability distribution: lower values make output more deterministic and focused, while higher values make it more diverse and creative. Top-p, also called nucleus sampling, restricts sampling to the smallest set of tokens whose cumulative probability exceeds the configured threshold. Top-k restricts sampling to the k most likely tokens. For reproducible benchmarks, these parameters must be set explicitly and held constant, because letting them vary or default silently makes results impossible to compare. Not every model supports every parameter; top-k in particular is passed through an additional model request fields channel and should be omitted for models that do not accept it, to avoid validation errors."} +{"text_prompt": "Using the reference document above, summarize the six pillars of the AWS Well-Architected Framework and explain how prompt caching reduces latency and input-token cost for workloads that reuse a large static context.", "expected_output_tokens": 200, "task_type": "Text-Generation", "model_id": "us.anthropic.claude-haiku-4-5-20251001-v1:0", "region": "us-east-1", "inference_profile": "standard", "prompt_caching": false, "cache_ttl": "5m", "cached_context": "## AWS Well-Architected Framework Overview\nThe AWS Well-Architected Framework helps cloud architects build secure, high-performing, resilient, and efficient infrastructure for a variety of applications and workloads. The framework is organized around six pillars: operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability. Each pillar contains a set of design principles and best practices that teams can apply when designing and operating systems in the cloud. The framework is not a checklist to be completed once; it is a continuous mechanism for reviewing workloads, identifying risks, and prioritizing remediation. Architects use the framework during design reviews, before major launches, and at regular intervals after a workload reaches production so that decisions stay aligned with evolving business goals and customer needs.\n\n## Operational Excellence Pillar\nThe operational excellence pillar focuses on running and monitoring systems to deliver business value and continually improving supporting processes and procedures. Key topics include organizing teams around business outcomes, implementing observability across the stack, automating responses to events, and performing frequent, small, reversible changes. Teams that practice operational excellence define runbooks for routine operations and playbooks for unexpected events. They instrument their applications with structured logs, metrics, and traces, and they review operational metrics in regular operational readiness reviews. Game days and failure injection exercises help teams validate that their procedures work under realistic conditions before a genuine incident occurs. The goal is to learn from every operational event and feed those lessons back into design and process.\n\n## Security Pillar\nThe security pillar describes how to protect data, systems, and assets to take advantage of cloud technologies to improve security. Core practices include implementing a strong identity foundation, applying least privilege, enabling traceability, applying security at all layers, automating security best practices, protecting data in transit and at rest, keeping people away from raw data, and preparing for security events. Identity and access management is central: every principal should have only the permissions it needs, and those permissions should be reviewed regularly. Encryption keys should be managed with a dedicated key management service, and sensitive material should never be embedded in source code or configuration files. Detective controls such as centralized logging and continuous monitoring allow teams to detect anomalies and respond before they escalate into incidents.\n\n## Reliability Pillar\nThe reliability pillar ensures a workload performs its intended function correctly and consistently when it is expected to. This includes the ability to operate and test the workload through its total lifecycle. Reliability practices cover foundations such as managing service quotas and network topology, workload architecture that isolates faults, change management that uses automation to deploy and roll back safely, and failure management that anticipates, withstands, and recovers from disruptions. Designing for reliability means assuming that components will fail and building systems that degrade gracefully. Techniques such as health checks, automatic failover across availability zones, backups with tested restores, and well-defined recovery time and recovery point objectives are foundational. Reliability is verified through testing, not assumed from design.\n\n## Performance Efficiency Pillar\nThe performance efficiency pillar focuses on using computing resources efficiently to meet system requirements and maintaining that efficiency as demand changes and technologies evolve. It encourages teams to democratize advanced technologies, go global in minutes, use serverless architectures, experiment more often, and consider mechanical sympathy when selecting resources. Choosing the right instance families, storage classes, and database engines for a workload can dramatically change both performance and cost. Caching is a recurring theme: caching static content at the edge, caching database query results, and caching expensive computations all reduce latency and load. Teams should benchmark candidate configurations with representative traffic rather than relying on intuition, and they should re-evaluate choices as new instance types and managed services become available.\n\n## Cost Optimization Pillar\nThe cost optimization pillar helps teams avoid unnecessary costs. It includes understanding spending over time, selecting the most cost-effective resources, matching supply with demand, and continuously analyzing and attributing expenditure. Practices include adopting a consumption model so you pay only for what you use, using managed services to reduce the cost of ownership, and applying commitment-based discounts where usage is predictable. Tagging resources consistently allows costs to be attributed to teams, products, and environments, which in turn drives accountability. Right-sizing compute, deleting idle resources, and moving infrequently accessed data to cheaper storage tiers are common, high-impact optimizations. Cost optimization is a continuous activity that pairs well with operational metrics and forecasting.\n\n## Sustainability Pillar\nThe sustainability pillar focuses on minimizing the environmental impact of running cloud workloads. It encourages teams to understand their impact, establish sustainability goals, maximize utilization, anticipate and adopt new more efficient hardware and software offerings, use managed services, and reduce the downstream impact of cloud workloads. Maximizing utilization means running fewer, busier resources rather than many idle ones. Choosing regions and scheduling flexible workloads thoughtfully can also reduce impact. The pillar treats sustainability as a shared responsibility between the cloud provider, which improves the efficiency of the underlying infrastructure, and the customer, which improves the efficiency of the workloads that run on it.\n\n## Prompt Caching on Amazon Bedrock\nPrompt caching is an Amazon Bedrock feature that lets applications cache a static prompt prefix so that repeated requests reusing that prefix can skip reprocessing it. This reduces time to first token and lowers input-token cost for workloads that send a large, stable context on every request, such as a system prompt, a long set of instructions, a knowledge document, or few-shot examples. To use prompt caching with the Converse API, the request places the static content first and then inserts a cache checkpoint block immediately after the content that should be cached. On a cache hit, the response usage block reports cache read input tokens; on the write that populates the cache, it reports cache write input tokens. Each model has a minimum number of tokens that the cached prefix must contain before caching is triggered, and the cache entry has a time to live after which it expires. Prompt caching is available for on-demand invocation and is most effective when the same large prefix is reused many times within the cache lifetime.\n\n## Latency Benchmarking Methodology\nBenchmarking model latency requires careful, repeatable methodology. Time to first token, sometimes called time to first byte, measures how quickly the model begins returning output and is heavily influenced by input length and any cached prefix. Time to last token measures the full response duration. Output tokens per second is computed from the output token count divided by the total streaming time. To produce trustworthy numbers, run many invocations per scenario, report percentiles such as the median, the ninetieth percentile, and the standard deviation rather than a single average, and hold inference parameters constant across compared scenarios. When comparing cached and uncached behavior, keep the model, region, prompt, and inference configuration identical so that the cache is the only variable. Warm the cache with an initial request before measuring cache hit latency, because the first request pays the write cost while later requests enjoy the read benefit.\n\n## Inference Parameters and Their Effects\nTemperature, top-p, and top-k shape how a model samples its next token. Temperature scales the probability distribution: lower values make output more deterministic and focused, while higher values make it more diverse and creative. Top-p, also called nucleus sampling, restricts sampling to the smallest set of tokens whose cumulative probability exceeds the configured threshold. Top-k restricts sampling to the k most likely tokens. For reproducible benchmarks, these parameters must be set explicitly and held constant, because letting them vary or default silently makes results impossible to compare. Not every model supports every parameter; top-k in particular is passed through an additional model request fields channel and should be omitted for models that do not accept it, to avoid validation errors.\n\n## AWS Well-Architected Framework Overview (Detailed Guidance, Part 2)\nThe AWS Well-Architected Framework helps cloud architects build secure, high-performing, resilient, and efficient infrastructure for a variety of applications and workloads. The framework is organized around six pillars: operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability. Each pillar contains a set of design principles and best practices that teams can apply when designing and operating systems in the cloud. The framework is not a checklist to be completed once; it is a continuous mechanism for reviewing workloads, identifying risks, and prioritizing remediation. Architects use the framework during design reviews, before major launches, and at regular intervals after a workload reaches production so that decisions stay aligned with evolving business goals and customer needs.\n\n## Operational Excellence Pillar (Detailed Guidance, Part 2)\nThe operational excellence pillar focuses on running and monitoring systems to deliver business value and continually improving supporting processes and procedures. Key topics include organizing teams around business outcomes, implementing observability across the stack, automating responses to events, and performing frequent, small, reversible changes. Teams that practice operational excellence define runbooks for routine operations and playbooks for unexpected events. They instrument their applications with structured logs, metrics, and traces, and they review operational metrics in regular operational readiness reviews. Game days and failure injection exercises help teams validate that their procedures work under realistic conditions before a genuine incident occurs. The goal is to learn from every operational event and feed those lessons back into design and process.\n\n## Security Pillar (Detailed Guidance, Part 2)\nThe security pillar describes how to protect data, systems, and assets to take advantage of cloud technologies to improve security. Core practices include implementing a strong identity foundation, applying least privilege, enabling traceability, applying security at all layers, automating security best practices, protecting data in transit and at rest, keeping people away from raw data, and preparing for security events. Identity and access management is central: every principal should have only the permissions it needs, and those permissions should be reviewed regularly. Encryption keys should be managed with a dedicated key management service, and sensitive material should never be embedded in source code or configuration files. Detective controls such as centralized logging and continuous monitoring allow teams to detect anomalies and respond before they escalate into incidents.\n\n## Reliability Pillar (Detailed Guidance, Part 2)\nThe reliability pillar ensures a workload performs its intended function correctly and consistently when it is expected to. This includes the ability to operate and test the workload through its total lifecycle. Reliability practices cover foundations such as managing service quotas and network topology, workload architecture that isolates faults, change management that uses automation to deploy and roll back safely, and failure management that anticipates, withstands, and recovers from disruptions. Designing for reliability means assuming that components will fail and building systems that degrade gracefully. Techniques such as health checks, automatic failover across availability zones, backups with tested restores, and well-defined recovery time and recovery point objectives are foundational. Reliability is verified through testing, not assumed from design.\n\n## Performance Efficiency Pillar (Detailed Guidance, Part 2)\nThe performance efficiency pillar focuses on using computing resources efficiently to meet system requirements and maintaining that efficiency as demand changes and technologies evolve. It encourages teams to democratize advanced technologies, go global in minutes, use serverless architectures, experiment more often, and consider mechanical sympathy when selecting resources. Choosing the right instance families, storage classes, and database engines for a workload can dramatically change both performance and cost. Caching is a recurring theme: caching static content at the edge, caching database query results, and caching expensive computations all reduce latency and load. Teams should benchmark candidate configurations with representative traffic rather than relying on intuition, and they should re-evaluate choices as new instance types and managed services become available.\n\n## Cost Optimization Pillar (Detailed Guidance, Part 2)\nThe cost optimization pillar helps teams avoid unnecessary costs. It includes understanding spending over time, selecting the most cost-effective resources, matching supply with demand, and continuously analyzing and attributing expenditure. Practices include adopting a consumption model so you pay only for what you use, using managed services to reduce the cost of ownership, and applying commitment-based discounts where usage is predictable. Tagging resources consistently allows costs to be attributed to teams, products, and environments, which in turn drives accountability. Right-sizing compute, deleting idle resources, and moving infrequently accessed data to cheaper storage tiers are common, high-impact optimizations. Cost optimization is a continuous activity that pairs well with operational metrics and forecasting.\n\n## Sustainability Pillar (Detailed Guidance, Part 2)\nThe sustainability pillar focuses on minimizing the environmental impact of running cloud workloads. It encourages teams to understand their impact, establish sustainability goals, maximize utilization, anticipate and adopt new more efficient hardware and software offerings, use managed services, and reduce the downstream impact of cloud workloads. Maximizing utilization means running fewer, busier resources rather than many idle ones. Choosing regions and scheduling flexible workloads thoughtfully can also reduce impact. The pillar treats sustainability as a shared responsibility between the cloud provider, which improves the efficiency of the underlying infrastructure, and the customer, which improves the efficiency of the workloads that run on it.\n\n## Prompt Caching on Amazon Bedrock (Detailed Guidance, Part 2)\nPrompt caching is an Amazon Bedrock feature that lets applications cache a static prompt prefix so that repeated requests reusing that prefix can skip reprocessing it. This reduces time to first token and lowers input-token cost for workloads that send a large, stable context on every request, such as a system prompt, a long set of instructions, a knowledge document, or few-shot examples. To use prompt caching with the Converse API, the request places the static content first and then inserts a cache checkpoint block immediately after the content that should be cached. On a cache hit, the response usage block reports cache read input tokens; on the write that populates the cache, it reports cache write input tokens. Each model has a minimum number of tokens that the cached prefix must contain before caching is triggered, and the cache entry has a time to live after which it expires. Prompt caching is available for on-demand invocation and is most effective when the same large prefix is reused many times within the cache lifetime.\n\n## Latency Benchmarking Methodology (Detailed Guidance, Part 2)\nBenchmarking model latency requires careful, repeatable methodology. Time to first token, sometimes called time to first byte, measures how quickly the model begins returning output and is heavily influenced by input length and any cached prefix. Time to last token measures the full response duration. Output tokens per second is computed from the output token count divided by the total streaming time. To produce trustworthy numbers, run many invocations per scenario, report percentiles such as the median, the ninetieth percentile, and the standard deviation rather than a single average, and hold inference parameters constant across compared scenarios. When comparing cached and uncached behavior, keep the model, region, prompt, and inference configuration identical so that the cache is the only variable. Warm the cache with an initial request before measuring cache hit latency, because the first request pays the write cost while later requests enjoy the read benefit.\n\n## Inference Parameters and Their Effects (Detailed Guidance, Part 2)\nTemperature, top-p, and top-k shape how a model samples its next token. Temperature scales the probability distribution: lower values make output more deterministic and focused, while higher values make it more diverse and creative. Top-p, also called nucleus sampling, restricts sampling to the smallest set of tokens whose cumulative probability exceeds the configured threshold. Top-k restricts sampling to the k most likely tokens. For reproducible benchmarks, these parameters must be set explicitly and held constant, because letting them vary or default silently makes results impossible to compare. Not every model supports every parameter; top-k in particular is passed through an additional model request fields channel and should be omitted for models that do not accept it, to avoid validation errors.\n\n## AWS Well-Architected Framework Overview (Detailed Guidance, Part 3)\nThe AWS Well-Architected Framework helps cloud architects build secure, high-performing, resilient, and efficient infrastructure for a variety of applications and workloads. The framework is organized around six pillars: operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability. Each pillar contains a set of design principles and best practices that teams can apply when designing and operating systems in the cloud. The framework is not a checklist to be completed once; it is a continuous mechanism for reviewing workloads, identifying risks, and prioritizing remediation. Architects use the framework during design reviews, before major launches, and at regular intervals after a workload reaches production so that decisions stay aligned with evolving business goals and customer needs.\n\n## Operational Excellence Pillar (Detailed Guidance, Part 3)\nThe operational excellence pillar focuses on running and monitoring systems to deliver business value and continually improving supporting processes and procedures. Key topics include organizing teams around business outcomes, implementing observability across the stack, automating responses to events, and performing frequent, small, reversible changes. Teams that practice operational excellence define runbooks for routine operations and playbooks for unexpected events. They instrument their applications with structured logs, metrics, and traces, and they review operational metrics in regular operational readiness reviews. Game days and failure injection exercises help teams validate that their procedures work under realistic conditions before a genuine incident occurs. The goal is to learn from every operational event and feed those lessons back into design and process.\n\n## Security Pillar (Detailed Guidance, Part 3)\nThe security pillar describes how to protect data, systems, and assets to take advantage of cloud technologies to improve security. Core practices include implementing a strong identity foundation, applying least privilege, enabling traceability, applying security at all layers, automating security best practices, protecting data in transit and at rest, keeping people away from raw data, and preparing for security events. Identity and access management is central: every principal should have only the permissions it needs, and those permissions should be reviewed regularly. Encryption keys should be managed with a dedicated key management service, and sensitive material should never be embedded in source code or configuration files. Detective controls such as centralized logging and continuous monitoring allow teams to detect anomalies and respond before they escalate into incidents.\n\n## Reliability Pillar (Detailed Guidance, Part 3)\nThe reliability pillar ensures a workload performs its intended function correctly and consistently when it is expected to. This includes the ability to operate and test the workload through its total lifecycle. Reliability practices cover foundations such as managing service quotas and network topology, workload architecture that isolates faults, change management that uses automation to deploy and roll back safely, and failure management that anticipates, withstands, and recovers from disruptions. Designing for reliability means assuming that components will fail and building systems that degrade gracefully. Techniques such as health checks, automatic failover across availability zones, backups with tested restores, and well-defined recovery time and recovery point objectives are foundational. Reliability is verified through testing, not assumed from design.\n\n## Performance Efficiency Pillar (Detailed Guidance, Part 3)\nThe performance efficiency pillar focuses on using computing resources efficiently to meet system requirements and maintaining that efficiency as demand changes and technologies evolve. It encourages teams to democratize advanced technologies, go global in minutes, use serverless architectures, experiment more often, and consider mechanical sympathy when selecting resources. Choosing the right instance families, storage classes, and database engines for a workload can dramatically change both performance and cost. Caching is a recurring theme: caching static content at the edge, caching database query results, and caching expensive computations all reduce latency and load. Teams should benchmark candidate configurations with representative traffic rather than relying on intuition, and they should re-evaluate choices as new instance types and managed services become available.\n\n## Cost Optimization Pillar (Detailed Guidance, Part 3)\nThe cost optimization pillar helps teams avoid unnecessary costs. It includes understanding spending over time, selecting the most cost-effective resources, matching supply with demand, and continuously analyzing and attributing expenditure. Practices include adopting a consumption model so you pay only for what you use, using managed services to reduce the cost of ownership, and applying commitment-based discounts where usage is predictable. Tagging resources consistently allows costs to be attributed to teams, products, and environments, which in turn drives accountability. Right-sizing compute, deleting idle resources, and moving infrequently accessed data to cheaper storage tiers are common, high-impact optimizations. Cost optimization is a continuous activity that pairs well with operational metrics and forecasting.\n\n## Sustainability Pillar (Detailed Guidance, Part 3)\nThe sustainability pillar focuses on minimizing the environmental impact of running cloud workloads. It encourages teams to understand their impact, establish sustainability goals, maximize utilization, anticipate and adopt new more efficient hardware and software offerings, use managed services, and reduce the downstream impact of cloud workloads. Maximizing utilization means running fewer, busier resources rather than many idle ones. Choosing regions and scheduling flexible workloads thoughtfully can also reduce impact. The pillar treats sustainability as a shared responsibility between the cloud provider, which improves the efficiency of the underlying infrastructure, and the customer, which improves the efficiency of the workloads that run on it.\n\n## Prompt Caching on Amazon Bedrock (Detailed Guidance, Part 3)\nPrompt caching is an Amazon Bedrock feature that lets applications cache a static prompt prefix so that repeated requests reusing that prefix can skip reprocessing it. This reduces time to first token and lowers input-token cost for workloads that send a large, stable context on every request, such as a system prompt, a long set of instructions, a knowledge document, or few-shot examples. To use prompt caching with the Converse API, the request places the static content first and then inserts a cache checkpoint block immediately after the content that should be cached. On a cache hit, the response usage block reports cache read input tokens; on the write that populates the cache, it reports cache write input tokens. Each model has a minimum number of tokens that the cached prefix must contain before caching is triggered, and the cache entry has a time to live after which it expires. Prompt caching is available for on-demand invocation and is most effective when the same large prefix is reused many times within the cache lifetime.\n\n## Latency Benchmarking Methodology (Detailed Guidance, Part 3)\nBenchmarking model latency requires careful, repeatable methodology. Time to first token, sometimes called time to first byte, measures how quickly the model begins returning output and is heavily influenced by input length and any cached prefix. Time to last token measures the full response duration. Output tokens per second is computed from the output token count divided by the total streaming time. To produce trustworthy numbers, run many invocations per scenario, report percentiles such as the median, the ninetieth percentile, and the standard deviation rather than a single average, and hold inference parameters constant across compared scenarios. When comparing cached and uncached behavior, keep the model, region, prompt, and inference configuration identical so that the cache is the only variable. Warm the cache with an initial request before measuring cache hit latency, because the first request pays the write cost while later requests enjoy the read benefit.\n\n## Inference Parameters and Their Effects (Detailed Guidance, Part 3)\nTemperature, top-p, and top-k shape how a model samples its next token. Temperature scales the probability distribution: lower values make output more deterministic and focused, while higher values make it more diverse and creative. Top-p, also called nucleus sampling, restricts sampling to the smallest set of tokens whose cumulative probability exceeds the configured threshold. Top-k restricts sampling to the k most likely tokens. For reproducible benchmarks, these parameters must be set explicitly and held constant, because letting them vary or default silently makes results impossible to compare. Not every model supports every parameter; top-k in particular is passed through an additional model request fields channel and should be omitted for models that do not accept it, to avoid validation errors."} +{"text_prompt": "Using the reference document above, summarize the six pillars of the AWS Well-Architected Framework and explain how prompt caching reduces latency and input-token cost for workloads that reuse a large static context.", "expected_output_tokens": 200, "task_type": "Text-Generation", "model_id": "us.amazon.nova-pro-v1:0", "region": "us-east-1", "inference_profile": "standard", "prompt_caching": true, "cache_ttl": "5m", "cached_context": "## AWS Well-Architected Framework Overview\nThe AWS Well-Architected Framework helps cloud architects build secure, high-performing, resilient, and efficient infrastructure for a variety of applications and workloads. The framework is organized around six pillars: operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability. Each pillar contains a set of design principles and best practices that teams can apply when designing and operating systems in the cloud. The framework is not a checklist to be completed once; it is a continuous mechanism for reviewing workloads, identifying risks, and prioritizing remediation. Architects use the framework during design reviews, before major launches, and at regular intervals after a workload reaches production so that decisions stay aligned with evolving business goals and customer needs.\n\n## Operational Excellence Pillar\nThe operational excellence pillar focuses on running and monitoring systems to deliver business value and continually improving supporting processes and procedures. Key topics include organizing teams around business outcomes, implementing observability across the stack, automating responses to events, and performing frequent, small, reversible changes. Teams that practice operational excellence define runbooks for routine operations and playbooks for unexpected events. They instrument their applications with structured logs, metrics, and traces, and they review operational metrics in regular operational readiness reviews. Game days and failure injection exercises help teams validate that their procedures work under realistic conditions before a genuine incident occurs. The goal is to learn from every operational event and feed those lessons back into design and process.\n\n## Security Pillar\nThe security pillar describes how to protect data, systems, and assets to take advantage of cloud technologies to improve security. Core practices include implementing a strong identity foundation, applying least privilege, enabling traceability, applying security at all layers, automating security best practices, protecting data in transit and at rest, keeping people away from raw data, and preparing for security events. Identity and access management is central: every principal should have only the permissions it needs, and those permissions should be reviewed regularly. Encryption keys should be managed with a dedicated key management service, and sensitive material should never be embedded in source code or configuration files. Detective controls such as centralized logging and continuous monitoring allow teams to detect anomalies and respond before they escalate into incidents.\n\n## Reliability Pillar\nThe reliability pillar ensures a workload performs its intended function correctly and consistently when it is expected to. This includes the ability to operate and test the workload through its total lifecycle. Reliability practices cover foundations such as managing service quotas and network topology, workload architecture that isolates faults, change management that uses automation to deploy and roll back safely, and failure management that anticipates, withstands, and recovers from disruptions. Designing for reliability means assuming that components will fail and building systems that degrade gracefully. Techniques such as health checks, automatic failover across availability zones, backups with tested restores, and well-defined recovery time and recovery point objectives are foundational. Reliability is verified through testing, not assumed from design.\n\n## Performance Efficiency Pillar\nThe performance efficiency pillar focuses on using computing resources efficiently to meet system requirements and maintaining that efficiency as demand changes and technologies evolve. It encourages teams to democratize advanced technologies, go global in minutes, use serverless architectures, experiment more often, and consider mechanical sympathy when selecting resources. Choosing the right instance families, storage classes, and database engines for a workload can dramatically change both performance and cost. Caching is a recurring theme: caching static content at the edge, caching database query results, and caching expensive computations all reduce latency and load. Teams should benchmark candidate configurations with representative traffic rather than relying on intuition, and they should re-evaluate choices as new instance types and managed services become available.\n\n## Cost Optimization Pillar\nThe cost optimization pillar helps teams avoid unnecessary costs. It includes understanding spending over time, selecting the most cost-effective resources, matching supply with demand, and continuously analyzing and attributing expenditure. Practices include adopting a consumption model so you pay only for what you use, using managed services to reduce the cost of ownership, and applying commitment-based discounts where usage is predictable. Tagging resources consistently allows costs to be attributed to teams, products, and environments, which in turn drives accountability. Right-sizing compute, deleting idle resources, and moving infrequently accessed data to cheaper storage tiers are common, high-impact optimizations. Cost optimization is a continuous activity that pairs well with operational metrics and forecasting.\n\n## Sustainability Pillar\nThe sustainability pillar focuses on minimizing the environmental impact of running cloud workloads. It encourages teams to understand their impact, establish sustainability goals, maximize utilization, anticipate and adopt new more efficient hardware and software offerings, use managed services, and reduce the downstream impact of cloud workloads. Maximizing utilization means running fewer, busier resources rather than many idle ones. Choosing regions and scheduling flexible workloads thoughtfully can also reduce impact. The pillar treats sustainability as a shared responsibility between the cloud provider, which improves the efficiency of the underlying infrastructure, and the customer, which improves the efficiency of the workloads that run on it.\n\n## Prompt Caching on Amazon Bedrock\nPrompt caching is an Amazon Bedrock feature that lets applications cache a static prompt prefix so that repeated requests reusing that prefix can skip reprocessing it. This reduces time to first token and lowers input-token cost for workloads that send a large, stable context on every request, such as a system prompt, a long set of instructions, a knowledge document, or few-shot examples. To use prompt caching with the Converse API, the request places the static content first and then inserts a cache checkpoint block immediately after the content that should be cached. On a cache hit, the response usage block reports cache read input tokens; on the write that populates the cache, it reports cache write input tokens. Each model has a minimum number of tokens that the cached prefix must contain before caching is triggered, and the cache entry has a time to live after which it expires. Prompt caching is available for on-demand invocation and is most effective when the same large prefix is reused many times within the cache lifetime.\n\n## Latency Benchmarking Methodology\nBenchmarking model latency requires careful, repeatable methodology. Time to first token, sometimes called time to first byte, measures how quickly the model begins returning output and is heavily influenced by input length and any cached prefix. Time to last token measures the full response duration. Output tokens per second is computed from the output token count divided by the total streaming time. To produce trustworthy numbers, run many invocations per scenario, report percentiles such as the median, the ninetieth percentile, and the standard deviation rather than a single average, and hold inference parameters constant across compared scenarios. When comparing cached and uncached behavior, keep the model, region, prompt, and inference configuration identical so that the cache is the only variable. Warm the cache with an initial request before measuring cache hit latency, because the first request pays the write cost while later requests enjoy the read benefit.\n\n## Inference Parameters and Their Effects\nTemperature, top-p, and top-k shape how a model samples its next token. Temperature scales the probability distribution: lower values make output more deterministic and focused, while higher values make it more diverse and creative. Top-p, also called nucleus sampling, restricts sampling to the smallest set of tokens whose cumulative probability exceeds the configured threshold. Top-k restricts sampling to the k most likely tokens. For reproducible benchmarks, these parameters must be set explicitly and held constant, because letting them vary or default silently makes results impossible to compare. Not every model supports every parameter; top-k in particular is passed through an additional model request fields channel and should be omitted for models that do not accept it, to avoid validation errors.\n\n## AWS Well-Architected Framework Overview (Detailed Guidance, Part 2)\nThe AWS Well-Architected Framework helps cloud architects build secure, high-performing, resilient, and efficient infrastructure for a variety of applications and workloads. The framework is organized around six pillars: operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability. Each pillar contains a set of design principles and best practices that teams can apply when designing and operating systems in the cloud. The framework is not a checklist to be completed once; it is a continuous mechanism for reviewing workloads, identifying risks, and prioritizing remediation. Architects use the framework during design reviews, before major launches, and at regular intervals after a workload reaches production so that decisions stay aligned with evolving business goals and customer needs.\n\n## Operational Excellence Pillar (Detailed Guidance, Part 2)\nThe operational excellence pillar focuses on running and monitoring systems to deliver business value and continually improving supporting processes and procedures. Key topics include organizing teams around business outcomes, implementing observability across the stack, automating responses to events, and performing frequent, small, reversible changes. Teams that practice operational excellence define runbooks for routine operations and playbooks for unexpected events. They instrument their applications with structured logs, metrics, and traces, and they review operational metrics in regular operational readiness reviews. Game days and failure injection exercises help teams validate that their procedures work under realistic conditions before a genuine incident occurs. The goal is to learn from every operational event and feed those lessons back into design and process.\n\n## Security Pillar (Detailed Guidance, Part 2)\nThe security pillar describes how to protect data, systems, and assets to take advantage of cloud technologies to improve security. Core practices include implementing a strong identity foundation, applying least privilege, enabling traceability, applying security at all layers, automating security best practices, protecting data in transit and at rest, keeping people away from raw data, and preparing for security events. Identity and access management is central: every principal should have only the permissions it needs, and those permissions should be reviewed regularly. Encryption keys should be managed with a dedicated key management service, and sensitive material should never be embedded in source code or configuration files. Detective controls such as centralized logging and continuous monitoring allow teams to detect anomalies and respond before they escalate into incidents.\n\n## Reliability Pillar (Detailed Guidance, Part 2)\nThe reliability pillar ensures a workload performs its intended function correctly and consistently when it is expected to. This includes the ability to operate and test the workload through its total lifecycle. Reliability practices cover foundations such as managing service quotas and network topology, workload architecture that isolates faults, change management that uses automation to deploy and roll back safely, and failure management that anticipates, withstands, and recovers from disruptions. Designing for reliability means assuming that components will fail and building systems that degrade gracefully. Techniques such as health checks, automatic failover across availability zones, backups with tested restores, and well-defined recovery time and recovery point objectives are foundational. Reliability is verified through testing, not assumed from design.\n\n## Performance Efficiency Pillar (Detailed Guidance, Part 2)\nThe performance efficiency pillar focuses on using computing resources efficiently to meet system requirements and maintaining that efficiency as demand changes and technologies evolve. It encourages teams to democratize advanced technologies, go global in minutes, use serverless architectures, experiment more often, and consider mechanical sympathy when selecting resources. Choosing the right instance families, storage classes, and database engines for a workload can dramatically change both performance and cost. Caching is a recurring theme: caching static content at the edge, caching database query results, and caching expensive computations all reduce latency and load. Teams should benchmark candidate configurations with representative traffic rather than relying on intuition, and they should re-evaluate choices as new instance types and managed services become available.\n\n## Cost Optimization Pillar (Detailed Guidance, Part 2)\nThe cost optimization pillar helps teams avoid unnecessary costs. It includes understanding spending over time, selecting the most cost-effective resources, matching supply with demand, and continuously analyzing and attributing expenditure. Practices include adopting a consumption model so you pay only for what you use, using managed services to reduce the cost of ownership, and applying commitment-based discounts where usage is predictable. Tagging resources consistently allows costs to be attributed to teams, products, and environments, which in turn drives accountability. Right-sizing compute, deleting idle resources, and moving infrequently accessed data to cheaper storage tiers are common, high-impact optimizations. Cost optimization is a continuous activity that pairs well with operational metrics and forecasting.\n\n## Sustainability Pillar (Detailed Guidance, Part 2)\nThe sustainability pillar focuses on minimizing the environmental impact of running cloud workloads. It encourages teams to understand their impact, establish sustainability goals, maximize utilization, anticipate and adopt new more efficient hardware and software offerings, use managed services, and reduce the downstream impact of cloud workloads. Maximizing utilization means running fewer, busier resources rather than many idle ones. Choosing regions and scheduling flexible workloads thoughtfully can also reduce impact. The pillar treats sustainability as a shared responsibility between the cloud provider, which improves the efficiency of the underlying infrastructure, and the customer, which improves the efficiency of the workloads that run on it.\n\n## Prompt Caching on Amazon Bedrock (Detailed Guidance, Part 2)\nPrompt caching is an Amazon Bedrock feature that lets applications cache a static prompt prefix so that repeated requests reusing that prefix can skip reprocessing it. This reduces time to first token and lowers input-token cost for workloads that send a large, stable context on every request, such as a system prompt, a long set of instructions, a knowledge document, or few-shot examples. To use prompt caching with the Converse API, the request places the static content first and then inserts a cache checkpoint block immediately after the content that should be cached. On a cache hit, the response usage block reports cache read input tokens; on the write that populates the cache, it reports cache write input tokens. Each model has a minimum number of tokens that the cached prefix must contain before caching is triggered, and the cache entry has a time to live after which it expires. Prompt caching is available for on-demand invocation and is most effective when the same large prefix is reused many times within the cache lifetime.\n\n## Latency Benchmarking Methodology (Detailed Guidance, Part 2)\nBenchmarking model latency requires careful, repeatable methodology. Time to first token, sometimes called time to first byte, measures how quickly the model begins returning output and is heavily influenced by input length and any cached prefix. Time to last token measures the full response duration. Output tokens per second is computed from the output token count divided by the total streaming time. To produce trustworthy numbers, run many invocations per scenario, report percentiles such as the median, the ninetieth percentile, and the standard deviation rather than a single average, and hold inference parameters constant across compared scenarios. When comparing cached and uncached behavior, keep the model, region, prompt, and inference configuration identical so that the cache is the only variable. Warm the cache with an initial request before measuring cache hit latency, because the first request pays the write cost while later requests enjoy the read benefit.\n\n## Inference Parameters and Their Effects (Detailed Guidance, Part 2)\nTemperature, top-p, and top-k shape how a model samples its next token. Temperature scales the probability distribution: lower values make output more deterministic and focused, while higher values make it more diverse and creative. Top-p, also called nucleus sampling, restricts sampling to the smallest set of tokens whose cumulative probability exceeds the configured threshold. Top-k restricts sampling to the k most likely tokens. For reproducible benchmarks, these parameters must be set explicitly and held constant, because letting them vary or default silently makes results impossible to compare. Not every model supports every parameter; top-k in particular is passed through an additional model request fields channel and should be omitted for models that do not accept it, to avoid validation errors.\n\n## AWS Well-Architected Framework Overview (Detailed Guidance, Part 3)\nThe AWS Well-Architected Framework helps cloud architects build secure, high-performing, resilient, and efficient infrastructure for a variety of applications and workloads. The framework is organized around six pillars: operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability. Each pillar contains a set of design principles and best practices that teams can apply when designing and operating systems in the cloud. The framework is not a checklist to be completed once; it is a continuous mechanism for reviewing workloads, identifying risks, and prioritizing remediation. Architects use the framework during design reviews, before major launches, and at regular intervals after a workload reaches production so that decisions stay aligned with evolving business goals and customer needs.\n\n## Operational Excellence Pillar (Detailed Guidance, Part 3)\nThe operational excellence pillar focuses on running and monitoring systems to deliver business value and continually improving supporting processes and procedures. Key topics include organizing teams around business outcomes, implementing observability across the stack, automating responses to events, and performing frequent, small, reversible changes. Teams that practice operational excellence define runbooks for routine operations and playbooks for unexpected events. They instrument their applications with structured logs, metrics, and traces, and they review operational metrics in regular operational readiness reviews. Game days and failure injection exercises help teams validate that their procedures work under realistic conditions before a genuine incident occurs. The goal is to learn from every operational event and feed those lessons back into design and process.\n\n## Security Pillar (Detailed Guidance, Part 3)\nThe security pillar describes how to protect data, systems, and assets to take advantage of cloud technologies to improve security. Core practices include implementing a strong identity foundation, applying least privilege, enabling traceability, applying security at all layers, automating security best practices, protecting data in transit and at rest, keeping people away from raw data, and preparing for security events. Identity and access management is central: every principal should have only the permissions it needs, and those permissions should be reviewed regularly. Encryption keys should be managed with a dedicated key management service, and sensitive material should never be embedded in source code or configuration files. Detective controls such as centralized logging and continuous monitoring allow teams to detect anomalies and respond before they escalate into incidents.\n\n## Reliability Pillar (Detailed Guidance, Part 3)\nThe reliability pillar ensures a workload performs its intended function correctly and consistently when it is expected to. This includes the ability to operate and test the workload through its total lifecycle. Reliability practices cover foundations such as managing service quotas and network topology, workload architecture that isolates faults, change management that uses automation to deploy and roll back safely, and failure management that anticipates, withstands, and recovers from disruptions. Designing for reliability means assuming that components will fail and building systems that degrade gracefully. Techniques such as health checks, automatic failover across availability zones, backups with tested restores, and well-defined recovery time and recovery point objectives are foundational. Reliability is verified through testing, not assumed from design.\n\n## Performance Efficiency Pillar (Detailed Guidance, Part 3)\nThe performance efficiency pillar focuses on using computing resources efficiently to meet system requirements and maintaining that efficiency as demand changes and technologies evolve. It encourages teams to democratize advanced technologies, go global in minutes, use serverless architectures, experiment more often, and consider mechanical sympathy when selecting resources. Choosing the right instance families, storage classes, and database engines for a workload can dramatically change both performance and cost. Caching is a recurring theme: caching static content at the edge, caching database query results, and caching expensive computations all reduce latency and load. Teams should benchmark candidate configurations with representative traffic rather than relying on intuition, and they should re-evaluate choices as new instance types and managed services become available.\n\n## Cost Optimization Pillar (Detailed Guidance, Part 3)\nThe cost optimization pillar helps teams avoid unnecessary costs. It includes understanding spending over time, selecting the most cost-effective resources, matching supply with demand, and continuously analyzing and attributing expenditure. Practices include adopting a consumption model so you pay only for what you use, using managed services to reduce the cost of ownership, and applying commitment-based discounts where usage is predictable. Tagging resources consistently allows costs to be attributed to teams, products, and environments, which in turn drives accountability. Right-sizing compute, deleting idle resources, and moving infrequently accessed data to cheaper storage tiers are common, high-impact optimizations. Cost optimization is a continuous activity that pairs well with operational metrics and forecasting.\n\n## Sustainability Pillar (Detailed Guidance, Part 3)\nThe sustainability pillar focuses on minimizing the environmental impact of running cloud workloads. It encourages teams to understand their impact, establish sustainability goals, maximize utilization, anticipate and adopt new more efficient hardware and software offerings, use managed services, and reduce the downstream impact of cloud workloads. Maximizing utilization means running fewer, busier resources rather than many idle ones. Choosing regions and scheduling flexible workloads thoughtfully can also reduce impact. The pillar treats sustainability as a shared responsibility between the cloud provider, which improves the efficiency of the underlying infrastructure, and the customer, which improves the efficiency of the workloads that run on it.\n\n## Prompt Caching on Amazon Bedrock (Detailed Guidance, Part 3)\nPrompt caching is an Amazon Bedrock feature that lets applications cache a static prompt prefix so that repeated requests reusing that prefix can skip reprocessing it. This reduces time to first token and lowers input-token cost for workloads that send a large, stable context on every request, such as a system prompt, a long set of instructions, a knowledge document, or few-shot examples. To use prompt caching with the Converse API, the request places the static content first and then inserts a cache checkpoint block immediately after the content that should be cached. On a cache hit, the response usage block reports cache read input tokens; on the write that populates the cache, it reports cache write input tokens. Each model has a minimum number of tokens that the cached prefix must contain before caching is triggered, and the cache entry has a time to live after which it expires. Prompt caching is available for on-demand invocation and is most effective when the same large prefix is reused many times within the cache lifetime.\n\n## Latency Benchmarking Methodology (Detailed Guidance, Part 3)\nBenchmarking model latency requires careful, repeatable methodology. Time to first token, sometimes called time to first byte, measures how quickly the model begins returning output and is heavily influenced by input length and any cached prefix. Time to last token measures the full response duration. Output tokens per second is computed from the output token count divided by the total streaming time. To produce trustworthy numbers, run many invocations per scenario, report percentiles such as the median, the ninetieth percentile, and the standard deviation rather than a single average, and hold inference parameters constant across compared scenarios. When comparing cached and uncached behavior, keep the model, region, prompt, and inference configuration identical so that the cache is the only variable. Warm the cache with an initial request before measuring cache hit latency, because the first request pays the write cost while later requests enjoy the read benefit.\n\n## Inference Parameters and Their Effects (Detailed Guidance, Part 3)\nTemperature, top-p, and top-k shape how a model samples its next token. Temperature scales the probability distribution: lower values make output more deterministic and focused, while higher values make it more diverse and creative. Top-p, also called nucleus sampling, restricts sampling to the smallest set of tokens whose cumulative probability exceeds the configured threshold. Top-k restricts sampling to the k most likely tokens. For reproducible benchmarks, these parameters must be set explicitly and held constant, because letting them vary or default silently makes results impossible to compare. Not every model supports every parameter; top-k in particular is passed through an additional model request fields channel and should be omitted for models that do not accept it, to avoid validation errors."} +{"text_prompt": "Using the reference document above, summarize the six pillars of the AWS Well-Architected Framework and explain how prompt caching reduces latency and input-token cost for workloads that reuse a large static context.", "expected_output_tokens": 200, "task_type": "Text-Generation", "model_id": "us.amazon.nova-pro-v1:0", "region": "us-east-1", "inference_profile": "standard", "prompt_caching": false, "cache_ttl": "5m", "cached_context": "## AWS Well-Architected Framework Overview\nThe AWS Well-Architected Framework helps cloud architects build secure, high-performing, resilient, and efficient infrastructure for a variety of applications and workloads. The framework is organized around six pillars: operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability. Each pillar contains a set of design principles and best practices that teams can apply when designing and operating systems in the cloud. The framework is not a checklist to be completed once; it is a continuous mechanism for reviewing workloads, identifying risks, and prioritizing remediation. Architects use the framework during design reviews, before major launches, and at regular intervals after a workload reaches production so that decisions stay aligned with evolving business goals and customer needs.\n\n## Operational Excellence Pillar\nThe operational excellence pillar focuses on running and monitoring systems to deliver business value and continually improving supporting processes and procedures. Key topics include organizing teams around business outcomes, implementing observability across the stack, automating responses to events, and performing frequent, small, reversible changes. Teams that practice operational excellence define runbooks for routine operations and playbooks for unexpected events. They instrument their applications with structured logs, metrics, and traces, and they review operational metrics in regular operational readiness reviews. Game days and failure injection exercises help teams validate that their procedures work under realistic conditions before a genuine incident occurs. The goal is to learn from every operational event and feed those lessons back into design and process.\n\n## Security Pillar\nThe security pillar describes how to protect data, systems, and assets to take advantage of cloud technologies to improve security. Core practices include implementing a strong identity foundation, applying least privilege, enabling traceability, applying security at all layers, automating security best practices, protecting data in transit and at rest, keeping people away from raw data, and preparing for security events. Identity and access management is central: every principal should have only the permissions it needs, and those permissions should be reviewed regularly. Encryption keys should be managed with a dedicated key management service, and sensitive material should never be embedded in source code or configuration files. Detective controls such as centralized logging and continuous monitoring allow teams to detect anomalies and respond before they escalate into incidents.\n\n## Reliability Pillar\nThe reliability pillar ensures a workload performs its intended function correctly and consistently when it is expected to. This includes the ability to operate and test the workload through its total lifecycle. Reliability practices cover foundations such as managing service quotas and network topology, workload architecture that isolates faults, change management that uses automation to deploy and roll back safely, and failure management that anticipates, withstands, and recovers from disruptions. Designing for reliability means assuming that components will fail and building systems that degrade gracefully. Techniques such as health checks, automatic failover across availability zones, backups with tested restores, and well-defined recovery time and recovery point objectives are foundational. Reliability is verified through testing, not assumed from design.\n\n## Performance Efficiency Pillar\nThe performance efficiency pillar focuses on using computing resources efficiently to meet system requirements and maintaining that efficiency as demand changes and technologies evolve. It encourages teams to democratize advanced technologies, go global in minutes, use serverless architectures, experiment more often, and consider mechanical sympathy when selecting resources. Choosing the right instance families, storage classes, and database engines for a workload can dramatically change both performance and cost. Caching is a recurring theme: caching static content at the edge, caching database query results, and caching expensive computations all reduce latency and load. Teams should benchmark candidate configurations with representative traffic rather than relying on intuition, and they should re-evaluate choices as new instance types and managed services become available.\n\n## Cost Optimization Pillar\nThe cost optimization pillar helps teams avoid unnecessary costs. It includes understanding spending over time, selecting the most cost-effective resources, matching supply with demand, and continuously analyzing and attributing expenditure. Practices include adopting a consumption model so you pay only for what you use, using managed services to reduce the cost of ownership, and applying commitment-based discounts where usage is predictable. Tagging resources consistently allows costs to be attributed to teams, products, and environments, which in turn drives accountability. Right-sizing compute, deleting idle resources, and moving infrequently accessed data to cheaper storage tiers are common, high-impact optimizations. Cost optimization is a continuous activity that pairs well with operational metrics and forecasting.\n\n## Sustainability Pillar\nThe sustainability pillar focuses on minimizing the environmental impact of running cloud workloads. It encourages teams to understand their impact, establish sustainability goals, maximize utilization, anticipate and adopt new more efficient hardware and software offerings, use managed services, and reduce the downstream impact of cloud workloads. Maximizing utilization means running fewer, busier resources rather than many idle ones. Choosing regions and scheduling flexible workloads thoughtfully can also reduce impact. The pillar treats sustainability as a shared responsibility between the cloud provider, which improves the efficiency of the underlying infrastructure, and the customer, which improves the efficiency of the workloads that run on it.\n\n## Prompt Caching on Amazon Bedrock\nPrompt caching is an Amazon Bedrock feature that lets applications cache a static prompt prefix so that repeated requests reusing that prefix can skip reprocessing it. This reduces time to first token and lowers input-token cost for workloads that send a large, stable context on every request, such as a system prompt, a long set of instructions, a knowledge document, or few-shot examples. To use prompt caching with the Converse API, the request places the static content first and then inserts a cache checkpoint block immediately after the content that should be cached. On a cache hit, the response usage block reports cache read input tokens; on the write that populates the cache, it reports cache write input tokens. Each model has a minimum number of tokens that the cached prefix must contain before caching is triggered, and the cache entry has a time to live after which it expires. Prompt caching is available for on-demand invocation and is most effective when the same large prefix is reused many times within the cache lifetime.\n\n## Latency Benchmarking Methodology\nBenchmarking model latency requires careful, repeatable methodology. Time to first token, sometimes called time to first byte, measures how quickly the model begins returning output and is heavily influenced by input length and any cached prefix. Time to last token measures the full response duration. Output tokens per second is computed from the output token count divided by the total streaming time. To produce trustworthy numbers, run many invocations per scenario, report percentiles such as the median, the ninetieth percentile, and the standard deviation rather than a single average, and hold inference parameters constant across compared scenarios. When comparing cached and uncached behavior, keep the model, region, prompt, and inference configuration identical so that the cache is the only variable. Warm the cache with an initial request before measuring cache hit latency, because the first request pays the write cost while later requests enjoy the read benefit.\n\n## Inference Parameters and Their Effects\nTemperature, top-p, and top-k shape how a model samples its next token. Temperature scales the probability distribution: lower values make output more deterministic and focused, while higher values make it more diverse and creative. Top-p, also called nucleus sampling, restricts sampling to the smallest set of tokens whose cumulative probability exceeds the configured threshold. Top-k restricts sampling to the k most likely tokens. For reproducible benchmarks, these parameters must be set explicitly and held constant, because letting them vary or default silently makes results impossible to compare. Not every model supports every parameter; top-k in particular is passed through an additional model request fields channel and should be omitted for models that do not accept it, to avoid validation errors.\n\n## AWS Well-Architected Framework Overview (Detailed Guidance, Part 2)\nThe AWS Well-Architected Framework helps cloud architects build secure, high-performing, resilient, and efficient infrastructure for a variety of applications and workloads. The framework is organized around six pillars: operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability. Each pillar contains a set of design principles and best practices that teams can apply when designing and operating systems in the cloud. The framework is not a checklist to be completed once; it is a continuous mechanism for reviewing workloads, identifying risks, and prioritizing remediation. Architects use the framework during design reviews, before major launches, and at regular intervals after a workload reaches production so that decisions stay aligned with evolving business goals and customer needs.\n\n## Operational Excellence Pillar (Detailed Guidance, Part 2)\nThe operational excellence pillar focuses on running and monitoring systems to deliver business value and continually improving supporting processes and procedures. Key topics include organizing teams around business outcomes, implementing observability across the stack, automating responses to events, and performing frequent, small, reversible changes. Teams that practice operational excellence define runbooks for routine operations and playbooks for unexpected events. They instrument their applications with structured logs, metrics, and traces, and they review operational metrics in regular operational readiness reviews. Game days and failure injection exercises help teams validate that their procedures work under realistic conditions before a genuine incident occurs. The goal is to learn from every operational event and feed those lessons back into design and process.\n\n## Security Pillar (Detailed Guidance, Part 2)\nThe security pillar describes how to protect data, systems, and assets to take advantage of cloud technologies to improve security. Core practices include implementing a strong identity foundation, applying least privilege, enabling traceability, applying security at all layers, automating security best practices, protecting data in transit and at rest, keeping people away from raw data, and preparing for security events. Identity and access management is central: every principal should have only the permissions it needs, and those permissions should be reviewed regularly. Encryption keys should be managed with a dedicated key management service, and sensitive material should never be embedded in source code or configuration files. Detective controls such as centralized logging and continuous monitoring allow teams to detect anomalies and respond before they escalate into incidents.\n\n## Reliability Pillar (Detailed Guidance, Part 2)\nThe reliability pillar ensures a workload performs its intended function correctly and consistently when it is expected to. This includes the ability to operate and test the workload through its total lifecycle. Reliability practices cover foundations such as managing service quotas and network topology, workload architecture that isolates faults, change management that uses automation to deploy and roll back safely, and failure management that anticipates, withstands, and recovers from disruptions. Designing for reliability means assuming that components will fail and building systems that degrade gracefully. Techniques such as health checks, automatic failover across availability zones, backups with tested restores, and well-defined recovery time and recovery point objectives are foundational. Reliability is verified through testing, not assumed from design.\n\n## Performance Efficiency Pillar (Detailed Guidance, Part 2)\nThe performance efficiency pillar focuses on using computing resources efficiently to meet system requirements and maintaining that efficiency as demand changes and technologies evolve. It encourages teams to democratize advanced technologies, go global in minutes, use serverless architectures, experiment more often, and consider mechanical sympathy when selecting resources. Choosing the right instance families, storage classes, and database engines for a workload can dramatically change both performance and cost. Caching is a recurring theme: caching static content at the edge, caching database query results, and caching expensive computations all reduce latency and load. Teams should benchmark candidate configurations with representative traffic rather than relying on intuition, and they should re-evaluate choices as new instance types and managed services become available.\n\n## Cost Optimization Pillar (Detailed Guidance, Part 2)\nThe cost optimization pillar helps teams avoid unnecessary costs. It includes understanding spending over time, selecting the most cost-effective resources, matching supply with demand, and continuously analyzing and attributing expenditure. Practices include adopting a consumption model so you pay only for what you use, using managed services to reduce the cost of ownership, and applying commitment-based discounts where usage is predictable. Tagging resources consistently allows costs to be attributed to teams, products, and environments, which in turn drives accountability. Right-sizing compute, deleting idle resources, and moving infrequently accessed data to cheaper storage tiers are common, high-impact optimizations. Cost optimization is a continuous activity that pairs well with operational metrics and forecasting.\n\n## Sustainability Pillar (Detailed Guidance, Part 2)\nThe sustainability pillar focuses on minimizing the environmental impact of running cloud workloads. It encourages teams to understand their impact, establish sustainability goals, maximize utilization, anticipate and adopt new more efficient hardware and software offerings, use managed services, and reduce the downstream impact of cloud workloads. Maximizing utilization means running fewer, busier resources rather than many idle ones. Choosing regions and scheduling flexible workloads thoughtfully can also reduce impact. The pillar treats sustainability as a shared responsibility between the cloud provider, which improves the efficiency of the underlying infrastructure, and the customer, which improves the efficiency of the workloads that run on it.\n\n## Prompt Caching on Amazon Bedrock (Detailed Guidance, Part 2)\nPrompt caching is an Amazon Bedrock feature that lets applications cache a static prompt prefix so that repeated requests reusing that prefix can skip reprocessing it. This reduces time to first token and lowers input-token cost for workloads that send a large, stable context on every request, such as a system prompt, a long set of instructions, a knowledge document, or few-shot examples. To use prompt caching with the Converse API, the request places the static content first and then inserts a cache checkpoint block immediately after the content that should be cached. On a cache hit, the response usage block reports cache read input tokens; on the write that populates the cache, it reports cache write input tokens. Each model has a minimum number of tokens that the cached prefix must contain before caching is triggered, and the cache entry has a time to live after which it expires. Prompt caching is available for on-demand invocation and is most effective when the same large prefix is reused many times within the cache lifetime.\n\n## Latency Benchmarking Methodology (Detailed Guidance, Part 2)\nBenchmarking model latency requires careful, repeatable methodology. Time to first token, sometimes called time to first byte, measures how quickly the model begins returning output and is heavily influenced by input length and any cached prefix. Time to last token measures the full response duration. Output tokens per second is computed from the output token count divided by the total streaming time. To produce trustworthy numbers, run many invocations per scenario, report percentiles such as the median, the ninetieth percentile, and the standard deviation rather than a single average, and hold inference parameters constant across compared scenarios. When comparing cached and uncached behavior, keep the model, region, prompt, and inference configuration identical so that the cache is the only variable. Warm the cache with an initial request before measuring cache hit latency, because the first request pays the write cost while later requests enjoy the read benefit.\n\n## Inference Parameters and Their Effects (Detailed Guidance, Part 2)\nTemperature, top-p, and top-k shape how a model samples its next token. Temperature scales the probability distribution: lower values make output more deterministic and focused, while higher values make it more diverse and creative. Top-p, also called nucleus sampling, restricts sampling to the smallest set of tokens whose cumulative probability exceeds the configured threshold. Top-k restricts sampling to the k most likely tokens. For reproducible benchmarks, these parameters must be set explicitly and held constant, because letting them vary or default silently makes results impossible to compare. Not every model supports every parameter; top-k in particular is passed through an additional model request fields channel and should be omitted for models that do not accept it, to avoid validation errors.\n\n## AWS Well-Architected Framework Overview (Detailed Guidance, Part 3)\nThe AWS Well-Architected Framework helps cloud architects build secure, high-performing, resilient, and efficient infrastructure for a variety of applications and workloads. The framework is organized around six pillars: operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability. Each pillar contains a set of design principles and best practices that teams can apply when designing and operating systems in the cloud. The framework is not a checklist to be completed once; it is a continuous mechanism for reviewing workloads, identifying risks, and prioritizing remediation. Architects use the framework during design reviews, before major launches, and at regular intervals after a workload reaches production so that decisions stay aligned with evolving business goals and customer needs.\n\n## Operational Excellence Pillar (Detailed Guidance, Part 3)\nThe operational excellence pillar focuses on running and monitoring systems to deliver business value and continually improving supporting processes and procedures. Key topics include organizing teams around business outcomes, implementing observability across the stack, automating responses to events, and performing frequent, small, reversible changes. Teams that practice operational excellence define runbooks for routine operations and playbooks for unexpected events. They instrument their applications with structured logs, metrics, and traces, and they review operational metrics in regular operational readiness reviews. Game days and failure injection exercises help teams validate that their procedures work under realistic conditions before a genuine incident occurs. The goal is to learn from every operational event and feed those lessons back into design and process.\n\n## Security Pillar (Detailed Guidance, Part 3)\nThe security pillar describes how to protect data, systems, and assets to take advantage of cloud technologies to improve security. Core practices include implementing a strong identity foundation, applying least privilege, enabling traceability, applying security at all layers, automating security best practices, protecting data in transit and at rest, keeping people away from raw data, and preparing for security events. Identity and access management is central: every principal should have only the permissions it needs, and those permissions should be reviewed regularly. Encryption keys should be managed with a dedicated key management service, and sensitive material should never be embedded in source code or configuration files. Detective controls such as centralized logging and continuous monitoring allow teams to detect anomalies and respond before they escalate into incidents.\n\n## Reliability Pillar (Detailed Guidance, Part 3)\nThe reliability pillar ensures a workload performs its intended function correctly and consistently when it is expected to. This includes the ability to operate and test the workload through its total lifecycle. Reliability practices cover foundations such as managing service quotas and network topology, workload architecture that isolates faults, change management that uses automation to deploy and roll back safely, and failure management that anticipates, withstands, and recovers from disruptions. Designing for reliability means assuming that components will fail and building systems that degrade gracefully. Techniques such as health checks, automatic failover across availability zones, backups with tested restores, and well-defined recovery time and recovery point objectives are foundational. Reliability is verified through testing, not assumed from design.\n\n## Performance Efficiency Pillar (Detailed Guidance, Part 3)\nThe performance efficiency pillar focuses on using computing resources efficiently to meet system requirements and maintaining that efficiency as demand changes and technologies evolve. It encourages teams to democratize advanced technologies, go global in minutes, use serverless architectures, experiment more often, and consider mechanical sympathy when selecting resources. Choosing the right instance families, storage classes, and database engines for a workload can dramatically change both performance and cost. Caching is a recurring theme: caching static content at the edge, caching database query results, and caching expensive computations all reduce latency and load. Teams should benchmark candidate configurations with representative traffic rather than relying on intuition, and they should re-evaluate choices as new instance types and managed services become available.\n\n## Cost Optimization Pillar (Detailed Guidance, Part 3)\nThe cost optimization pillar helps teams avoid unnecessary costs. It includes understanding spending over time, selecting the most cost-effective resources, matching supply with demand, and continuously analyzing and attributing expenditure. Practices include adopting a consumption model so you pay only for what you use, using managed services to reduce the cost of ownership, and applying commitment-based discounts where usage is predictable. Tagging resources consistently allows costs to be attributed to teams, products, and environments, which in turn drives accountability. Right-sizing compute, deleting idle resources, and moving infrequently accessed data to cheaper storage tiers are common, high-impact optimizations. Cost optimization is a continuous activity that pairs well with operational metrics and forecasting.\n\n## Sustainability Pillar (Detailed Guidance, Part 3)\nThe sustainability pillar focuses on minimizing the environmental impact of running cloud workloads. It encourages teams to understand their impact, establish sustainability goals, maximize utilization, anticipate and adopt new more efficient hardware and software offerings, use managed services, and reduce the downstream impact of cloud workloads. Maximizing utilization means running fewer, busier resources rather than many idle ones. Choosing regions and scheduling flexible workloads thoughtfully can also reduce impact. The pillar treats sustainability as a shared responsibility between the cloud provider, which improves the efficiency of the underlying infrastructure, and the customer, which improves the efficiency of the workloads that run on it.\n\n## Prompt Caching on Amazon Bedrock (Detailed Guidance, Part 3)\nPrompt caching is an Amazon Bedrock feature that lets applications cache a static prompt prefix so that repeated requests reusing that prefix can skip reprocessing it. This reduces time to first token and lowers input-token cost for workloads that send a large, stable context on every request, such as a system prompt, a long set of instructions, a knowledge document, or few-shot examples. To use prompt caching with the Converse API, the request places the static content first and then inserts a cache checkpoint block immediately after the content that should be cached. On a cache hit, the response usage block reports cache read input tokens; on the write that populates the cache, it reports cache write input tokens. Each model has a minimum number of tokens that the cached prefix must contain before caching is triggered, and the cache entry has a time to live after which it expires. Prompt caching is available for on-demand invocation and is most effective when the same large prefix is reused many times within the cache lifetime.\n\n## Latency Benchmarking Methodology (Detailed Guidance, Part 3)\nBenchmarking model latency requires careful, repeatable methodology. Time to first token, sometimes called time to first byte, measures how quickly the model begins returning output and is heavily influenced by input length and any cached prefix. Time to last token measures the full response duration. Output tokens per second is computed from the output token count divided by the total streaming time. To produce trustworthy numbers, run many invocations per scenario, report percentiles such as the median, the ninetieth percentile, and the standard deviation rather than a single average, and hold inference parameters constant across compared scenarios. When comparing cached and uncached behavior, keep the model, region, prompt, and inference configuration identical so that the cache is the only variable. Warm the cache with an initial request before measuring cache hit latency, because the first request pays the write cost while later requests enjoy the read benefit.\n\n## Inference Parameters and Their Effects (Detailed Guidance, Part 3)\nTemperature, top-p, and top-k shape how a model samples its next token. Temperature scales the probability distribution: lower values make output more deterministic and focused, while higher values make it more diverse and creative. Top-p, also called nucleus sampling, restricts sampling to the smallest set of tokens whose cumulative probability exceeds the configured threshold. Top-k restricts sampling to the k most likely tokens. For reproducible benchmarks, these parameters must be set explicitly and held constant, because letting them vary or default silently makes results impossible to compare. Not every model supports every parameter; top-k in particular is passed through an additional model request fields channel and should be omitted for models that do not accept it, to avoid validation errors."} diff --git a/model-latency-benchmarking/dummy-prompts-for-benchmarking.jsonl b/model-latency-benchmarking/dummy-prompts-for-benchmarking.jsonl index c11d72b08..eeff9469c 100644 --- a/model-latency-benchmarking/dummy-prompts-for-benchmarking.jsonl +++ b/model-latency-benchmarking/dummy-prompts-for-benchmarking.jsonl @@ -1,5 +1,5 @@ -{"text_prompt": "Explain the concept of machine learning in simple terms.", "expected_output_tokens": 50, "task_type": "Text-Generation", "model_id": "us.meta.llama3-1-70b-instruct-v1:0", "region": "us-east-2", "inference_profile": "optimized"} -{"text_prompt": "Explain the concept of machine learning in simple terms.", "expected_output_tokens": 50, "task_type": "Text-Generation", "model_id": "us.anthropic.claude-3-5-haiku-20241022-v1:0", "region": "us-east-2", "inference_profile": "optimized"} -{"text_prompt": "Explain the concept of machine learning in simple terms.", "expected_output_tokens": 50, "task_type": "Text-Generation", "model_id": "us.anthropic.claude-3-5-haiku-20241022-v1:0", "region": "us-east-2", "inference_profile": "standard"} -{"text_prompt": "Explain the concept of machine learning in simple terms.", "expected_output_tokens": 50, "task_type": "Text-Generation", "model_id": "amazon.nova-micro-v1:0", "region": "us-east-1", "inference_profile": "standard"} -{"text_prompt": "Explain the concept of machine learning in simple terms.", "expected_output_tokens": 50, "task_type": "Text-Generation", "model_id": "anthropic.claude-3-5-sonnet-20241022-v2:0", "region": "us-west-2", "inference_profile": "standard"} \ No newline at end of file +{"text_prompt": "Explain the concept of machine learning in simple terms.", "expected_output_tokens": 50, "task_type": "Text-Generation", "model_id": "us.anthropic.claude-opus-4-5-20251101-v1:0", "region": "us-east-1", "inference_profile": "standard"} +{"text_prompt": "Explain the concept of machine learning in simple terms.", "expected_output_tokens": 50, "task_type": "Text-Generation", "model_id": "us.anthropic.claude-sonnet-4-5-20250929-v1:0", "region": "us-east-1", "inference_profile": "optimized"} +{"text_prompt": "Explain the concept of machine learning in simple terms.", "expected_output_tokens": 50, "task_type": "Text-Generation", "model_id": "us.anthropic.claude-3-7-sonnet-20250219-v1:0", "region": "us-west-2", "inference_profile": "standard"} +{"text_prompt": "Explain the concept of machine learning in simple terms.", "expected_output_tokens": 50, "task_type": "Text-Generation", "model_id": "us.amazon.nova-pro-v1:0", "region": "us-east-1", "inference_profile": "standard"} +{"text_prompt": "Based on the platform engineering handbook above, summarize the recommended approach to multi-account governance and incident response.", "expected_output_tokens": 100, "task_type": "Text-Generation", "model_id": "us.anthropic.claude-haiku-4-5-20251001-v1:0", "region": "us-east-1", "inference_profile": "standard", "prompt_caching": true, "cache_ttl": "5m", "cached_context": "GlobalRetail Platform Engineering Handbook. Section 1. Purpose and Scope. This handbook describes the technical standards, operational practices, and architectural principles that govern the GlobalRetail cloud platform. It applies to every engineering team that builds, deploys, or operates workloads on the shared platform. The goal of this document is to provide a single authoritative reference so that teams make consistent decisions, reduce operational risk, and accelerate delivery without sacrificing reliability or security. Every standard in this handbook exists because of a lesson learned during an incident, an audit finding, or a cost overrun. Teams are expected to read the relevant sections before designing a new service and to revisit them during architecture reviews. Section 2. Organizational Principles. GlobalRetail organizes engineering around small autonomous teams that own their services from design through production support. Each team is responsible for the full lifecycle of the services it owns, including on-call duties, capacity planning, security posture, and cost management. Ownership is explicit and recorded in the service catalog, which maps every production service to a single owning team, a primary on-call rotation, and an escalation path. Teams operate with a high degree of independence, but that independence is bounded by the platform standards described here. The platform team provides paved roads, which are pre-approved patterns and tooling that make the secure and reliable choice the easy choice. Teams that follow the paved road inherit security controls, observability, and deployment automation by default. Teams that need to leave the paved road must document the deviation, justify it during an architecture review, and accept responsibility for the additional operational burden. Section 3. Account Structure and Multi-Account Governance. The platform uses a multi-account strategy to provide strong isolation boundaries between workloads, environments, and teams. Each team receives separate accounts for development, staging, and production. Production accounts are subject to the strictest controls and the most conservative change management. Accounts are organized into a hierarchy of organizational units that reflect business function and environment. Security and compliance controls are applied centrally through service control policies attached to organizational units, so that guardrails cannot be removed by individual teams. The management account is reserved for billing, organization administration, and the configuration of organization wide policies. No workloads run in the management account. A dedicated security account aggregates audit logs, findings, and configuration snapshots from every other account. A dedicated logging account stores immutable copies of access logs and audit trails with restrictive retention and deletion protection. Network connectivity between accounts is established through a central transit hub, and direct peering between team accounts is discouraged in favor of the hub and spoke model, which centralizes inspection and routing. Cross account access always uses temporary credentials obtained through role assumption rather than long lived access keys. Long lived access keys are prohibited in all environments, and automated scanners detect and report any that appear. Section 4. Identity and Access Management. Access to the platform follows the principle of least privilege. Human access is granted through federated single sign on, and no human is given standing access to production. Access to production is requested just in time, approved by a second engineer, and granted for a limited window after which it expires automatically. All access is logged and reviewed. Service to service access uses scoped roles with narrowly defined permissions. Wildcard permissions are forbidden in production policies except where a specific documented exception has been approved. Permission boundaries are applied to every role that teams can create themselves, so that even a misconfigured policy cannot exceed the boundary set by the platform. Secrets such as database passwords and third party API keys are stored in a managed secrets service, encrypted at rest, and rotated automatically. Secrets are never committed to source control, embedded in container images, or printed in logs. Automated pipelines scan every commit and every image for secrets and block the build when a secret is detected. Section 5. Networking Standards. Every workload runs inside a virtual private network with private subnets for compute and data tiers and public subnets reserved for load balancers and managed ingress. Direct internet exposure of compute instances is prohibited. Outbound internet access from private subnets is routed through managed network address translation, and egress is restricted to approved destinations where the threat model requires it. Internal traffic between services is encrypted in transit, and mutual authentication is required for sensitive internal communication. Security groups are configured to allow only the specific ports and sources that a service requires, and broad rules that allow traffic from any source are rejected during review. Network access control lists provide a coarse second layer of defense at the subnet boundary. DNS is managed centrally so that service discovery is consistent across accounts, and private hosted zones are used for internal names that should never resolve publicly. Section 6. Compute and Container Standards. Stateless services are the default. Teams are encouraged to design services that hold no durable state locally so that instances can be replaced freely and scaled horizontally. Containerized workloads run on the managed orchestration platform, and container images are built from approved minimal base images that are patched centrally. Images are scanned for vulnerabilities before they are allowed into the production registry, and images with critical findings are blocked. Every container declares its resource requests and limits so that the scheduler can place workloads efficiently and prevent noisy neighbor problems. Autoscaling is configured based on meaningful application metrics rather than raw infrastructure metrics alone, so that capacity tracks real demand. Serverless functions are preferred for event driven and bursty workloads because they reduce idle cost and operational overhead. Long running batch jobs use managed batch services with spot capacity where interruption is tolerable, which significantly reduces cost. Section 7. Data Management and Storage. Data is classified into public, internal, confidential, and restricted tiers, and the classification determines the controls that apply. Confidential and restricted data is encrypted at rest using customer managed keys, and access to those keys is tightly controlled and audited. Databases are deployed in private subnets and are never directly reachable from the internet. Backups are automated, encrypted, and tested through periodic restore exercises, because a backup that has never been restored is not a backup. Point in time recovery is enabled for critical databases. Object storage buckets are private by default, and public access is blocked at the account level so that an individual misconfiguration cannot expose data. Lifecycle policies move infrequently accessed data to lower cost storage tiers and expire data that is no longer needed, which controls both cost and compliance risk. Data retention follows the legal and regulatory requirements for each data class, and teams document the retention period for every data store they own. Section 8. Observability and Monitoring. Every service emits structured logs, metrics, and distributed traces. Logs are written in a consistent structured format with correlation identifiers so that a single request can be followed across service boundaries. Metrics cover both technical health, such as latency, error rate, and saturation, and business health, such as orders placed and carts abandoned. Dashboards present the golden signals for each service, and alerts are defined against service level objectives rather than arbitrary thresholds. Service level objectives are agreed with stakeholders and expressed as a target reliability over a rolling window, with an error budget that governs how aggressively a team can ship changes. When the error budget is exhausted, teams slow down and prioritize reliability work over new features. Traces are sampled intelligently so that the system captures enough detail to debug problems without overwhelming storage. Alerts are actionable, meaning every alert links to a runbook and describes a condition that a human can and should respond to. Noisy alerts that do not require action are tuned or removed, because alert fatigue is itself an operational risk. Section 9. Deployment and Change Management. All infrastructure is defined as code and stored in version control. Manual changes to production through consoles are prohibited except during a declared emergency, and any emergency change must be reconciled back into code afterward. Deployments are automated through pipelines that run tests, security scans, and policy checks before any change reaches production. Changes are rolled out progressively using canary or blue green strategies so that a bad change affects a small fraction of traffic before it is detected and rolled back automatically. Every deployment is reversible, and teams maintain the ability to roll back quickly. Database schema changes are designed to be backward compatible so that application and schema can be deployed independently. Feature flags decouple deployment from release, allowing teams to deploy code in a dormant state and enable it gradually. Section 10. Reliability and Resilience. Services are designed to tolerate the failure of any single component without customer impact. Critical workloads are deployed across multiple availability zones, and the most critical workloads are designed for multi region operation. Dependencies are treated as unreliable, and every remote call has a timeout, a bounded retry policy with backoff and jitter, and a circuit breaker that prevents a struggling dependency from cascading failures upstream. Bulkheads isolate pools of resources so that a problem in one area cannot exhaust the resources needed elsewhere. Teams conduct regular game days in which they deliberately inject failures into non production and sometimes production environments to validate that their resilience mechanisms work as designed. Capacity is planned ahead of known demand peaks, and load tests validate that services meet their objectives under expected and stress conditions. Section 11. Incident Response. When an incident occurs, the owning team follows a defined incident response process. The first priority is always to restore service for customers, and root cause analysis follows recovery rather than blocking it. An incident commander is designated to coordinate the response, make decisions, and own communication, while responders focus on diagnosis and mitigation. A dedicated communications channel is opened for every significant incident, and status updates are posted at regular intervals to keep stakeholders informed. Severity levels are defined clearly so that the response scales appropriately, with the highest severity reserved for incidents that cause broad customer impact or data risk. After the incident is resolved, the team writes a blameless post incident review that documents the timeline, the contributing factors, the customer impact, and the corrective actions. The review focuses on the systemic and process factors that allowed the incident rather than blaming individuals, because a culture of blame discourages the honest reporting that makes systems safer. Corrective actions are tracked to completion and prioritized against feature work according to the risk they address. Recurring incident themes are escalated to leadership so that systemic investment can be made. Section 12. Security Operations. Security is a shared responsibility across every team rather than the exclusive domain of a central group. Threat detection runs continuously across all accounts, and findings are routed to the owning team with a severity and a remediation deadline. Vulnerabilities in dependencies and base images are tracked, and teams patch within a window defined by the severity of the vulnerability. Penetration tests and security reviews are conducted for significant new services and major changes. Audit logs are immutable and centralized so that an attacker cannot cover their tracks by deleting logs in a compromised account. Access reviews are conducted periodically to remove permissions that are no longer needed, because permissions tend to accumulate over time. Section 13. Cost Management. Cost is treated as a first class operational metric. Every resource is tagged with the owning team, the environment, and the cost center so that spend can be attributed accurately. Teams review their spend regularly and are accountable for the cost of the services they own. The platform provides dashboards that highlight the largest cost drivers and flag anomalies such as a sudden increase in spend. Reserved capacity and savings commitments are used for predictable baseline load, while elastic and spot capacity absorbs variable and interruptible load. Idle and orphaned resources are detected automatically and cleaned up, because forgotten resources are a common and avoidable source of waste. Right sizing recommendations are reviewed during regular operational reviews. Section 14. Architecture Reviews. Significant new services and major changes go through an architecture review before substantial work begins. The review evaluates the design against the standards in this handbook, focusing on security, reliability, cost, and operability. Reviews are collaborative and educational rather than adversarial, and their purpose is to surface risks early when they are cheap to address. The reviewers look for single points of failure, missing failure handling, unclear ownership, and deviations from the paved road that lack justification. The output of a review is a set of agreed actions and any documented exceptions. Exceptions are time bound and revisited so that temporary deviations do not become permanent. Section 15. Testing Standards. Quality is built in rather than inspected at the end. Every service maintains a layered testing strategy that combines fast unit tests, focused integration tests, and a small number of end to end tests that exercise critical user journeys. Unit tests run on every commit and must complete quickly so that engineers receive feedback within minutes. Integration tests validate the contracts between a service and its dependencies, using ephemeral environments that are created and destroyed automatically. Contract tests guard the boundaries between services so that a provider cannot break a consumer without detection. Performance tests validate that services meet their latency and throughput objectives under representative load, and they run before major releases. Test data is synthetic and never contains real customer information, because using production data in test environments creates unacceptable privacy risk. Flaky tests are treated as defects and are quarantined and fixed rather than ignored, because a test suite that engineers do not trust provides no protection. Code coverage is measured as a signal rather than a target, since high coverage of trivial code provides little assurance while thoughtful tests of complex logic provide a great deal. Section 16. Documentation and Knowledge Sharing. Every service maintains living documentation that describes its purpose, its interfaces, its dependencies, its operational characteristics, and its runbooks. Documentation lives close to the code it describes and is reviewed alongside code changes so that it does not drift out of date. Each service publishes an architecture diagram, an interface specification, and an on call guide that a new responder can follow during an incident. Decisions of consequence are recorded as architecture decision records that capture the context, the options considered, the decision, and the consequences, so that future engineers understand why the system is the way it is. Teams share lessons learned through regular technical forums, and significant incidents and their reviews are circulated widely so that the whole organization benefits from each lesson. Onboarding documentation is kept current so that new engineers become productive quickly. Section 17. Sustainability and Efficiency. Engineering teams consider the efficiency of the resources they consume, because efficient systems are usually both cheaper and lower in environmental impact. Teams choose regions and capacity options thoughtfully, decommission unused workloads promptly, and design for elasticity so that resources scale down when demand falls. Efficiency is reviewed during operational reviews alongside cost and reliability. Section 18. Compliance and Audit. The platform operates under several regulatory and contractual obligations, and teams that handle regulated data must understand the controls that apply to them. Compliance requirements are translated into concrete technical controls that are enforced automatically wherever possible, so that compliance is a property of the system rather than a manual checklist. Configuration rules continuously evaluate every account against the required controls and report drift, such as an unencrypted volume or a publicly accessible bucket, so that violations are detected and corrected quickly. Evidence for audits is collected automatically from the centralized logging and configuration stores, which reduces the burden on teams during an audit and ensures that evidence is complete and tamper resistant. Changes to regulated systems are tracked with a clear record of who made the change, what changed, and why, and that record is retained for the period required by the relevant regulation. Teams that are unsure whether a control applies to their workload consult the compliance function early rather than discovering a gap during an audit. Section 19. Vendor and Dependency Management. Every external dependency, whether a software library, a managed service, or a third party API, introduces risk that must be understood and managed. Teams maintain an inventory of the dependencies their services rely on, including the version in use and the license under which it is distributed. Dependencies are kept reasonably current so that the team can apply security patches without first completing a large and risky upgrade. New dependencies are evaluated before adoption for their security posture, their maintenance health, their license compatibility, and the blast radius of a potential failure or compromise. Critical third party services are assessed for their reliability commitments, and teams design graceful degradation so that the failure of a non essential vendor does not take down the whole experience. Where a single vendor represents a concentration risk, teams document the risk and, where justified, design an exit path. Supply chain integrity is protected by verifying the provenance of artifacts and by pinning dependencies to known good versions rather than floating to the latest version automatically. Section 20. Continuous Improvement. The platform and its standards are never finished. Teams regularly reflect on what is working and what is not, through retrospectives after projects and incidents and through ongoing operational reviews. Improvement ideas are captured, prioritized, and acted upon rather than lost, and the platform team treats the paved road itself as a product that must continuously earn the adoption of the teams that use it. Metrics about the health of the engineering system, such as deployment frequency, lead time for changes, change failure rate, and time to restore service, are tracked over time so that the organization can see whether it is getting better. Investments in tooling, automation, and developer experience are justified by their effect on these metrics and on the daily experience of engineers. Section 21. Developer Experience and Platform Adoption. A platform succeeds only when teams choose to use it, so the platform team measures and invests in developer experience deliberately. Setting up a new service from the approved templates should take minutes rather than days, and the templates generate a working pipeline, baseline observability, security controls, and documentation scaffolding automatically. Local development environments mirror production closely enough that surprises are rare, and engineers can run and test their services without depending on shared environments that create contention. Self service tooling lets teams provision the resources they are entitled to without filing tickets and waiting, while policy guardrails ensure that self service never compromises security or cost discipline. Feedback from engineers is gathered regularly through surveys and direct conversation, and the friction points that engineers report most often are prioritized for elimination. The platform publishes a clear catalog of the capabilities it offers, with examples and guidance, so that teams can discover the paved road rather than reinventing solutions that already exist. When a team repeatedly leaves the paved road for the same reason, the platform team treats that as a signal that the paved road has a gap and works to close it, because the most durable way to drive adoption is to make the supported path genuinely the best path. Section 22. Summary. The standards in this handbook exist to help teams move quickly with confidence. By following the paved road, designing for failure, automating everything, measuring what matters, and treating security and cost as everyone's responsibility, teams deliver reliable services that customers trust. This document is living and is updated as the platform evolves and as new lessons are learned."} diff --git a/model-latency-benchmarking/latency-benchmarking-tool.ipynb b/model-latency-benchmarking/latency-benchmarking-tool.ipynb index 2e34ccbe8..7081761c0 100644 --- a/model-latency-benchmarking/latency-benchmarking-tool.ipynb +++ b/model-latency-benchmarking/latency-benchmarking-tool.ipynb @@ -50,6 +50,27 @@ "Note: if you configure `\"inference_profile\": \"optimized\"`, you must use `us-east-2` region only because optimized inference is currently only available in `us-east-2` region. " ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Enabling prompt caching\n", + "\n", + "Prompt caching is optional and turned off by default, so existing datasets run unchanged. When enabled, the tool caches a large static prefix so repeated requests that reuse it skip reprocessing, which lowers Time to First Token and input-token cost.\n", + "\n", + "To enable prompt caching, add these optional fields to a scenario line in your JSONL dataset:\n", + "\n", + "- `prompt_caching`: set to `true` to cache the static prefix for that scenario (default `false`).\n", + "- `cached_context`: the long, static text to cache. Required when `prompt_caching` is `true`, and it must meet the model minimum cacheable token count (for example, 4,096 tokens for Claude Haiku 4.5) for caching to trigger.\n", + "- `cache_ttl`: how long the cache entry lives, either `5m` (default) or `1h`.\n", + "\n", + "You can also flip the global `PROMPT_CACHING` toggle in the Configuration cell to enable caching for every scenario by default. A per-scenario `prompt_caching` value always overrides the global toggle.\n", + "\n", + "Prompt caching works for on-demand invocation only and only on caching-capable models, so set `prompt_caching: true` only on models that support it. The file `caching-demo-prompts-for-benchmarking.jsonl` ships ready-to-run cached and uncached pairs for Claude Haiku 4.5 and Amazon Nova Pro.\n", + "\n", + "When caching is active, each invocation also reports `cache_read_input_tokens`, `cache_write_input_tokens`, and `Cache_Hit_Rate`, and the analysis reports Time to First Token separately for cached and uncached calls.\n" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -148,7 +169,17 @@ "TEMPERATURE = 1\n", "TOP_P = 1\n", "TOP_K = 250\n", - "EXPERIMENT_NAME = '' # your custom experiment name" + "\n", + "# Some Bedrock models (for example Amazon Nova and the latest Claude models)\n", + "# reject temperature and topP in the same request, so the tool sends only one\n", + "# sampling parameter. Set this to 'temperature' or 'topP'.\n", + "INFERENCE_SAMPLING = 'temperature'\n", + "EXPERIMENT_NAME = '' # your custom experiment name\n", + "\n", + "# Global prompt caching default toggle. Shipped default is\n", + "# False so existing datasets behave exactly as before. A per-scenario\n", + "# `prompt_caching` field in the JSONL always overrides this global default.\n", + "PROMPT_CACHING = False" ] }, { @@ -180,6 +211,46 @@ "from threading import Lock\n", "from typing import List, Dict\n", "\n", + "# ---------------------------------------------------------------------------\n", + "# Minimum required boto3 version\n", + "# Minimum boto3 that supports the cachePoint block in the Converse API and the\n", + "# latest listed Bedrock models (Claude Opus/Sonnet/Haiku 4.5, Claude 3.7\n", + "# Sonnet, Amazon Nova). The tool checks this before running any benchmark.\n", + "# This value is parseable by check_boto3_version below.\n", + "# ---------------------------------------------------------------------------\n", + "MIN_BOTO3_VERSION = \"1.40.0\"\n", + "\n", + "def _parse_version(version):\n", + " \"\"\"Parse a dotted version string into a 3-tuple of ints (major, minor, patch).\n", + "\n", + " Non-numeric suffixes are stripped so values like '1.40.0' or '1.40.0rc1'\n", + " parse cleanly. Missing components default to 0 so the result always has\n", + " three elements and tuple comparison is well defined.\"\"\"\n", + " parts = []\n", + " for token in str(version).strip().split('.'):\n", + " digits = ''.join(ch for ch in token if ch.isdigit())\n", + " parts.append(int(digits) if digits else 0)\n", + " while len(parts) < 3:\n", + " parts.append(0)\n", + " return tuple(parts[:3])\n", + "\n", + "def check_boto3_version(installed, minimum):\n", + " \"\"\"Return (is_satisfied, message) from a pure comparison of version tuples.\n", + "\n", + " is_satisfied is True if and only if the installed version is greater than\n", + " or equal to the minimum. When unsatisfied, the message states the required\n", + " minimum version so the user knows the target to upgrade to.\"\"\"\n", + " is_satisfied = _parse_version(installed) >= _parse_version(minimum)\n", + " if is_satisfied:\n", + " message = f\"boto3 {installed} satisfies the minimum required version {minimum}.\"\n", + " else:\n", + " message = (\n", + " f\"boto3 {installed} is below the required minimum version {minimum}. \"\n", + " f\"Please upgrade boto3 to {minimum} or later before running this tool \"\n", + " f\"(for example: pip install --upgrade 'boto3>={minimum}').\"\n", + " )\n", + " return is_satisfied, message\n", + "\n", "logging_lock = Lock()\n", "os.makedirs(f\"{directory}\", exist_ok=True)\n", "os.makedirs(f\"{directory}-analysis\", exist_ok=True)\n", @@ -211,23 +282,89 @@ " content = file.read()\n", " return content\n", "\n", - "def get_body(model_id, file_path, prompt, max_tokens):\n", - " body = [\n", + "def _is_anthropic_model(model_id):\n", + " '''Return True when the target model is an Anthropic Claude model.'''\n", + " normalized = (model_id or '').lower()\n", + " return 'anthropic' in normalized or 'claude' in normalized\n", + "\n", + "def _build_cache_point_block(cache_ttl, model_id):\n", + " '''Return a Converse cachePoint content block for the target model.\n", + "\n", + " The Converse API expresses a cache checkpoint as a content block of the\n", + " form {'cachePoint': {'type': 'default'}}. The default cache duration is\n", + " five minutes. The optional 'ttl' field requests an extended duration and\n", + " is only supported for Anthropic models, so it is attached only for the\n", + " '1h' option on Anthropic models. Every other case uses a plain cachePoint\n", + " with the default duration. cache_ttl is validated upstream by\n", + " validate_caching_fields, so it is one of {'5m', '1h'} here.'''\n", + " block = {'cachePoint': {'type': 'default'}}\n", + " if cache_ttl == '1h' and _is_anthropic_model(model_id):\n", + " block['cachePoint']['ttl'] = '1h'\n", + " return block\n", + "\n", + "def _model_supports_top_k(model_id):\n", + " '''Return True when the target model accepts top_k.\n", + "\n", + " top_k is not part of the standard Converse inferenceConfig and is passed\n", + " through additionalModelRequestFields. The Anthropic Claude family accepts\n", + " top_k there, while other families (for example Amazon Nova) reject the\n", + " field and fail validation. We gate on the model identifier so top_k is\n", + " sent only when supported and omitted otherwise.'''\n", + " normalized = (model_id or '').lower()\n", + " return 'anthropic' in normalized or 'claude' in normalized\n", + "\n", + "def build_converse_request(prompt, cached_context, prompt_caching, cache_ttl, max_tokens, temperature, top_p, top_k, model_id):\n", + " '''Build the Converse request parts for a single invocation.\n", + "\n", + " Returns (messages, inferenceConfig, additionalModelRequestFields). This\n", + " replaces the legacy get_body and fixes the temperature/top_p defect. The\n", + " benchmark function passes modelId and performanceConfig.latency separately\n", + " when it calls the model.\n", + "\n", + " Behavior:\n", + " - inferenceConfig sends only one sampling parameter (temperature or\n", + " topP) because some models reject both together; neither is hardcoded.\n", + " - When prompt_caching is True, the user message places the cached_context\n", + " text block first, then exactly one cachePoint block carrying the TTL\n", + " derived from cache_ttl, then the user prompt.\n", + " - When prompt_caching is False, no cachePoint block is included and the\n", + " message matches the original single-text structure\n", + " with no cachePoint block.\n", + " - top_k is passed through additionalModelRequestFields only when it is\n", + " set and supported by the target model, otherwise it is omitted and\n", + " this return value is None.'''\n", + " if prompt_caching:\n", + " content = [\n", + " {'text': cached_context},\n", + " _build_cache_point_block(cache_ttl, model_id),\n", + " {'text': prompt},\n", + " ]\n", + " else:\n", + " content = [\n", + " {'text': prompt},\n", + " ]\n", + "\n", + " messages = [\n", " {\n", " 'role': 'user',\n", - " 'content': [\n", - " {\n", - " 'text': prompt\n", - " },\n", - " ]\n", + " 'content': content,\n", " },\n", " ]\n", - " inferenceConfig={\n", - " 'maxTokens': max_tokens,\n", - " 'temperature': 0,\n", - " 'topP': 1\n", - " }\n", - " return body, inferenceConfig\n", + "\n", + " # Some models reject temperature and topP together, so send only the\n", + " # configured sampling parameter (see INFERENCE_SAMPLING in the config cell).\n", + " sampling = globals().get('INFERENCE_SAMPLING', 'temperature')\n", + " inferenceConfig = {'maxTokens': max_tokens}\n", + " if sampling == 'topP':\n", + " inferenceConfig['topP'] = top_p\n", + " else:\n", + " inferenceConfig['temperature'] = temperature\n", + "\n", + " additionalModelRequestFields = None\n", + " if top_k is not None and _model_supports_top_k(model_id):\n", + " additionalModelRequestFields = {'top_k': top_k}\n", + "\n", + " return messages, inferenceConfig, additionalModelRequestFields\n", "\n", "def read_jsonl_files(directory_path):\n", " all_data = []\n", @@ -243,11 +380,107 @@ " print(f\"Error decoding JSON in file {filename}: {e}\")\n", " return all_data\n", "\n", + "def parse_scenario(raw, file_path, global_prompt_caching):\n", + " \"\"\"Map a raw JSONL object to a Scenario dict with caching fields resolved.\n", + "\n", + " Backward compatibility is enforced here. A raw object\n", + " that contains only the original fields produces a Scenario that behaves\n", + " exactly like prompt_caching=False when the global default is False\n", + " so existing datasets keep working unchanged.\n", + "\n", + " Optional caching fields and their defaults:\n", + " - prompt_caching: bool, default = global_prompt_caching\n", + " - cache_ttl: str, default \\\"5m\\\"\n", + " - cached_context: str or None, default None\n", + "\n", + " All existing fields are preserved: text_prompt, expected_output_tokens,\n", + " task_type, model_id, region, inference_profile.\"\"\"\n", + " scenario = {\n", + " 'file_path': file_path,\n", + " 'text_prompt': raw.get('text_prompt'),\n", + " 'expected_output_tokens': raw.get('expected_output_tokens', 100),\n", + " 'task_type': raw.get('task_type'),\n", + " 'model_id': raw.get('model_id'),\n", + " 'region': raw.get('region'),\n", + " 'inference_profile': raw.get('inference_profile', 'optimized'),\n", + " }\n", + " # Resolve the effective prompt caching value: the per-scenario field always\n", + " # wins when present, otherwise fall back to the global default.\n", + " scenario['prompt_caching'] = raw.get('prompt_caching', global_prompt_caching)\n", + " scenario['cache_ttl'] = raw.get('cache_ttl', '5m')\n", + " scenario['cached_context'] = raw.get('cached_context', None)\n", + " return scenario\n", + "\n", + "def validate_caching_fields(scenario):\n", + " \"\"\"Return (is_valid, error_message) for a scenario's caching fields.\n", + "\n", + " Validation runs per scenario before invocation and\n", + " MUST NOT raise. The run loop turns an invalid result\n", + " into an error status and descriptive message; this function only reports\n", + " validity. Rules:\n", + " - When prompt_caching is True, cached_context must be a non-empty\n", + " string; None, empty string, and whitespace-only are all treated as\n", + " missing and are invalid.\n", + " - cache_ttl must be one of {\\\"5m\\\", \\\"1h\\\"}.\n", + " - Otherwise the scenario is valid.\"\"\"\n", + " prompt_caching = scenario.get('prompt_caching', False)\n", + " cache_ttl = scenario.get('cache_ttl', '5m')\n", + " cached_context = scenario.get('cached_context', None)\n", + "\n", + " if prompt_caching and (cached_context is None or not str(cached_context).strip()):\n", + " return False, (\n", + " 'Prompt caching is enabled but cached_context is empty or missing. '\n", + " 'Provide a non-empty cached_context or disable prompt_caching for '\n", + " 'this scenario.'\n", + " )\n", + "\n", + " if cache_ttl not in ('5m', '1h'):\n", + " return False, (\n", + " f\"Unsupported cache_ttl value '{cache_ttl}'. cache_ttl must be \"\n", + " f\"either '5m' or '1h'.\"\n", + " )\n", + "\n", + " return True, ''\n", + "\n", + "def extract_usage_metrics(usage):\n", + " \"\"\"Extract token counts from a Converse usage block.\n", + "\n", + " Returns a dict with input_tokens, output_tokens, cache_read_input_tokens,\n", + " and cache_write_input_tokens. The usage block is the 'usage' object found\n", + " in the converse_stream metadata event. This factors the usage parsing out\n", + " of benchmark so it can be tested directly.\n", + "\n", + " Behavior:\n", + " - input_tokens and output_tokens are read from 'inputTokens' and\n", + " 'outputTokens'.\n", + " - cache_read_input_tokens and cache_write_input_tokens are read from\n", + " 'cacheReadInputTokens' and 'cacheWriteInputTokens' and default to 0\n", + " when the corresponding field is absent.\n", + " - Missing fields never raise; usage may be None or empty.\"\"\"\n", + " usage = usage or {}\n", + " return {\n", + " 'input_tokens': usage.get('inputTokens'),\n", + " 'output_tokens': usage.get('outputTokens'),\n", + " 'cache_read_input_tokens': usage.get('cacheReadInputTokens', 0) or 0,\n", + " 'cache_write_input_tokens': usage.get('cacheWriteInputTokens', 0) or 0,\n", + " }\n", + "\n", + "def derive_cache_hit_rate(cache_read_input_tokens, total_input_tokens):\n", + " \"\"\"Return the per-invocation Cache_Hit_Rate.\n", + "\n", + " Cache_Hit_Rate is cache_read_input_tokens divided by total_input_tokens\n", + " when total_input_tokens is greater than zero, and 0.0 otherwise. Returning\n", + " 0.0 on a zero (or missing) total avoids a divide-by-zero and keeps the\n", + " metric well defined for every invocation.\"\"\"\n", + " if total_input_tokens and total_input_tokens > 0:\n", + " return cache_read_input_tokens / total_input_tokens\n", + " return 0.0\n", + "\n", "def post_iteration(scenario_config):\n", " logging.info(f'Sleeping for {scenario_config[\"sleep_between_invocations\"]} seconds.')\n", " time.sleep(scenario_config[\"sleep_between_invocations\"])\n", "\n", - "def benchmark(bedrock, file_path, prompt, latency_inference_profile, max_tokens, model_id=\"\", stream=True, sleep_on_throttling=5):\n", + "def benchmark(bedrock, file_path, prompt, latency_inference_profile, max_tokens, model_id=\"\", stream=True, sleep_on_throttling=5, prompt_caching=False, cached_context=None, cache_ttl=\"5m\", temperature=0, top_p=1, top_k=None):\n", " accept = 'application/json'\n", " content_type = 'application/json'\n", " api_call_status = 'Success'\n", @@ -256,20 +489,38 @@ " dt = datetime.fromtimestamp(time.time(), tz=pytz.utc)\n", " job_timestamp_iso = dt.strftime('%Y-%m-%dT%H:%M:%SZ')\n", "\n", - " body, inference_config = get_body(model_id, file_path, prompt, max_tokens)\n", + " body, inference_config, additional_model_request_fields = build_converse_request(\n", + " prompt=prompt,\n", + " cached_context=cached_context,\n", + " prompt_caching=prompt_caching,\n", + " cache_ttl=cache_ttl,\n", + " max_tokens=max_tokens,\n", + " temperature=temperature,\n", + " top_p=top_p,\n", + " top_k=top_k,\n", + " model_id=model_id,\n", + " )\n", " output_token_size, input_token_size = None, None\n", + " # Cache token fields default to zero so the return schema stays stable even\n", + " # on a failed invocation or when the response omits cache usage fields.\n", + " # so the CSV schema stays stable.\n", + " cache_read_input_tokens, cache_write_input_tokens = 0, 0\n", "\n", " while True:\n", " try:\n", " start = time.time()\n", - " response = bedrock.converse_stream(\n", - " messages=body,\n", - " modelId=model_id,\n", - " inferenceConfig=inference_config,\n", - " performanceConfig={\n", - " 'latency': latency_inference_profile\n", - " }\n", - " )\n", + " converse_kwargs = {\n", + " 'messages': body,\n", + " 'modelId': model_id,\n", + " 'inferenceConfig': inference_config,\n", + " 'performanceConfig': {'latency': latency_inference_profile},\n", + " }\n", + " # Pass additionalModelRequestFields (for example top_k) only when the\n", + " # request builder produced it, so models that reject the field are\n", + " # not sent an empty/invalid parameter.\n", + " if additional_model_request_fields is not None:\n", + " converse_kwargs['additionalModelRequestFields'] = additional_model_request_fields\n", + " response = bedrock.converse_stream(**converse_kwargs)\n", " first_byte = None\n", " event_stream = response.get('stream')\n", " for event in event_stream: \n", @@ -283,8 +534,11 @@ " elif 'metadata' in event:\n", " metadata = event['metadata']\n", " if 'usage' in metadata:\n", - " output_token_size = metadata['usage'].get('outputTokens', None)\n", - " input_token_size = metadata['usage'].get('inputTokens', None)\n", + " usage_metrics = extract_usage_metrics(metadata['usage'])\n", + " output_token_size = usage_metrics['output_tokens']\n", + " input_token_size = usage_metrics['input_tokens']\n", + " cache_read_input_tokens = usage_metrics['cache_read_input_tokens']\n", + " cache_write_input_tokens = usage_metrics['cache_write_input_tokens']\n", " last_byte = time.time()\n", " duration_to_first_byte = round(first_byte - start, 2)\n", " duration_to_last_byte = round(last_byte - start, 2)\n", @@ -296,7 +550,7 @@ " break\n", " else:\n", " break\n", - " return duration_to_first_byte, duration_to_last_byte, job_timestamp_iso, api_call_status, full_error_message, output_token_size, input_token_size\n", + " return duration_to_first_byte, duration_to_last_byte, job_timestamp_iso, api_call_status, full_error_message, output_token_size, input_token_size, cache_read_input_tokens, cache_write_input_tokens\n", "\n", "def execute_benchmark(client, scenarios, scenario_config, num_parallel_calls=4, early_break=False):\n", " pp = pprint.PrettyPrinter(indent=2)\n", @@ -306,27 +560,78 @@ " local_client = get_bedrock_client(scenario['region'])\n", " local_invocations = []\n", " file_path = scenario['file_path']\n", - " prompt = scenario['prompt']\n", - " \n", + " # Resolve fields tolerant of both parse_scenario output (text_prompt,\n", + " # inference_profile, expected_output_tokens) and pre-derived keys so the\n", + " # run loop can build scenarios directly via parse_scenario.\n", + " prompt = scenario.get('prompt', scenario.get('text_prompt'))\n", + " latency_inference_profile = scenario.get('latency_inference_profile', scenario.get('inference_profile', 'optimized'))\n", + " configured_output_tokens = scenario.get('configured_output_tokens_for_request', scenario.get('expected_output_tokens', 100))\n", + " stream = scenario.get('stream', True)\n", + " prompt_caching = scenario.get('prompt_caching', PROMPT_CACHING)\n", + " cached_context = scenario.get('cached_context', None)\n", + " cache_ttl = scenario.get('cache_ttl', '5m')\n", + "\n", + " # Validate caching fields before any Bedrock call.\n", + " # An invalid configuration is recorded as a CachingConfigError invocation\n", + " # and the scenario is skipped without calling Bedrock, so the run continues\n", + " # and the CSV schema stays stable.\n", + " is_valid, validation_error = validate_caching_fields(scenario)\n", + " if not is_valid:\n", + " invocation = {\n", + " 'time_to_first_byte': None,\n", + " 'time_to_last_byte': None,\n", + " 'job_timestamp_iso': get_timestamp(),\n", + " 'configured_output_tokens_for_request': configured_output_tokens,\n", + " 'model_input_tokens': None,\n", + " 'model_output_tokens': None,\n", + " 'model': scenario['model_id'],\n", + " 'region': scenario['region'],\n", + " 'invocation_id': 0,\n", + " 'api_call_status': 'CachingConfigError',\n", + " 'full_error_message': validation_error,\n", + " 'TEMPERATURE': TEMPERATURE,\n", + " 'TOP_P': TOP_P,\n", + " 'TOP_K': TOP_K,\n", + " 'EXPERIMENT_NAME': EXPERIMENT_NAME,\n", + " 'task_type': scenario['task_type'],\n", + " 'inference_profile': latency_inference_profile,\n", + " 'prompt_caching': prompt_caching,\n", + " 'cache_ttl': cache_ttl,\n", + " 'cache_read_input_tokens': 0,\n", + " 'cache_write_input_tokens': 0,\n", + " 'cache_hit_rate': derive_cache_hit_rate(0, None),\n", + " }\n", + " local_invocations.append(invocation)\n", + " with logging_lock:\n", + " logging.error(f\"Skipping scenario {scenario['model_id']} due to caching config error: {validation_error}\")\n", + " return local_invocations\n", + "\n", " for invocation_id in range(scenario_config[\"invocations_per_scenario\"]):\n", " try:\n", " time_to_first_byte, time_to_last_byte, job_timestamp_iso, api_call_status, \\\n", - " full_error_message, model_output_tokens, model_input_tokens = benchmark(\n", + " full_error_message, model_output_tokens, model_input_tokens, \\\n", + " cache_read_input_tokens, cache_write_input_tokens = benchmark(\n", " local_client,\n", " file_path,\n", " prompt,\n", - " latency_inference_profile=scenario['latency_inference_profile'],\n", - " max_tokens=scenario['configured_output_tokens_for_request'],\n", + " latency_inference_profile=latency_inference_profile,\n", + " max_tokens=configured_output_tokens,\n", " model_id=scenario['model_id'],\n", - " stream=scenario['stream'],\n", - " sleep_on_throttling=scenario_config['sleep_between_invocations']\n", + " stream=stream,\n", + " sleep_on_throttling=scenario_config['sleep_between_invocations'],\n", + " prompt_caching=prompt_caching,\n", + " cached_context=cached_context,\n", + " cache_ttl=cache_ttl,\n", + " temperature=TEMPERATURE,\n", + " top_p=TOP_P,\n", + " top_k=TOP_K\n", " )\n", "\n", " invocation = {\n", " 'time_to_first_byte': time_to_first_byte,\n", " 'time_to_last_byte': time_to_last_byte,\n", " 'job_timestamp_iso': job_timestamp_iso,\n", - " 'configured_output_tokens_for_request': scenario['configured_output_tokens_for_request'],\n", + " 'configured_output_tokens_for_request': configured_output_tokens,\n", " 'model_input_tokens': model_input_tokens,\n", " 'model_output_tokens': model_output_tokens,\n", " 'model': scenario['model_id'],\n", @@ -339,7 +644,12 @@ " 'TOP_K': TOP_K,\n", " 'EXPERIMENT_NAME': EXPERIMENT_NAME,\n", " 'task_type': scenario['task_type'],\n", - " 'inference_profile': scenario['latency_inference_profile'],\n", + " 'inference_profile': latency_inference_profile,\n", + " 'prompt_caching': prompt_caching,\n", + " 'cache_ttl': cache_ttl,\n", + " 'cache_read_input_tokens': cache_read_input_tokens,\n", + " 'cache_write_input_tokens': cache_write_input_tokens,\n", + " 'cache_hit_rate': derive_cache_hit_rate(cache_read_input_tokens, model_input_tokens),\n", " }\n", " local_invocations.append(invocation)\n", " \n", @@ -350,8 +660,36 @@ " post_iteration(scenario_config=scenario_config)\n", " \n", " except Exception as e:\n", + " # Fail-soft: any failed invocation records its status and full\n", + " # error message, then the run continues. Cache\n", + " # token columns are zeroed so the CSV schema stays stable.\n", " with logging_lock:\n", " logging.error(f\"Error while processing scenario: {scenario['model_id']}. Error: {e}\")\n", + " invocation = {\n", + " 'time_to_first_byte': None,\n", + " 'time_to_last_byte': None,\n", + " 'job_timestamp_iso': get_timestamp(),\n", + " 'configured_output_tokens_for_request': configured_output_tokens,\n", + " 'model_input_tokens': None,\n", + " 'model_output_tokens': None,\n", + " 'model': scenario['model_id'],\n", + " 'region': scenario['region'],\n", + " 'invocation_id': invocation_id,\n", + " 'api_call_status': 'InvocationError',\n", + " 'full_error_message': str(e),\n", + " 'TEMPERATURE': TEMPERATURE,\n", + " 'TOP_P': TOP_P,\n", + " 'TOP_K': TOP_K,\n", + " 'EXPERIMENT_NAME': EXPERIMENT_NAME,\n", + " 'task_type': scenario['task_type'],\n", + " 'inference_profile': latency_inference_profile,\n", + " 'prompt_caching': prompt_caching,\n", + " 'cache_ttl': cache_ttl,\n", + " 'cache_read_input_tokens': 0,\n", + " 'cache_write_input_tokens': 0,\n", + " 'cache_hit_rate': derive_cache_hit_rate(0, None),\n", + " }\n", + " local_invocations.append(invocation)\n", " \n", " return local_invocations\n", "\n", @@ -383,29 +721,30 @@ " return all_invocations\n", "\n", "if __name__ == \"__main__\":\n", + " # Version gate: stop before any invocation when boto3 is too old to support\n", + " # cachePoint in Converse and the latest listed models.\n", + " is_boto3_satisfied, boto3_version_message = check_boto3_version(boto3.__version__, MIN_BOTO3_VERSION)\n", + " print(boto3_version_message)\n", + " if not is_boto3_satisfied:\n", + " logging.error(boto3_version_message)\n", + " raise SystemExit(boto3_version_message)\n", + "\n", " use_cases_scenarios = []\n", "\n", " # Read the JSONL file and process each line\n", " with open(file_path, 'r', encoding='utf-8') as f:\n", " for line in f:\n", - " file = json.loads(line.strip())\n", - " prompt = file.get('text_prompt')\n", - " task_type = file.get('task_type')\n", - " model_id = file.get('model_id')\n", - " region = file.get('region')\n", - " latency_inference_profile = file.get('inference_profile', 'optimized')\n", - "\n", - " out_tokens = file.get('expected_output_tokens', 100)\n", - " use_cases_scenarios.append({\n", - " \"file_path\": file_path,\n", - " \"configured_output_tokens_for_request\": out_tokens,\n", - " \"prompt\": prompt,\n", - " \"stream\": True,\n", - " \"model_id\": model_id,\n", - " \"region\": region,\n", - " \"task_type\": task_type,\n", - " \"latency_inference_profile\": latency_inference_profile\n", - " })\n", + " raw = json.loads(line.strip())\n", + " # Build each scenario via parse_scenario so it carries the resolved\n", + " # caching fields (prompt_caching, cache_ttl, cached_context) with the\n", + " # global PROMPT_CACHING default applied. Datasets without the new\n", + " # fields behave exactly as before.\n", + " use_cases_scenarios.append(parse_scenario(raw, file_path, PROMPT_CACHING))\n", + "\n", + " # The main-thread client below is a placeholder; each scenario opens its\n", + " # own regional client inside process_scenario. Derive a region for it from\n", + " # the first parsed scenario so the variable is always defined.\n", + " region = use_cases_scenarios[0]['region'] if use_cases_scenarios else 'us-east-1'\n", "\n", " # Main loop\n", " run_count = 1\n", @@ -505,6 +844,19 @@ "\n", "def calculate_metrics(df, group_columns):\n", " \"\"\"Calculate latency metrics grouped by model, region, and inference profile.\"\"\"\n", + " # Be robust to older CSVs that predate the cache columns: if a cache\n", + " # column is absent in the combined DataFrame, treat it as 0 so analysis\n", + " # of pre-existing result files still works.\n", + " for cache_col in ['cache_read_input_tokens', 'cache_write_input_tokens']:\n", + " if cache_col not in df.columns:\n", + " df[cache_col] = 0\n", + " df['cache_read_input_tokens'] = pd.to_numeric(df['cache_read_input_tokens'], errors='coerce').fillna(0)\n", + " df['cache_write_input_tokens'] = pd.to_numeric(df['cache_write_input_tokens'], errors='coerce').fillna(0)\n", + "\n", + " # Derive is_cached per invocation: a cache hit means cache read tokens > 0\n", + " # A cache hit means cache read tokens are above zero.\n", + " df['is_cached'] = df['cache_read_input_tokens'] > 0\n", + "\n", " metrics = df.groupby(group_columns).agg({\n", " 'time_to_first_byte': ['count', 'mean', 'median', \n", " lambda x: x.quantile(0.9), \n", @@ -527,6 +879,32 @@ " otps_metrics.columns = ['OTPS_mean', 'OTPS_p50', 'OTPS_p90', 'OTPS_std']\n", " \n", " metrics = pd.concat([metrics, otps_metrics], axis=1)\n", + "\n", + " # TTFT reported separately for cached vs uncached invocations\n", + " # Each group is reindexed to the full metrics index so\n", + " # a model with no cached (or no uncached) samples yields NaN instead of\n", + " # dropping from the table.\n", + " ttft_cached_mean = df[df['is_cached']].groupby(group_columns)['time_to_first_byte'].mean().round(3)\n", + " ttft_uncached_mean = df[~df['is_cached']].groupby(group_columns)['time_to_first_byte'].mean().round(3)\n", + " metrics['TTFT_cached_mean'] = ttft_cached_mean.reindex(metrics.index)\n", + " metrics['TTFT_uncached_mean'] = ttft_uncached_mean.reindex(metrics.index)\n", + "\n", + " # Aggregated cache token columns.\n", + " cache_metrics = df.groupby(group_columns).agg({\n", + " 'cache_read_input_tokens': ['mean'],\n", + " 'cache_write_input_tokens': ['mean'],\n", + " }).round(3)\n", + " cache_metrics.columns = ['avg_cache_read_input_tokens', 'avg_cache_write_input_tokens']\n", + " metrics = pd.concat([metrics, cache_metrics], axis=1)\n", + "\n", + " # Aggregated cache_hit_rate as total cache read tokens over total input\n", + " # tokens per group; 0.0 when a group has no input tokens.\n", + " grouped = df.groupby(group_columns)\n", + " total_cache_read = grouped['cache_read_input_tokens'].sum()\n", + " total_input = grouped['model_input_tokens'].sum()\n", + " cache_hit_rate = (total_cache_read / total_input).where(total_input > 0, 0.0).round(3)\n", + " metrics['cache_hit_rate'] = cache_hit_rate.reindex(metrics.index)\n", + "\n", " return metrics\n", "\n", "def create_performance_summary_tables(df, metrics, pdf):\n", diff --git a/model-latency-benchmarking/readme.md b/model-latency-benchmarking/readme.md index c7fe36bc4..571763772 100644 --- a/model-latency-benchmarking/readme.md +++ b/model-latency-benchmarking/readme.md @@ -8,51 +8,122 @@ This tool helps you benchmark model latency metrics for Large Language Models (L 2. Measures how many tokens the model generates per second (Output Tokens Per Second) 3. Tests different model versions and settings 4. Handles multiple API calls at once -5. Provides analysis and statistics that you can use to take decisions +5. Supports optional, opt-in prompt caching per scenario and captures cache metrics +6. Provides analysis and statistics that you can use to take decisions -## How to use it +## Prerequisites + +- AWS account with Amazon Bedrock access. +- Model access enabled for every model you reference. The latest models (Claude Opus 4.5, Claude Sonnet 4.5, Claude Haiku 4.5, Claude 3.7 Sonnet, and Amazon Nova Pro) require model access to be enabled in the account and region where you run the benchmark. Enable access in the Amazon Bedrock console under Model access before running the tool. +- A recent `boto3` version. The notebook declares a minimum required `boto3` version that supports the latest Bedrock models and prompt caching through the Converse API. If your installed version is below that minimum, the notebook reports the requirement before invoking any model. Upgrade with `pip install -U boto3`. +- A file with your prompts in `JSONL` format (see the required format below). +- Access to the AWS region you want to use. -1. Make sure you have: - - AWS account with Amazon Bedrock access - - A file with your prompts (in `JSONL format` check required format below) - - Access to the AWS region you want to use +## How to use it -2. Set up your test: +1. Set up your test: - Put your prompts in a JSONL file - Change settings in the first code cell (like file paths and test details) + - Some models (for example Amazon Nova and the latest Claude models) reject `temperature` and `topP` in the same request, so the tool sends only one. Set `INFERENCE_SAMPLING` in the configuration cell to `'temperature'` (default) or `'topP'` to choose which one. -3. Run the code: +2. Run the code: - All cells will run automatically - Results will be saved in your chosen folder - You'll get a log file with details about what happened -4. Check your results: +3. Check your results: - Look at the CSV files for detailed metrics - Review the final analysis for overall performance ## Required Dataset Format -Your input JSONL file should contain one JSON object per line with the following fields: +Your input JSONL file should contain one JSON object per line. The following fields are supported. Existing datasets keep working unchanged: the caching fields are optional and additive, and a scenario without them behaves exactly as before. + +| Field | Type | Required | Default | Notes | +|-------|------|----------|---------|-------| +| `text_prompt` | string | yes | - | The user prompt. | +| `expected_output_tokens` | int | no | 100 | Maximum output tokens for the request. | +| `task_type` | string | no | - | Currently supports `Text-Generation`. | +| `model_id` | string | yes | - | Model ID or cross-region inference profile ID. | +| `region` | string | yes | - | AWS region where the model is invoked. | +| `inference_profile` | string | no | `optimized` | Latency setting passed via Converse `performanceConfig`, either `optimized` or `standard`. | +| `prompt_caching` | bool | no | `false` | Opt-in prompt caching for the scenario. When omitted, caching is off. | +| `cache_ttl` | string | no | `5m` | Cache checkpoint time-to-live. Allowed values are `5m` and `1h`. The `1h` extended TTL is only supported on Anthropic models; on other models the tool falls back to the default `5m`. | +| `cached_context` | string | no | none | The long, static prompt prefix to cache. Required when `prompt_caching` is `true`. | ```json { "text_prompt": "Your question or instruction here", - "expected_output_tokens": 50, // number of tokens expected in output - "task_type": "Text-Generation", // currently supports Text-Generation - "model_id": "us.meta.llama3-1-70b-instruct-v1:0", // model identifier - "region": "us-west-2", // region where you want to benchmark model latency metrics - "inference_profile": "optimized" // optimization setting + "expected_output_tokens": 50, + "task_type": "Text-Generation", + "model_id": "us.anthropic.claude-haiku-4-5-20251001-v1:0", + "region": "us-east-1", + "inference_profile": "optimized" } ``` #### Example entries from the test dataset: ```json -{"text_prompt": "Summarize the key features of cloud computing in one sentence.", "expected_output_tokens": 50, "task_type": "Text-Generation", "model_id": "us.meta.llama3-1-70b-instruct-v1:0", "region": "us-east-2", "inference_profile": "optimized"} -{"text_prompt": "Explain the concept of machine learning in simple terms.", "expected_output_tokens": 50, "task_type": "Text-Generation", "model_id": "us.anthropic.claude-3-5-haiku-20241022-v1:0", "region": "us-east-2", "inference_profile": "optimized"} -{"text_prompt": "Explain the concept of machine learning in simple terms.", "expected_output_tokens": 50, "task_type": "Text-Generation", "model_id": "us.anthropic.claude-3-5-haiku-20241022-v1:0", "region": "us-east-2", "inference_profile": "standard"} +{"text_prompt": "Explain the concept of machine learning in simple terms.", "expected_output_tokens": 50, "task_type": "Text-Generation", "model_id": "us.anthropic.claude-opus-4-5-20251101-v1:0", "region": "us-east-1", "inference_profile": "standard"} +{"text_prompt": "Explain the concept of machine learning in simple terms.", "expected_output_tokens": 50, "task_type": "Text-Generation", "model_id": "us.anthropic.claude-sonnet-4-5-20250929-v1:0", "region": "us-east-1", "inference_profile": "optimized"} +{"text_prompt": "Explain the concept of machine learning in simple terms.", "expected_output_tokens": 50, "task_type": "Text-Generation", "model_id": "us.amazon.nova-pro-v1:0", "region": "us-east-1", "inference_profile": "standard"} +``` + +#### Example entry with prompt caching enabled: + +```json +{"text_prompt": "Using the reference document above, summarize the key points.", "expected_output_tokens": 200, "task_type": "Text-Generation", "model_id": "us.anthropic.claude-haiku-4-5-20251001-v1:0", "region": "us-east-1", "inference_profile": "standard", "prompt_caching": true, "cache_ttl": "5m", "cached_context": ""} ``` +## Prompt caching + +Prompt caching lets you cache a static prompt prefix (the `cached_context`) so that repeated requests reusing that prefix skip reprocessing it. This reduces Time to First Token and lowers input-token cost for workloads that send a large, stable context on every request, such as a system prompt, a long instruction set, a knowledge document, or few-shot examples. + +How it works in the tool: + +- Set `prompt_caching` to `true` on a scenario and supply the static prefix in `cached_context`. The tool places the cached context first and inserts a cache checkpoint immediately after it, then appends your `text_prompt`. +- Set `cache_ttl` to `5m` (default) or `1h` to control how long the cache entry lives. The `1h` extended TTL is only supported on Anthropic models; other models use the default `5m`. +- When `prompt_caching` is `false` or absent, the request is sent without any cache checkpoint, exactly as before. +- If `prompt_caching` is `true` but `cached_context` is empty, or `cache_ttl` is not `5m` or `1h`, the tool records an error status with a descriptive message for that invocation and continues with the remaining scenarios. + +A global prompt caching toggle in the configuration cell sets the default. A scenario's `prompt_caching` field always wins when present. + +### Supported models, regions, and minimum token thresholds + +Prompt caching is available for **on-demand invocation only**. It is not supported with provisioned throughput. + +Caching only triggers when the cached prefix meets or exceeds a per-model minimum token threshold. Below the threshold, the request still runs but no cache write or read occurs. The thresholds and the list of caching-capable models and supported regions change over time, so confirm the current values in the [Amazon Bedrock prompt caching documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html) for your model and region before authoring a dataset. + +The models demonstrated by this tool and their minimum cached-prefix token thresholds: + +| Model | Inference profile ID | Minimum tokens to trigger caching | +|-------|----------------------|-----------------------------------| +| Claude Haiku 4.5 | `us.anthropic.claude-haiku-4-5-20251001-v1:0` | 4,096 | +| Amazon Nova Pro | `us.amazon.nova-pro-v1:0` | 1,000 (1K) | + +Only reference caching-capable models for any scenario where `prompt_caching` is `true`. + +## Cache metrics in output + +When prompt caching is active, the tool captures cache usage from the Converse response and adds these columns to the per-invocation output. When a response omits the cache fields, the tool records zero for both token counts so the output schema stays stable. + +| Metric | Description | +|--------|-------------| +| `cache_read_input_tokens` | Input tokens served from cache on a cache hit. `0` when absent. | +| `cache_write_input_tokens` | Input tokens written to the cache when it is populated. `0` when absent. | +| `Cache_Hit_Rate` | Fraction of input tokens served from cache, derived per invocation as cache read input tokens divided by total input tokens. | + +The analysis cell reports Time to First Token for cached invocations separately from uncached invocations and includes the cache token columns and Cache_Hit_Rate in the aggregated output. All existing per-invocation columns and aggregated metrics are preserved; the cache columns are added rather than replacing anything. + +## Caching demonstration dataset + +The repository ships a dedicated demo dataset, `caching-demo-prompts-for-benchmarking.jsonl`, alongside the sample dataset. Its purpose is to let you observe the cache effect directly without authoring your own data. + +It demonstrates prompt caching on two models, Claude Haiku 4.5 and Amazon Nova Pro. For each model it pairs a cached scenario (`prompt_caching: true`) with a matching uncached baseline (`prompt_caching: false`) that uses the same model and prompt, so cached and uncached results compare directly. The cached context in each scenario meets the per-model minimum token threshold so caching is triggered. + +To run it, point the dataset file path in the configuration cell at `caching-demo-prompts-for-benchmarking.jsonl` and run the notebook as usual. No tool changes are needed. After the run, compare the cached scenarios against their baselines: cached invocations should report non-zero `cache_read_input_tokens` or `cache_write_input_tokens` and lower Time to First Token. Because the first request pays the cache write cost while later requests benefit from the read, run enough invocations per scenario to warm the cache before comparing cache-hit latency. + ## Important notes - Test results depend on your specific prompts and setup @@ -60,4 +131,4 @@ Your input JSONL file should contain one JSON object per line with the following - More test runs give more accurate results ## Need help? -Check the comments in the code for more detailed information about each part of the framework or open an issue. \ No newline at end of file +Check the comments in the code for more detailed information about each part of the framework or open an issue. From cd0a946a801ada49549ae7a242e8817954ad5aa1 Mon Sep 17 00:00:00 2001 From: Evgeni Sokolov Date: Fri, 12 Jun 2026 15:53:52 +0200 Subject: [PATCH 2/2] Trim caching docs and drop oversized sample prompt - Remove the large cached-context scenario from the sample dataset; the caching demo dataset already covers caching end to end. - Condense the readme caching docs into a single concise section that does not repeat the dataset field table, aligning with the original style. --- .../dummy-prompts-for-benchmarking.jsonl | 1 - model-latency-benchmarking/readme.md | 50 +++---------------- 2 files changed, 6 insertions(+), 45 deletions(-) diff --git a/model-latency-benchmarking/dummy-prompts-for-benchmarking.jsonl b/model-latency-benchmarking/dummy-prompts-for-benchmarking.jsonl index eeff9469c..12c24006e 100644 --- a/model-latency-benchmarking/dummy-prompts-for-benchmarking.jsonl +++ b/model-latency-benchmarking/dummy-prompts-for-benchmarking.jsonl @@ -2,4 +2,3 @@ {"text_prompt": "Explain the concept of machine learning in simple terms.", "expected_output_tokens": 50, "task_type": "Text-Generation", "model_id": "us.anthropic.claude-sonnet-4-5-20250929-v1:0", "region": "us-east-1", "inference_profile": "optimized"} {"text_prompt": "Explain the concept of machine learning in simple terms.", "expected_output_tokens": 50, "task_type": "Text-Generation", "model_id": "us.anthropic.claude-3-7-sonnet-20250219-v1:0", "region": "us-west-2", "inference_profile": "standard"} {"text_prompt": "Explain the concept of machine learning in simple terms.", "expected_output_tokens": 50, "task_type": "Text-Generation", "model_id": "us.amazon.nova-pro-v1:0", "region": "us-east-1", "inference_profile": "standard"} -{"text_prompt": "Based on the platform engineering handbook above, summarize the recommended approach to multi-account governance and incident response.", "expected_output_tokens": 100, "task_type": "Text-Generation", "model_id": "us.anthropic.claude-haiku-4-5-20251001-v1:0", "region": "us-east-1", "inference_profile": "standard", "prompt_caching": true, "cache_ttl": "5m", "cached_context": "GlobalRetail Platform Engineering Handbook. Section 1. Purpose and Scope. This handbook describes the technical standards, operational practices, and architectural principles that govern the GlobalRetail cloud platform. It applies to every engineering team that builds, deploys, or operates workloads on the shared platform. The goal of this document is to provide a single authoritative reference so that teams make consistent decisions, reduce operational risk, and accelerate delivery without sacrificing reliability or security. Every standard in this handbook exists because of a lesson learned during an incident, an audit finding, or a cost overrun. Teams are expected to read the relevant sections before designing a new service and to revisit them during architecture reviews. Section 2. Organizational Principles. GlobalRetail organizes engineering around small autonomous teams that own their services from design through production support. Each team is responsible for the full lifecycle of the services it owns, including on-call duties, capacity planning, security posture, and cost management. Ownership is explicit and recorded in the service catalog, which maps every production service to a single owning team, a primary on-call rotation, and an escalation path. Teams operate with a high degree of independence, but that independence is bounded by the platform standards described here. The platform team provides paved roads, which are pre-approved patterns and tooling that make the secure and reliable choice the easy choice. Teams that follow the paved road inherit security controls, observability, and deployment automation by default. Teams that need to leave the paved road must document the deviation, justify it during an architecture review, and accept responsibility for the additional operational burden. Section 3. Account Structure and Multi-Account Governance. The platform uses a multi-account strategy to provide strong isolation boundaries between workloads, environments, and teams. Each team receives separate accounts for development, staging, and production. Production accounts are subject to the strictest controls and the most conservative change management. Accounts are organized into a hierarchy of organizational units that reflect business function and environment. Security and compliance controls are applied centrally through service control policies attached to organizational units, so that guardrails cannot be removed by individual teams. The management account is reserved for billing, organization administration, and the configuration of organization wide policies. No workloads run in the management account. A dedicated security account aggregates audit logs, findings, and configuration snapshots from every other account. A dedicated logging account stores immutable copies of access logs and audit trails with restrictive retention and deletion protection. Network connectivity between accounts is established through a central transit hub, and direct peering between team accounts is discouraged in favor of the hub and spoke model, which centralizes inspection and routing. Cross account access always uses temporary credentials obtained through role assumption rather than long lived access keys. Long lived access keys are prohibited in all environments, and automated scanners detect and report any that appear. Section 4. Identity and Access Management. Access to the platform follows the principle of least privilege. Human access is granted through federated single sign on, and no human is given standing access to production. Access to production is requested just in time, approved by a second engineer, and granted for a limited window after which it expires automatically. All access is logged and reviewed. Service to service access uses scoped roles with narrowly defined permissions. Wildcard permissions are forbidden in production policies except where a specific documented exception has been approved. Permission boundaries are applied to every role that teams can create themselves, so that even a misconfigured policy cannot exceed the boundary set by the platform. Secrets such as database passwords and third party API keys are stored in a managed secrets service, encrypted at rest, and rotated automatically. Secrets are never committed to source control, embedded in container images, or printed in logs. Automated pipelines scan every commit and every image for secrets and block the build when a secret is detected. Section 5. Networking Standards. Every workload runs inside a virtual private network with private subnets for compute and data tiers and public subnets reserved for load balancers and managed ingress. Direct internet exposure of compute instances is prohibited. Outbound internet access from private subnets is routed through managed network address translation, and egress is restricted to approved destinations where the threat model requires it. Internal traffic between services is encrypted in transit, and mutual authentication is required for sensitive internal communication. Security groups are configured to allow only the specific ports and sources that a service requires, and broad rules that allow traffic from any source are rejected during review. Network access control lists provide a coarse second layer of defense at the subnet boundary. DNS is managed centrally so that service discovery is consistent across accounts, and private hosted zones are used for internal names that should never resolve publicly. Section 6. Compute and Container Standards. Stateless services are the default. Teams are encouraged to design services that hold no durable state locally so that instances can be replaced freely and scaled horizontally. Containerized workloads run on the managed orchestration platform, and container images are built from approved minimal base images that are patched centrally. Images are scanned for vulnerabilities before they are allowed into the production registry, and images with critical findings are blocked. Every container declares its resource requests and limits so that the scheduler can place workloads efficiently and prevent noisy neighbor problems. Autoscaling is configured based on meaningful application metrics rather than raw infrastructure metrics alone, so that capacity tracks real demand. Serverless functions are preferred for event driven and bursty workloads because they reduce idle cost and operational overhead. Long running batch jobs use managed batch services with spot capacity where interruption is tolerable, which significantly reduces cost. Section 7. Data Management and Storage. Data is classified into public, internal, confidential, and restricted tiers, and the classification determines the controls that apply. Confidential and restricted data is encrypted at rest using customer managed keys, and access to those keys is tightly controlled and audited. Databases are deployed in private subnets and are never directly reachable from the internet. Backups are automated, encrypted, and tested through periodic restore exercises, because a backup that has never been restored is not a backup. Point in time recovery is enabled for critical databases. Object storage buckets are private by default, and public access is blocked at the account level so that an individual misconfiguration cannot expose data. Lifecycle policies move infrequently accessed data to lower cost storage tiers and expire data that is no longer needed, which controls both cost and compliance risk. Data retention follows the legal and regulatory requirements for each data class, and teams document the retention period for every data store they own. Section 8. Observability and Monitoring. Every service emits structured logs, metrics, and distributed traces. Logs are written in a consistent structured format with correlation identifiers so that a single request can be followed across service boundaries. Metrics cover both technical health, such as latency, error rate, and saturation, and business health, such as orders placed and carts abandoned. Dashboards present the golden signals for each service, and alerts are defined against service level objectives rather than arbitrary thresholds. Service level objectives are agreed with stakeholders and expressed as a target reliability over a rolling window, with an error budget that governs how aggressively a team can ship changes. When the error budget is exhausted, teams slow down and prioritize reliability work over new features. Traces are sampled intelligently so that the system captures enough detail to debug problems without overwhelming storage. Alerts are actionable, meaning every alert links to a runbook and describes a condition that a human can and should respond to. Noisy alerts that do not require action are tuned or removed, because alert fatigue is itself an operational risk. Section 9. Deployment and Change Management. All infrastructure is defined as code and stored in version control. Manual changes to production through consoles are prohibited except during a declared emergency, and any emergency change must be reconciled back into code afterward. Deployments are automated through pipelines that run tests, security scans, and policy checks before any change reaches production. Changes are rolled out progressively using canary or blue green strategies so that a bad change affects a small fraction of traffic before it is detected and rolled back automatically. Every deployment is reversible, and teams maintain the ability to roll back quickly. Database schema changes are designed to be backward compatible so that application and schema can be deployed independently. Feature flags decouple deployment from release, allowing teams to deploy code in a dormant state and enable it gradually. Section 10. Reliability and Resilience. Services are designed to tolerate the failure of any single component without customer impact. Critical workloads are deployed across multiple availability zones, and the most critical workloads are designed for multi region operation. Dependencies are treated as unreliable, and every remote call has a timeout, a bounded retry policy with backoff and jitter, and a circuit breaker that prevents a struggling dependency from cascading failures upstream. Bulkheads isolate pools of resources so that a problem in one area cannot exhaust the resources needed elsewhere. Teams conduct regular game days in which they deliberately inject failures into non production and sometimes production environments to validate that their resilience mechanisms work as designed. Capacity is planned ahead of known demand peaks, and load tests validate that services meet their objectives under expected and stress conditions. Section 11. Incident Response. When an incident occurs, the owning team follows a defined incident response process. The first priority is always to restore service for customers, and root cause analysis follows recovery rather than blocking it. An incident commander is designated to coordinate the response, make decisions, and own communication, while responders focus on diagnosis and mitigation. A dedicated communications channel is opened for every significant incident, and status updates are posted at regular intervals to keep stakeholders informed. Severity levels are defined clearly so that the response scales appropriately, with the highest severity reserved for incidents that cause broad customer impact or data risk. After the incident is resolved, the team writes a blameless post incident review that documents the timeline, the contributing factors, the customer impact, and the corrective actions. The review focuses on the systemic and process factors that allowed the incident rather than blaming individuals, because a culture of blame discourages the honest reporting that makes systems safer. Corrective actions are tracked to completion and prioritized against feature work according to the risk they address. Recurring incident themes are escalated to leadership so that systemic investment can be made. Section 12. Security Operations. Security is a shared responsibility across every team rather than the exclusive domain of a central group. Threat detection runs continuously across all accounts, and findings are routed to the owning team with a severity and a remediation deadline. Vulnerabilities in dependencies and base images are tracked, and teams patch within a window defined by the severity of the vulnerability. Penetration tests and security reviews are conducted for significant new services and major changes. Audit logs are immutable and centralized so that an attacker cannot cover their tracks by deleting logs in a compromised account. Access reviews are conducted periodically to remove permissions that are no longer needed, because permissions tend to accumulate over time. Section 13. Cost Management. Cost is treated as a first class operational metric. Every resource is tagged with the owning team, the environment, and the cost center so that spend can be attributed accurately. Teams review their spend regularly and are accountable for the cost of the services they own. The platform provides dashboards that highlight the largest cost drivers and flag anomalies such as a sudden increase in spend. Reserved capacity and savings commitments are used for predictable baseline load, while elastic and spot capacity absorbs variable and interruptible load. Idle and orphaned resources are detected automatically and cleaned up, because forgotten resources are a common and avoidable source of waste. Right sizing recommendations are reviewed during regular operational reviews. Section 14. Architecture Reviews. Significant new services and major changes go through an architecture review before substantial work begins. The review evaluates the design against the standards in this handbook, focusing on security, reliability, cost, and operability. Reviews are collaborative and educational rather than adversarial, and their purpose is to surface risks early when they are cheap to address. The reviewers look for single points of failure, missing failure handling, unclear ownership, and deviations from the paved road that lack justification. The output of a review is a set of agreed actions and any documented exceptions. Exceptions are time bound and revisited so that temporary deviations do not become permanent. Section 15. Testing Standards. Quality is built in rather than inspected at the end. Every service maintains a layered testing strategy that combines fast unit tests, focused integration tests, and a small number of end to end tests that exercise critical user journeys. Unit tests run on every commit and must complete quickly so that engineers receive feedback within minutes. Integration tests validate the contracts between a service and its dependencies, using ephemeral environments that are created and destroyed automatically. Contract tests guard the boundaries between services so that a provider cannot break a consumer without detection. Performance tests validate that services meet their latency and throughput objectives under representative load, and they run before major releases. Test data is synthetic and never contains real customer information, because using production data in test environments creates unacceptable privacy risk. Flaky tests are treated as defects and are quarantined and fixed rather than ignored, because a test suite that engineers do not trust provides no protection. Code coverage is measured as a signal rather than a target, since high coverage of trivial code provides little assurance while thoughtful tests of complex logic provide a great deal. Section 16. Documentation and Knowledge Sharing. Every service maintains living documentation that describes its purpose, its interfaces, its dependencies, its operational characteristics, and its runbooks. Documentation lives close to the code it describes and is reviewed alongside code changes so that it does not drift out of date. Each service publishes an architecture diagram, an interface specification, and an on call guide that a new responder can follow during an incident. Decisions of consequence are recorded as architecture decision records that capture the context, the options considered, the decision, and the consequences, so that future engineers understand why the system is the way it is. Teams share lessons learned through regular technical forums, and significant incidents and their reviews are circulated widely so that the whole organization benefits from each lesson. Onboarding documentation is kept current so that new engineers become productive quickly. Section 17. Sustainability and Efficiency. Engineering teams consider the efficiency of the resources they consume, because efficient systems are usually both cheaper and lower in environmental impact. Teams choose regions and capacity options thoughtfully, decommission unused workloads promptly, and design for elasticity so that resources scale down when demand falls. Efficiency is reviewed during operational reviews alongside cost and reliability. Section 18. Compliance and Audit. The platform operates under several regulatory and contractual obligations, and teams that handle regulated data must understand the controls that apply to them. Compliance requirements are translated into concrete technical controls that are enforced automatically wherever possible, so that compliance is a property of the system rather than a manual checklist. Configuration rules continuously evaluate every account against the required controls and report drift, such as an unencrypted volume or a publicly accessible bucket, so that violations are detected and corrected quickly. Evidence for audits is collected automatically from the centralized logging and configuration stores, which reduces the burden on teams during an audit and ensures that evidence is complete and tamper resistant. Changes to regulated systems are tracked with a clear record of who made the change, what changed, and why, and that record is retained for the period required by the relevant regulation. Teams that are unsure whether a control applies to their workload consult the compliance function early rather than discovering a gap during an audit. Section 19. Vendor and Dependency Management. Every external dependency, whether a software library, a managed service, or a third party API, introduces risk that must be understood and managed. Teams maintain an inventory of the dependencies their services rely on, including the version in use and the license under which it is distributed. Dependencies are kept reasonably current so that the team can apply security patches without first completing a large and risky upgrade. New dependencies are evaluated before adoption for their security posture, their maintenance health, their license compatibility, and the blast radius of a potential failure or compromise. Critical third party services are assessed for their reliability commitments, and teams design graceful degradation so that the failure of a non essential vendor does not take down the whole experience. Where a single vendor represents a concentration risk, teams document the risk and, where justified, design an exit path. Supply chain integrity is protected by verifying the provenance of artifacts and by pinning dependencies to known good versions rather than floating to the latest version automatically. Section 20. Continuous Improvement. The platform and its standards are never finished. Teams regularly reflect on what is working and what is not, through retrospectives after projects and incidents and through ongoing operational reviews. Improvement ideas are captured, prioritized, and acted upon rather than lost, and the platform team treats the paved road itself as a product that must continuously earn the adoption of the teams that use it. Metrics about the health of the engineering system, such as deployment frequency, lead time for changes, change failure rate, and time to restore service, are tracked over time so that the organization can see whether it is getting better. Investments in tooling, automation, and developer experience are justified by their effect on these metrics and on the daily experience of engineers. Section 21. Developer Experience and Platform Adoption. A platform succeeds only when teams choose to use it, so the platform team measures and invests in developer experience deliberately. Setting up a new service from the approved templates should take minutes rather than days, and the templates generate a working pipeline, baseline observability, security controls, and documentation scaffolding automatically. Local development environments mirror production closely enough that surprises are rare, and engineers can run and test their services without depending on shared environments that create contention. Self service tooling lets teams provision the resources they are entitled to without filing tickets and waiting, while policy guardrails ensure that self service never compromises security or cost discipline. Feedback from engineers is gathered regularly through surveys and direct conversation, and the friction points that engineers report most often are prioritized for elimination. The platform publishes a clear catalog of the capabilities it offers, with examples and guidance, so that teams can discover the paved road rather than reinventing solutions that already exist. When a team repeatedly leaves the paved road for the same reason, the platform team treats that as a signal that the paved road has a gap and works to close it, because the most durable way to drive adoption is to make the supported path genuinely the best path. Section 22. Summary. The standards in this handbook exist to help teams move quickly with confidence. By following the paved road, designing for failure, automating everything, measuring what matters, and treating security and cost as everyone's responsibility, teams deliver reliable services that customers trust. This document is living and is updated as the platform evolves and as new lessons are learned."} diff --git a/model-latency-benchmarking/readme.md b/model-latency-benchmarking/readme.md index 571763772..6225abb63 100644 --- a/model-latency-benchmarking/readme.md +++ b/model-latency-benchmarking/readme.md @@ -24,7 +24,7 @@ This tool helps you benchmark model latency metrics for Large Language Models (L 1. Set up your test: - Put your prompts in a JSONL file - Change settings in the first code cell (like file paths and test details) - - Some models (for example Amazon Nova and the latest Claude models) reject `temperature` and `topP` in the same request, so the tool sends only one. Set `INFERENCE_SAMPLING` in the configuration cell to `'temperature'` (default) or `'topP'` to choose which one. + - Some models reject `temperature` and `topP` together; set `INFERENCE_SAMPLING` in the configuration cell to `'temperature'` (default) or `'topP'`. 2. Run the code: - All cells will run automatically @@ -78,51 +78,13 @@ Your input JSONL file should contain one JSON object per line. The following fie ## Prompt caching -Prompt caching lets you cache a static prompt prefix (the `cached_context`) so that repeated requests reusing that prefix skip reprocessing it. This reduces Time to First Token and lowers input-token cost for workloads that send a large, stable context on every request, such as a system prompt, a long instruction set, a knowledge document, or few-shot examples. +Prompt caching reuses a static prompt prefix (`cached_context`) across requests to lower Time to First Token and input-token cost. Set `prompt_caching` to `true` and provide the prefix in `cached_context`; the optional fields are described in the dataset table above. Caching is off by default, so existing datasets are unaffected. -How it works in the tool: +- Available for on-demand invocation only. +- Caching only triggers when the cached prefix meets the model's minimum token threshold (for example, 4,096 tokens for Claude Haiku 4.5). Confirm current thresholds and supported models in the [Amazon Bedrock prompt caching documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html). +- Output adds `cache_read_input_tokens`, `cache_write_input_tokens`, and `Cache_Hit_Rate`, and the analysis reports Time to First Token for cached and uncached calls separately. Existing columns are unchanged. -- Set `prompt_caching` to `true` on a scenario and supply the static prefix in `cached_context`. The tool places the cached context first and inserts a cache checkpoint immediately after it, then appends your `text_prompt`. -- Set `cache_ttl` to `5m` (default) or `1h` to control how long the cache entry lives. The `1h` extended TTL is only supported on Anthropic models; other models use the default `5m`. -- When `prompt_caching` is `false` or absent, the request is sent without any cache checkpoint, exactly as before. -- If `prompt_caching` is `true` but `cached_context` is empty, or `cache_ttl` is not `5m` or `1h`, the tool records an error status with a descriptive message for that invocation and continues with the remaining scenarios. - -A global prompt caching toggle in the configuration cell sets the default. A scenario's `prompt_caching` field always wins when present. - -### Supported models, regions, and minimum token thresholds - -Prompt caching is available for **on-demand invocation only**. It is not supported with provisioned throughput. - -Caching only triggers when the cached prefix meets or exceeds a per-model minimum token threshold. Below the threshold, the request still runs but no cache write or read occurs. The thresholds and the list of caching-capable models and supported regions change over time, so confirm the current values in the [Amazon Bedrock prompt caching documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html) for your model and region before authoring a dataset. - -The models demonstrated by this tool and their minimum cached-prefix token thresholds: - -| Model | Inference profile ID | Minimum tokens to trigger caching | -|-------|----------------------|-----------------------------------| -| Claude Haiku 4.5 | `us.anthropic.claude-haiku-4-5-20251001-v1:0` | 4,096 | -| Amazon Nova Pro | `us.amazon.nova-pro-v1:0` | 1,000 (1K) | - -Only reference caching-capable models for any scenario where `prompt_caching` is `true`. - -## Cache metrics in output - -When prompt caching is active, the tool captures cache usage from the Converse response and adds these columns to the per-invocation output. When a response omits the cache fields, the tool records zero for both token counts so the output schema stays stable. - -| Metric | Description | -|--------|-------------| -| `cache_read_input_tokens` | Input tokens served from cache on a cache hit. `0` when absent. | -| `cache_write_input_tokens` | Input tokens written to the cache when it is populated. `0` when absent. | -| `Cache_Hit_Rate` | Fraction of input tokens served from cache, derived per invocation as cache read input tokens divided by total input tokens. | - -The analysis cell reports Time to First Token for cached invocations separately from uncached invocations and includes the cache token columns and Cache_Hit_Rate in the aggregated output. All existing per-invocation columns and aggregated metrics are preserved; the cache columns are added rather than replacing anything. - -## Caching demonstration dataset - -The repository ships a dedicated demo dataset, `caching-demo-prompts-for-benchmarking.jsonl`, alongside the sample dataset. Its purpose is to let you observe the cache effect directly without authoring your own data. - -It demonstrates prompt caching on two models, Claude Haiku 4.5 and Amazon Nova Pro. For each model it pairs a cached scenario (`prompt_caching: true`) with a matching uncached baseline (`prompt_caching: false`) that uses the same model and prompt, so cached and uncached results compare directly. The cached context in each scenario meets the per-model minimum token threshold so caching is triggered. - -To run it, point the dataset file path in the configuration cell at `caching-demo-prompts-for-benchmarking.jsonl` and run the notebook as usual. No tool changes are needed. After the run, compare the cached scenarios against their baselines: cached invocations should report non-zero `cache_read_input_tokens` or `cache_write_input_tokens` and lower Time to First Token. Because the first request pays the cache write cost while later requests benefit from the read, run enough invocations per scenario to warm the cache before comparing cache-hit latency. +To see it in action, point the dataset path at `caching-demo-prompts-for-benchmarking.jsonl`, which pairs cached and uncached scenarios for Claude Haiku 4.5 and Amazon Nova Pro. ## Important notes