Skip to content

Degrade retrieval to test CI gate - #7

Closed
NaveenBuidl wants to merge 1 commit into
mainfrom
degrade-retrieval
Closed

NaveenBuidl wants to merge 1 commit into
mainfrom
degrade-retrieval

Conversation

@NaveenBuidl

Copy link
Copy Markdown
Owner

Intentional degradation PR to verify the eval CI gate blocks lower retrieval quality.

Change:

  • retrieval_k: 4 -> 1

Expected:

  • Eval Gate should fail if ContextualPrecisionMetric drops below threshold.

Copilot AI review requested due to automatic review settings April 9, 2026 08:28

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR intentionally degrades retrieval quality by lowering the configured number of retrieved chunks to validate that the eval CI gate blocks regressions.

Changes:

  • Commented out the prior retrieval_k: 4 value.
  • Set retrieval_k: 1 in config.yaml to force lower retrieval quality and trigger the CI eval gate.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread config.yaml
Comment on lines +4 to +5
# retrieval_k: 4
retrieval_k: 1

Copilot AI Apr 9, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR changes the committed default retrieval_k to a degraded value. Since the CI workflow already supports overriding retrieval_k via workflow_dispatch (see .github/workflows/eval-gate.yml), consider keeping config.yaml at the normal default and exercising the gate by running the workflow with an override or by introducing a separate CI-only config. This avoids accidentally merging a low-quality default into mainline usage.

Copilot uses AI. Check for mistakes.
@NaveenBuidl NaveenBuidl closed this Apr 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants