feat: Generic Konflux integration for ABEvalFlow evaluation pipeline

[ikrispin]

Summary

Adds ABEvalFlow as a generic, reusable Konflux IntegrationTestScenario. Any Konflux application can reference the ABEvalFlow evaluation pipeline to run A/B testing (A2A agents, MCP servers, skills), security scanning, quality review, and scorecard certification against its built images as part of the CI/CD pipeline.

The framework publishes 7 core Tekton tasks as versioned bundles. Consumers compose their own pipeline by wrapping these core tasks with application-specific deploy/cleanup logic. ABEvalFlow knows nothing about any specific agent, MCP server, or deployment model.

Tested end-to-end with the google-lightspeed-agent example on the stone-prod-p02 Konflux cluster. The pipeline parses the Konflux Snapshot, deploys the agent on a workload cluster, runs 3 Harbor A2A trials with LLM-as-judge scoring, computes the unified scorecard with certification, and returns structured TEST_OUTPUT back to Konflux for release gating.
Successful run: https://konflux-ui.apps.stone-prod-p02.hjvn.p1.openshiftapps.com/ns/ai5-marketplace-tenant/applications/google-lightspeed-agent/pipelineruns/lightspeed-abevalflow-eval-qhx2p

Architecture

Uses a two-cluster model following the standard Konflux pattern for integration tests that need live workloads:

• Konflux cluster (stone-prod-p02) — orchestrates the pipeline, parses Snapshots, computes scorecards, reports TEST_OUTPUT
• Workload cluster (cn-ai-lab / ab-eval-flow) — runs the agent deployment and evaluation where LiteLLM and eval infrastructure are co-located

The evaluation runs either locally (in the Tekton task step, when the target is reachable from the pipeline cluster) or as a remote Pod on the workload cluster (for cross-cluster scenarios). The eval mode is controlled by the EVAL_MODE parameter.

Konflux (stone-prod-p02)                     cn-ai-lab (ab-eval-flow)
├── parse-snapshot                           
├── [consumer: deploy-agent] ── oc login ──> Agent Deployment + Service
├── prepare (clone submission repo)               
├── test (security + quality)                
├── evaluate ──── oc login ────────────────> Eval Pod (Harbor + analyze.py)
│   └─ extracts report.json <──────────────  └─ LiteLLM (svc:4000)
├── analyze-scorecard                          └─ Agent (svc:8000)
├── store                                    
├── emit-result → TEST_OUTPUT                
└── [consumer: cleanup-agent] ─────────────> Deletes Deployment + eval Pod

What's included

Core Tekton tasks (7 files — published as Tekton Bundles)

Generic, reusable tasks under pipeline/tasks/konflux/:

Task	Purpose
parse-snapshot	Extracts container image, git URL, revision from Konflux SNAPSHOT JSON
prepare	Clones submission repo, validates structure, generates eval files
test	Security scanning + quality review (MinIO/DB optional, graceful no-op when secrets missing)
evaluate	Runs evaluation engine (a2a, mcpchecker, harbor, ase) in local or remote mode
analyze-scorecard	Computes report.json, aggregates unified scorecard with certification levels
store	Persists results to PostgreSQL/MinIO (all external deps optional)
emit-result	Maps scorecard pass/warn/fail + certification to Konflux TEST_OUTPUT format

Pipeline + CI

• konflux-eval-pipelinerun.yaml — Generic 7-stage reference PipelineRun with standardized parameter contract
• Makefile — make bundles to push all tasks as Tekton Bundles to Quay
• push-bundles.yaml — GitHub Actions workflow for automated bundle publishing

Config

• config/konflux/secrets-template.yaml — Templates for workload-cluster-credentials, llm-credentials, compass-facts-api with mode-conditional documentation
• Docs/konflux-integration-guide.md — Full integration guide covering parameter contract, submission formats, integration patterns, and quick-start

Example integration

A complete working example with the Google Lightspeed Agent is at:
github.com/ikrispin/abevalflow-konflux-example

This example demonstrates how a consumer wraps ABEvalFlow's core tasks with application-specific deploy-agent and cleanup-agent tasks. The deploy/cleanup tasks are resolved from the example repo via git resolver, while the 7 core tasks are resolved from Tekton Bundles on Quay.

Parameter contract

The generic pipeline accepts these parameters:

Parameter	Description	Default
SNAPSHOT	Konflux Snapshot JSON (automatic)	—
EVAL_ENGINE	a2a, harbor, ase, mcpchecker	a2a
AGENT_ENDPOINT	HTTP endpoint of the A2A agent	""
MCP_URL	URL of the MCP server (for mcpchecker)	""
EVAL_MODE	local (run in task step) or remote (Pod on workload cluster)	local
SUBMISSION_REPO_URL	Git repo with the evaluation definition	—
SUBMISSION_DIR	Directory under submissions/	—
LLM_API_BASE	LLM proxy URL for judging	""
LLM_MODEL	Judge model	claude-sonnet
WORKLOAD_CLUSTER_URL	Required when EVAL_MODE=remote	""
WORKLOAD_NAMESPACE	Required when EVAL_MODE=remote	""
WORKLOAD_CREDENTIALS_SECRET	Secret name for remote mode auth	workload-cluster-credentials

OCI artifacts in Quay

Container image

Image	Purpose
quay.io/rh-ee-ikrispin/abevalflow-eval-base:latest	Pre-built image with Harbor, Python deps, and the A2A adapter. Used by the evaluate task's remote eval Pod.

Tekton Bundles

All 7 core tasks are published as OCI bundles under quay.io/rh-ee-ikrispin/:

Bundle	Task
abevalflow-task-parse-snapshot:0.1	parse-snapshot
abevalflow-task-prepare:0.1	prepare
abevalflow-task-test:0.1	test
abevalflow-task-evaluate:0.1	evaluate
abevalflow-task-analyze-scorecard:0.1	analyze-scorecard
abevalflow-task-store:0.1	store
abevalflow-task-emit-result:0.1	emit-result

The Makefile provides make bundles to push all and make digests to print SHA digests. When moving to production, these should be relocated to a team Quay org (e.g., quay.io/rhecosystemappeng/).

Key design decisions

Generic framework, not application-specific — The core tasks know nothing about any specific agent or MCP server. Application-specific logic (deployment, cleanup, env vars, sidecars) lives in the consumer's repo. ABEvalFlow only needs an endpoint URL and a submission definition.

Local/remote eval modes — The evaluate task supports local mode (runs eval directly in the task step) and remote mode (submits an eval Pod to a workload cluster). Local mode works when the target is reachable from the pipeline cluster. Remote mode is for cross-cluster scenarios.

Multi-engine support — The evaluate task dispatches to A2A (Harbor trials), MCPChecker (MCP server testing), ASE (LLM-as-judge for skills), and Harbor (containerized A/B) based on the EVAL_ENGINE parameter.

Tekton Bundles — All 7 core tasks are published as versioned OCI bundles to Quay and referenced via the bundles resolver. This follows the same pattern as Konflux's own task catalog and allows independent versioning.

Parameterized secrets — The workload cluster credentials secret name is a parameter (default workload-cluster-credentials), not hardcoded. Consumers can use their own secret names.

Dual-publish model — The PipelineRun is resolved from git (via the ITS), but the tasks inside it are resolved from Tekton Bundles. Updating task logic requires both a bundle push and a git push. This is documented in the PipelineRun header.

How another project onboards

Pattern 1: Pre-deployed target (simplest)

For agents or MCP servers that are already running, reference the generic pipeline directly:

apiVersion: appstudio.redhat.com/v1beta2
kind: IntegrationTestScenario
metadata:
  name: abevalflow-eval
  namespace: <their-tenant>
spec:
  application: <their-app>
  resolverRef:
    resolver: git
    resourceKind: pipelinerun
    params:
      - name: url
        value: https://github.com/RHEcosystemAppEng/ABEvalFlow
      - name: revision
        value: main
      - name: pathInRepo
        value: pipeline/integration/konflux-eval-pipelinerun.yaml

They also need: a submission directory, AGENT_ENDPOINT or MCP_URL parameter, and LLM_API_BASE configured.

Pattern 2: Pipeline-deployed target

For targets that need deployment, create a pipeline that wraps ABEvalFlow's core tasks with deploy/cleanup. See the example repo for the full pattern.

Testing

Test environment

• Konflux cluster: stone-prod-p02 (api.stone-prod-p02.hjvn.p1.openshiftapps.com)
• Konflux tenant: ai5-marketplace-tenant
• Workload cluster: cn-ai-lab (api.cn-ai-lab.2vn8.p1.openshiftapps.com)
• Workload namespace: ab-eval-flow
• Target application: google-lightspeed-agent (A2A agent)
• Example repo: ikrispin/abevalflow-konflux-example

Test execution

Testing was done by creating override Snapshots on the Konflux cluster to trigger the IntegrationTestScenario. The example repo's pipeline deployed the agent on the workload cluster using a public image (quay.io/ecosystem-appeng/google-lightspeed-agent), while the 7 core evaluation tasks were resolved from Tekton Bundles on Quay.

Test results

PipelineRun lightspeed-abevalflow-eval-qhx2p — 9/9 tasks succeeded:

Task	Source	Status	Duration
parse-snapshot	Bundle	Succeeded	9s
deploy-agent	Example repo (git)	Succeeded	9s
prepare	Bundle	Succeeded	25s
test	Bundle	Succeeded	7s
evaluate	Bundle	Succeeded	27s
analyze-scorecard	Bundle	Succeeded	17s
store	Bundle	Succeeded	7s
emit-result	Bundle	Succeeded	8s
cleanup-agent	Example repo (git)	Succeeded	7s

What was validated

• Tekton Bundle resolution from Quay (all 7 core tasks)
• Git resolver for consumer-specific tasks (deploy-agent, cleanup-agent from example repo)
• SNAPSHOT JSON parsing and component image extraction
• Cross-cluster agent deployment via remote oc login
• Remote eval Pod execution (Harbor + analyze.py on workload cluster)
• Report.json extraction from pod logs via delimited markers
• Scorecard aggregation with all gates
• TEST_OUTPUT mapping to Konflux format
• Agent cleanup in finally block
• Graceful handling when optional secrets (MinIO, PostgreSQL, Compass) are missing

What was NOT validated

• Using the actual Konflux snapshot image (private registry auth not configured on workload cluster)
• Compass fact push (SSL cert issue from Konflux cluster — non-blocking, scorecard still computed)
• SHA-pinned bundle references (currently using :0.1 tags)
• Local eval mode (tested remote mode only; local mode is for simpler setups)

Known limitations and TODOs

Item	Status	Details
--insecure-skip-tls-verify	TODO in code	Cross-cluster oc calls skip TLS verification. Should inject workload cluster CA cert as a Secret.
Log-based result transfer	TODO in code	report.json is extracted from eval Pod logs via delimited markers. Should migrate to ConfigMap-based transfer for robustness.
Snapshot image auth	Workaround	The example pipeline accepts AGENT_IMAGE_OVERRIDE because the Konflux-built image is private. The generic pipeline doesn't have this issue — consumers provide AGENT_ENDPOINT directly.
Bundle SHA pinning	Deferred	Bundle references use :0.1 tags (mutable). Should pin by @sha256: digest for production. The Makefile's make digests target provides the SHAs.
Compass fact push	SSL issue	The scorecard task on Konflux can't push facts to compass.stage.redhat.com due to SSL cert verification. Non-blocking — scorecard is computed locally.
Runtime package installs	Deferred	Tasks install jq/pip packages at runtime. Could be pre-baked into a custom base image.
Quay org	Temporary	Bundles and eval-base image are under quay.io/rh-ee-ikrispin/. Should be relocated to a team org for production.

Related

• APPENG-5740
• Example integration repo
• Konflux integration guide

[GuyZivRH]

Consolidated PR Review: #60 — Generic Konflux integration

PR: #60
Author: ikrispin
Branch: ikrispin:konflux-generic → main
HEAD: fd2f8072
Size: +2,848 / −0 · 12 files · 2 commits
CI: PASS
Follow-on: #65 (red-team) stacks on this
Consolidated from: htwpl_, nyjsj_, qmxpt_, vbnkp_, xkqpd_ (2026-08-04)

---

Verdict

Request changes on correctness items; architecture is ready to land afterward.

Reviewer stances ranged from Approve → Approve-with-fixes → Request-changes. Consensus on product direction is strongly positive: reusable Konflux ITS via 7 bundle-published tasks, consumer-owned deploy/cleanup, local/remote eval, optional secrets, and TEST_OUTPUT gating are the right design, and the Lightspeed E2E on stone-prod-p02 is convincing.

Do not merge until at least: (1) remote mode uses the prepared/submission repo correctly, (2) Failed remote pods fail the Tekton task, (3) org git/Quay defaults are aligned (or explicitly documented as republish-required). Treat fail-open engine paths, LLM secret wiring, and digest pinning as merge-bar or immediate follow-up depending on how hard you want the first gate.

---

What this PR is

Deliverable	Purpose
7 tasks under pipeline/tasks/konflux/	parse-snapshot → prepare → test → evaluate → analyze-scorecard → store → emit-result
konflux-eval-pipelinerun.yaml	Reference PipelineRun + param contract
Makefile + push-bundles.yaml	Publish Tekton Bundles to Quay
Docs/konflux-integration-guide.md	Consumer guide
config/konflux/secrets-template.yaml	workload / llm / compass templates

Two-cluster model: Konflux orchestrates; optional workload cluster runs agent + eval Pod (EVAL_MODE=local|remote). ABEvalFlow does not deploy the app under test.

---

What looks good (consensus)

1. Ownership split — core tasks never deploy; consumers wrap deploy/cleanup (example repo). Matches Konflux ITS patterns.
2. Local vs remote — solves pipeline-cluster reachability to in-cluster Services via workload SA token.
3. Dual-publish documented — PipelineRun from git, task logic from Quay bundles; Makefile/digests support ops.
4. Graceful optional deps — MinIO / DB / Compass absent → no-op, not hard fail.
5. emit-result → TEST_OUTPUT — scorecard pass/warn/fail → Konflux SUCCESS/WARNING/FAILURE.
6. Parameterized secret names — e.g. WORKLOAD_CREDENTIALS_SECRET, not hardcoded.
7. E2E validation — Lightspeed run cited (9/9 with consumer deploy/cleanup).
8. CI green — ruff/pytest pass (does not exercise Tekton tasks themselves).
9. Limitations acknowledged — TLS skip, log transfer, mutable tags, personal Quay called out in PR/docs.

Complementary to main OpenShift ci-pipeline (PR comments / in-repo tasks), not a replacement.

---

Must fix (blocking)

1. Remote mode ignores prepared submission workspace

prepare clones SUBMISSION_REPO_URL into the shared workspace. Remote evaluate clones only PIPELINE_REPO_URL and uses:

SUBMISSION_PATH="/tmp/abevalflow/submissions/$SUBMISSION_DIR"

Works when the submission lives inside ABEvalFlow (likely Lightspeed POC). Breaks the documented generic case SUBMISSION_REPO_URL ≠ PIPELINE_REPO_URL.

Fix: Clone submission repo in the Pod, or mount/copy workspace submission (PVC / oc cp / tar ConfigMap). Document as hard limitation until fixed is the minimum acceptable interim.

2. Remote Failed pods still exit 0

On PHASE=Failed, script logs and breaks the wait loop, then still scrapes logs and exit 0. A crashed eval Pod can look like a successful Tekton evaluate step.

Fix: After wait loop, if phase ≠ Succeeded, exit 1 (after log capture).

3. Personal Quay + inconsistent git defaults

• Bundles / eval-base: quay.io/rh-ee-ikrispin/... (:0.1, :latest)
• Several task defaults: https://github.com/ikrispin/ABEvalFlow.git (prepare, test, store, analyze); evaluate already uses org URL
• PipelineRun params often override correctly to org — standalone task / omitted param usage does not

Fix before broad adoption: org Quay (ecosystem-appeng or agreed) + all task defaults → RHEcosystemAppEng/ABEvalFlow. Digest-pin reference PipelineRun for production. Acceptable as immediate follow-up only if merge commit documents “republish required.”

4. Evaluate fail-open masks engine failures

Path	Behavior
mcpchecker … || true	Non-zero ignored
harbor run … || true / unused HARBOR_EXIT	Continues
agent-skills-eval … || true	Iteration failures ignored
analyze.py … || echo WARNING	Continues without report
Remote log scrape miss	Soft WARNING

For a Konflux release gate this is dangerous.

Fix: Fail the step when the primary engine command fails, and/or fail if report.json is missing after a non-skipped run.

---

Should fix

#	Issue	Ask
5	LLM key: secrets template has llm-credentials, but reference PipelineRun does not mount it; evaluate defaults sk-dummy	Optional SecretKeyRef like other secrets; pass through PipelineRun
6	parse-snapshot defaults to .components[0]	Require component-name or document multi-component footgun; surface component-image for consumer deploy
7	Snapshot component-image unused in reference pipeline	Document consumer must wire image → deploy; optionally expose as Pipeline result
8	Local evaluate on ose-cli:latest + runtime pip/npm/microdnf	Prefer eval-base-image for local too; pre-bake deps
9	Harbor Konflux path degraded (no scaffold/build; environment.type: local / harbor-bench)	Document ≠ main Harbor A/B pipeline; pin Harbor package source
10	Engine × mode matrix oversold	A2A validated; ASE remote missing; Harbor stub; MCPChecker fail-open — document and fail fast on unsupported pairs
11	No AEH in Konflux evaluate	OK for v1 if documented / roadmap
12	--insecure-skip-tls-verify on remote oc login	CA Secret + --certificate-authority
13	Log-marker report.json transfer	ConfigMap / oc cp / PVC follow-up (P1 robustness)
14	Reference PipelineRun disables security + quality	Comment or provide stricter “marketplace” example
15	Mutable :0.1 / :latest	Digest-pinned reference variant; use Makefile digests
16	Task duplication vs pipeline/tasks/phases/	Long-term share/generate to avoid drift (AEH, MLflow, #65)
17	No tests for Konflux-specific logic	Snapshot jq helpers, emit-result JSON, remote extract
18	Hard-coded LiteLLM / OpenShift console defaults in tasks	Empty or parameterized
19	MCPChecker skips test / limited scorecard path	Call out prominently in guide
20	Merge order with #65	Merge #60 first, then rebase thin #65

---

Minor / nits

• GHA Quay login / stray tkn bundle push --help line
• date -u --iso-8601=seconds is GNU-only (OK on ubi-minimal)
• MCPChecker model defaults assume specific providers
• Dual-publish easy to miss — consider CI that bundle digests match task YAML

---

Merge bar

• Remote eval uses prepared submission or clones SUBMISSION_REPO_URL (or docs hard-limit + fail if mismatch)
• Remote Pod Failed → Task exit 1
• Org git defaults on all tasks; Quay relocation plan or org refs
• Fail closed when engine/analyze yields no report.json (or explicit error artifact + failed task)
• Wire or document llm-credentials in reference PipelineRun
• Guide: engine × mode matrix, Harbor limits, snapshot image ownership, multi-component selection
• Digest-pinning path for production ITS
• Coordinate #65 rebase after #60

---

Design questions

1. Expose component-image as a Pipeline result for consumer deploy?
2. Should documented default for in-cluster agents be EVAL_MODE=local?
3. Longer term: one evaluate dispatcher shared with main phases/evaluate.yaml?

---

Sources

File	Stance	Notable unique findings
htwpl_pr60_konflux_integration.md	Approve w/ minor fixes	Remote submission gap; disabled gates in reference PR; no AEH; task drift vs phases
nyjsj_pr_60_review.md	Request changes	Remote Failed→exit 0; fail-open matrix; ose-cli local; Harbor/local env; LLM secret; #65 order
qmxpt_pr_60_review.md	Approve	Architecture praise; TLS/tags/log transfer as documented non-blockers
vbnkp_pr60_review.md	Approve	Follow-up priority list; MCPChecker model defaults; verified TEST_OUTPUT / timeouts
xkqpd_pr60_review.md	Approve w/ changes	LLM secret inconsistency; .components[0] footgun; MCPChecker gate gaps

Authoritative blockers from deeper reviews: remote submission path, Failed-pod exit code, org defaults/Quay, evaluate fail-open, LLM credentials wiring.

[ikrispin]

Consolidated PR Review: #60 — Generic Konflux integration

PR: #60 Author: ikrispin Branch: ikrispin:konflux-generic → main HEAD: fd2f8072 Size: +2,848 / −0 · 12 files · 2 commits CI: PASS Follow-on: #65 (red-team) stacks on this Consolidated from: htwpl_, nyjsj_, qmxpt_, vbnkp_, xkqpd_ (2026-08-04)

Verdict

Request changes on correctness items; architecture is ready to land afterward.

Reviewer stances ranged from Approve → Approve-with-fixes → Request-changes. Consensus on product direction is strongly positive: reusable Konflux ITS via 7 bundle-published tasks, consumer-owned deploy/cleanup, local/remote eval, optional secrets, and TEST_OUTPUT gating are the right design, and the Lightspeed E2E on stone-prod-p02 is convincing.

Do not merge until at least: (1) remote mode uses the prepared/submission repo correctly, (2) Failed remote pods fail the Tekton task, (3) org git/Quay defaults are aligned (or explicitly documented as republish-required). Treat fail-open engine paths, LLM secret wiring, and digest pinning as merge-bar or immediate follow-up depending on how hard you want the first gate.

What this PR is

Deliverable Purpose
7 tasks under pipeline/tasks/konflux/ parse-snapshot → prepare → test → evaluate → analyze-scorecard → store → emit-result
konflux-eval-pipelinerun.yaml Reference PipelineRun + param contract
Makefile + push-bundles.yaml Publish Tekton Bundles to Quay
Docs/konflux-integration-guide.md Consumer guide
config/konflux/secrets-template.yaml workload / llm / compass templates
Two-cluster model: Konflux orchestrates; optional workload cluster runs agent + eval Pod (EVAL_MODE=local|remote). ABEvalFlow does not deploy the app under test.

What looks good (consensus)

1. Ownership split — core tasks never deploy; consumers wrap deploy/cleanup (example repo). Matches Konflux ITS patterns.
2. Local vs remote — solves pipeline-cluster reachability to in-cluster Services via workload SA token.
3. Dual-publish documented — PipelineRun from git, task logic from Quay bundles; Makefile/digests support ops.
4. Graceful optional deps — MinIO / DB / Compass absent → no-op, not hard fail.
5. emit-result → TEST_OUTPUT — scorecard pass/warn/fail → Konflux SUCCESS/WARNING/FAILURE.
6. Parameterized secret names — e.g. WORKLOAD_CREDENTIALS_SECRET, not hardcoded.
7. E2E validation — Lightspeed run cited (9/9 with consumer deploy/cleanup).
8. CI green — ruff/pytest pass (does not exercise Tekton tasks themselves).
9. Limitations acknowledged — TLS skip, log transfer, mutable tags, personal Quay called out in PR/docs.

Complementary to main OpenShift ci-pipeline (PR comments / in-repo tasks), not a replacement.

Must fix (blocking)

1. Remote mode ignores prepared submission workspace

prepare clones SUBMISSION_REPO_URL into the shared workspace. Remote evaluate clones only PIPELINE_REPO_URL and uses:

SUBMISSION_PATH="/tmp/abevalflow/submissions/$SUBMISSION_DIR"

Works when the submission lives inside ABEvalFlow (likely Lightspeed POC). Breaks the documented generic case SUBMISSION_REPO_URL ≠ PIPELINE_REPO_URL.

Fix: Clone submission repo in the Pod, or mount/copy workspace submission (PVC / oc cp / tar ConfigMap). Document as hard limitation until fixed is the minimum acceptable interim.

2. Remote Failed pods still exit 0

On PHASE=Failed, script logs and breaks the wait loop, then still scrapes logs and exit 0. A crashed eval Pod can look like a successful Tekton evaluate step.

Fix: After wait loop, if phase ≠ Succeeded, exit 1 (after log capture).

3. Personal Quay + inconsistent git defaults

• Bundles / eval-base: quay.io/rh-ee-ikrispin/... (:0.1, :latest)
• Several task defaults: https://github.com/ikrispin/ABEvalFlow.git (prepare, test, store, analyze); evaluate already uses org URL
• PipelineRun params often override correctly to org — standalone task / omitted param usage does not

Fix before broad adoption: org Quay (ecosystem-appeng or agreed) + all task defaults → RHEcosystemAppEng/ABEvalFlow. Digest-pin reference PipelineRun for production. Acceptable as immediate follow-up only if merge commit documents “republish required.”

4. Evaluate fail-open masks engine failures

Path Behavior
mcpchecker … || true Non-zero ignored
harbor run … || true / unused HARBOR_EXIT Continues
agent-skills-eval … || true Iteration failures ignored
analyze.py … || echo WARNING Continues without report
Remote log scrape miss Soft WARNING
For a Konflux release gate this is dangerous.

Fix: Fail the step when the primary engine command fails, and/or fail if report.json is missing after a non-skipped run.

Should fix

Issue Ask

5 LLM key: secrets template has llm-credentials, but reference PipelineRun does not mount it; evaluate defaults sk-dummy Optional SecretKeyRef like other secrets; pass through PipelineRun
6 parse-snapshot defaults to .components[0] Require component-name or document multi-component footgun; surface component-image for consumer deploy
7 Snapshot component-image unused in reference pipeline Document consumer must wire image → deploy; optionally expose as Pipeline result
8 Local evaluate on ose-cli:latest + runtime pip/npm/microdnf Prefer eval-base-image for local too; pre-bake deps
9 Harbor Konflux path degraded (no scaffold/build; environment.type: local / harbor-bench) Document ≠ main Harbor A/B pipeline; pin Harbor package source
10 Engine × mode matrix oversold A2A validated; ASE remote missing; Harbor stub; MCPChecker fail-open — document and fail fast on unsupported pairs
11 No AEH in Konflux evaluate OK for v1 if documented / roadmap
12 --insecure-skip-tls-verify on remote oc login CA Secret + --certificate-authority
13 Log-marker report.json transfer ConfigMap / oc cp / PVC follow-up (P1 robustness)
14 Reference PipelineRun disables security + quality Comment or provide stricter “marketplace” example
15 Mutable :0.1 / :latest Digest-pinned reference variant; use Makefile digests
16 Task duplication vs pipeline/tasks/phases/ Long-term share/generate to avoid drift (AEH, MLflow, #65)
17 No tests for Konflux-specific logic Snapshot jq helpers, emit-result JSON, remote extract
18 Hard-coded LiteLLM / OpenShift console defaults in tasks Empty or parameterized
19 MCPChecker skips test / limited scorecard path Call out prominently in guide
20 Merge order with #65 Merge #60 first, then rebase thin #65

Minor / nits

• GHA Quay login / stray tkn bundle push --help line
• date -u --iso-8601=seconds is GNU-only (OK on ubi-minimal)
• MCPChecker model defaults assume specific providers
• Dual-publish easy to miss — consider CI that bundle digests match task YAML

Merge bar

• Remote eval uses prepared submission or clones SUBMISSION_REPO_URL (or docs hard-limit + fail if mismatch)
• Remote Pod Failed → Task exit 1
• Org git defaults on all tasks; Quay relocation plan or org refs
• Fail closed when engine/analyze yields no report.json (or explicit error artifact + failed task)
• Wire or document llm-credentials in reference PipelineRun
• Guide: engine × mode matrix, Harbor limits, snapshot image ownership, multi-component selection
• Digest-pinning path for production ITS
• Coordinate feat: Add generic red-team adversarial testing stage (Promptfoo) #65 rebase after feat: Generic Konflux integration for ABEvalFlow evaluation pipeline #60

Design questions

1. Expose component-image as a Pipeline result for consumer deploy?
2. Should documented default for in-cluster agents be EVAL_MODE=local?
3. Longer term: one evaluate dispatcher shared with main phases/evaluate.yaml?

Sources

File Stance Notable unique findings
htwpl_pr60_konflux_integration.md Approve w/ minor fixes Remote submission gap; disabled gates in reference PR; no AEH; task drift vs phases
nyjsj_pr_60_review.md Request changes Remote Failed→exit 0; fail-open matrix; ose-cli local; Harbor/local env; LLM secret; #65 order
qmxpt_pr_60_review.md Approve Architecture praise; TLS/tags/log transfer as documented non-blockers
vbnkp_pr60_review.md Approve Follow-up priority list; MCPChecker model defaults; verified TEST_OUTPUT / timeouts
xkqpd_pr60_review.md Approve w/ changes LLM secret inconsistency; .components[0] footgun; MCPChecker gate gaps
Authoritative blockers from deeper reviews: remote submission path, Failed-pod exit code, org defaults/Quay, evaluate fail-open, LLM credentials wiring.

Addressed in eec559e

[GuyZivRH]

PR Review: #60 — Generic Konflux integration for ABEvalFlow

Prefix: mzkpt_
PR: #60
Author: ikrispin
Branch: konflux-generic → main
Reviewed: 2026-08-06
Scope: +2,971 lines, 12 files, 3 commits
Follow-on: #65 (red-team stage) builds on this branch
Ticket: APPENG-5740

Verdict

Approve with minor fixes — strong, production-minded Konflux integration. Fix commit eec559e addressed the major correctness issues (remote submission clone, fail-closed eval, upstream repo defaults). CI green; E2E validated on stone-prod-p02. Remaining items are org Quay relocation, TLS hardening, and long-term drift/bundle pinning — not blockers for merging the framework.

Summary

Adds ABEvalFlow as a reusable Konflux IntegrationTestScenario — 7 core Tekton tasks published as bundles:

parse-snapshot → prepare → test → evaluate → analyze-scorecard → store → emit-result

Consumers wrap with deploy/cleanup (example: abevalflow-konflux-example). Validated: PipelineRun lightspeed-abevalflow-eval-qhx2p — 9/9 tasks on Konflux + cn-ai-lab workload cluster.

Deliverable	Purpose
pipeline/tasks/konflux/*.yaml	7 generic eval tasks
pipeline/integration/konflux-eval-pipelinerun.yaml	Reference PipelineRun + parameter contract
pipeline/integration/Makefile + push-bundles.yaml	Publish Tekton Bundles to Quay
Docs/konflux-integration-guide.md	Integration guide (~340 lines)
config/konflux/secrets-template.yaml	Workload cluster + LLM secrets

---

Commits

Commit	Summary
e8284c7	Initial Konflux ITS (later refactored)
fd2f807	Generic 7-task framework; deploy/cleanup moved to example repo
eec559e	Review fixes: remote submission clone, fail-closed, upstream defaults, docs

---

CI status

Check	Status
test	✅ PASS (run 30987898881)
Note: CI validates Python/ruff in this repo, not Konflux task execution — E2E proof is the documented Konflux run.

---

Prior review items — status

Item	Status
Remote mode missing separate submission repo	✅ Fixed — clones SUBMISSION_REPO_URL in eval Pod
Silent pass on failed/missing report.json	✅ Fixed — fail closed (remote + local)
Task defaults pointed at author fork	✅ Fixed — RHEcosystemAppEng/ABEvalFlow
Reference pipeline hides disabled security/quality	✅ Fixed — comments + guide note
Hardcoded LiteLLM / console URL defaults	✅ Fixed — removed from analyze-scorecard
llm-credentials Secret wiring	✅ Fixed — optional SecretKeyRef in evaluate
Engine × mode matrix undocumented	✅ Fixed — in task description + guide
Multi-component snapshot footgun	✅ Documented
Personal Quay org for bundles/images	❌ Still quay.io/rh-ee-ikrispin/ (acknowledged in PR)
--insecure-skip-tls-verify on oc login	❌ Still present (TODO in code)
Log-based report.json extraction	❌ Still markers in pod logs
Mutable :0.1 bundle tags	❌ Deferred
Task duplication vs pipeline/tasks/phases/	❌ Inherent maintenance cost
No AEH engine in Konflux evaluate	❌ Out of scope for v1

---

What's good

1. Correct architectural split

Core tasks never deploy the application. Generic eval framework + consumer-specific deploy/cleanup is the right Konflux ITS pattern.

2. Local vs remote eval modes

Solves real cross-cluster networking: Konflux orchestrates on stone-prod-p02; eval Pod runs on cn-ai-lab where LiteLLM and agent Services are reachable.

3. Fail-closed hardening (eec559e)

Remote Failed pods exit 1; missing submission path, Harbor with no results, missing post-analyze report.json, and log extraction failures now fail the pipeline instead of emitting synthetic pass/fail JSON.

4. Honest engine × mode matrix

Documents what is tested (a2a + remote), supported-but-untested, limited (harbor + local), and unsupported (harbor + remote).

5. Konflux-native release gating

emit-result.yaml maps scorecard recommendation + certification to TEST_OUTPUT (SUCCESS/WARNING/FAILURE).

6. Graceful optional infra

Store/test/analyze no-op cleanly without MinIO, Postgres, or Compass — appropriate for Konflux tenants without full ABEvalFlow stack.

7. Parameterized secrets

WORKLOAD_CREDENTIALS_SECRET, optional llm-credentials, mode-conditional docs in secrets template.

8. Dual-publish model documented

Git PipelineRun + Quay bundles — update requires both make bundles and git push. Clear in PipelineRun header.

Should fix (non-blocking)

1. Relocate Quay org before production default

Bundles and eval-base-image remain under quay.io/rh-ee-ikrispin/. PR acknowledges relocation to team org — track as pre-production task or follow-up PR immediately after merge.

2. Pin bundle digests

Reference PipelineRun uses mutable :0.1 tags. Makefile make digests exists — document pinning pattern for consumers (@sha256:...).

3. TLS verification for cross-cluster oc login

Still --insecure-skip-tls-verify=true. Inject workload cluster CA via Secret before enterprise rollout.

4. ConfigMap-based result transfer

Log markers (REPORT_JSON_START/END) work but are fragile for large reports. PR tracks migration — keep on roadmap.

5. Konflux task drift from main pipeline

~3k lines parallel to pipeline/tasks/phases/. As main gains AEH, MLflow, red-team (#65), Konflux tasks will lag unless there's a sync strategy.

6. No AEH engine

Main ci-pipeline.yaml supports aeh; Konflux evaluate does not. Document in guide roadmap if AEH Konflux support is planned.

7. Reference pipeline security/quality disabled by default

Intentional for minimal ITS — guide now explains how to enable stricter gates. Consider a second konflux-eval-strict-pipelinerun.yaml example later.

Merge relationship with #65

#65 adds 8th bundle (red-team) + Promptfoo stage on top of this branch (+555 lines). Merge #60 first, then #65 (or rebase stack).

Recommended next steps

1. Merge #60 — framework is ready for Konflux consumers
2. Follow-up: relocate bundles to quay.io/ecosystem-appeng/, pin digests
3. Merge/rebase #65 for optional adversarial testing stage
4. Add Konflux eval to Docs/infrastructure_ops.md cross-link
5. Long-term: reduce duplication with phase tasks or shared fragments

---

Review generated from branch pr-60-review (through eec559e), CI run 30987898881.