compaction.go

package agent

import (
	"context"
	"fmt"
	"strings"

	"github.com/dmmdea/offload-harness/internal/contextbudget"
	"github.com/dmmdea/offload-harness/internal/gcf"
)

// Transcript compaction. The agent loop resends the FULL running transcript on
// every step, but the local models are served with a small window (the shipped
// templates use --ctx-size 8192) and the loop reserves --max-tokens for the
// completion, so the effective INPUT budget is only a few thousand tokens. A
// few tool results overflow it, the server 400s, and the run aborts. This file
// keeps the transcript inside budget so long tasks finish, WITHOUT breaking
// assistant<->tool pairing (an orphaned tool result or tool call is a wire
// error) and WITHOUT touching system or the objective (the two things the model
// must never lose sight of).

// bytesPerToken is the crude bytes-per-token heuristic. ~4 bytes/token is the
// long-standing rule of thumb for English-ish text; the local Gemma tokenizer
// is close enough for a BUDGET (we only need to know "roughly how full are we",
// not an exact count). Deliberately conservative: over-estimating slightly just
// means we compact a touch early, which is the safe direction.
const bytesPerToken = 4

// perMsgOverhead approximates the wire framing each message costs beyond its
// content: role tag, delimiters, and (for assistant turns) the tool-call
// envelope. A flat small constant is enough for a budget estimate.
const perMsgOverhead = 4

// estimateTokens returns a cheap, approximate token count for a transcript:
// sum over messages of len(content)/4 + a small per-message overhead, plus the
// tool-call argument bytes (they go on the wire too). Approximate by design —
// see bytesPerToken. Never returns a negative number.
func estimateTokens(msgs []Msg) int {
	total := 0
	for _, m := range msgs {
		total += len(m.Content) / bytesPerToken
		total += perMsgOverhead
		for _, c := range m.ToolCalls {
			total += (len(c.Name) + len(c.Args)) / bytesPerToken
		}
	}
	return total
}

// compact reduces msgs to fit within budget (in estimated tokens) using the
// least-destructive edit that works, in this fixed order:
//
//  1. If already under budget, return msgs UNCHANGED (byte-for-byte) — this
//     preserves prefix stability so the server's KV cache is not invalidated on
//     the happy path.
//  2. ALWAYS keep, full and in place: the contiguous protected preamble
//     [0, protectedPrefix) — the caller passes its real length (system +
//     profile exemplars + recall + AGENT.md + objective), so the objective is
//     never guessed at and never dropped even when exemplars precede it — plus
//     the most recent keepRecent turns.
//     2a. (opts.GCF only, LOSSLESS) re-encode OLDER tool bodies that are
//     eligible JSON arrays into GCF columnar form (internal/gcf) — states the
//     repeated keys once, loses nothing; always tried before any lossy rung.
//     2a-i. (opts.LogCollapse only, lossy but gentler than 2b) fold each run of
//     near-identical lines in the output of an OLDER build or test command
//     into its first line, a count and its last line (see rung_logs.go),
//     keeping every signal, diagnostic and summary line, oldest-first and
//     stopping at the first fit. A search listing, a file read and a linter's
//     output are not touched (the producer gate reads the call).
//     2a-ii. (opts.SearchResults only, default off) THIN older search listings to the
//     lines that matter for the objective (BM25, a per-file cap, an error boost, a
//     counted footer; see rung_search.go), before 2b so the skeleton never sees one.
//     The two rungs never claim the same body (each reads the call that produced it:
//     a build or test invocation is 2a-i, a search is 2a-ii); 2a-i runs first because it
//     keeps every distinct line and 2a-ii drops most of a listing, so with both on the
//     gentler edit gets the first chance to fit the budget.
//     2b. (opts.Skeleton only) SKELETONIZE the bodies of OLDER tool-role messages
//     oldest-first until under budget: head/tail windows + buried signal lines
//     kept, the rest elided to counted run markers (see skeleton.go). Strictly
//     less destructive than step 3 — the error a task later needs usually
//     survives — and a skeleton can still fall through to a bare marker below.
//  3. Elide the bodies of OLDER tool-role messages to a compact marker
//     (preserving Role + ToolCallID so pairing is intact), oldest-first, until
//     under budget. Eliding a body is preferred over dropping a message.
//  4. If still over budget, drop WHOLE messages. With a Tokenizer configured
//     (opts.Tok) this is the token-exact middle cut (cutmiddle.go, TO-4):
//     the served model's real token counts decide what fits, whole
//     assistant+tool units are dropped from the MIDDLE, and the head/tail
//     windows survive. Without one (or when the tokenizer fails open) the
//     legacy estimate-driven rung drops whole OLDER turns oldest-first. In
//     both paths a turn is an assistant message that has ToolCalls PLUS all
//     its matching tool results, dropped as a unit so assistant<->tool
//     pairing is never broken, and the protected preamble (incl. the
//     objective) is never dropped.
//
// keepRecent counts assistant/tool turns to keep verbatim from the end; a
// non-positive value is treated as 0 (nothing pinned as "recent", though the
// protected prefix is still always kept).
//
// protectedPrefix is the length of the contiguous leading preamble the caller
// guarantees must be kept verbatim: system + profile exemplars + recall +
// AGENT.md + objective — everything before the first model turn. The loop passes
// its real preamble length so the objective is never guessed at (it is NOT
// necessarily the first user message once profile exemplars precede it). It is
// clamped to [0, len(msgs)]. The preamble is bounded (recall/AGENT.md are capped
// and exemplars are a small fixed set); if it ALONE exceeds budget, compaction
// cannot shrink it — the transcript stays over budget and the run errors honestly
// on the next Chat rather than silently dropping the objective to fit.
// compactOpts selects the optional gentler rungs of the ladder. Zero value =
// the original two-rung ladder (markers, drops) — the pinned default path.
type compactOpts struct {
	GCF      bool // lossless: re-encode eligible JSON-array tool bodies (internal/gcf)
	Skeleton bool // lossy-structural: reduce older tool bodies to signal skeletons
	// LogCollapse (default off) folds runs of near-identical lines in OLDER build
	// and test output into the first line, a count and the last line, keeping
	// every error/warning/diagnostic/summary line (rung_logs.go). It runs after
	// the lossless GCF rung and before the search rung and skeleton.
	LogCollapse bool
	// SearchResults (default OFF) thins older search listings (rung_search.go): BM25
	// against the objective, a per-file cap, an error boost, a counted footer. It
	// runs after the log rung and before skeleton.
	SearchResults bool
	// Pinned marks tool_call_ids whose results the model DEMONSTRABLY re-requested
	// (the circuit breaker refused an exact-repeat call for them) — the H8 ramp:
	// content the model re-reads stops being lossily compacted. A pinned body is
	// exempt from dedupe/skeleton/elide, and its unit from drop; the LOSSLESS GCF
	// rung still applies (nothing is lost). emergencyShrink stays pin-blind — at
	// that point the alternative is a dead run.
	Pinned map[string]bool
	// Tok, when non-nil, REPLACES the estimate-driven whole-turn-drop rung
	// (step 4) with the token-exact middle cut (cutmiddle.go, TO-4): the drop
	// decision is then made on the served model's REAL token counts against
	// RealBudget (the real-token allowance, i.e. the loop's uncorrected
	// inputBudget), never on the chars/4 estimate. The legacy rung below runs
	// only when Tok is nil or the tokenizer fails open — one truncation
	// mechanism at a time, by explicit gate.
	Tok        Tokenizer
	RealBudget int
}

// fitVerdict is what compact KNOWS about the returned transcript's fit — and,
// critically, which yardstick produced that knowledge.
type fitVerdict int

const (
	// fitUnknown: only the estimate ladder ran; the caller judges fit by
	// estimateTokens, exactly as before the token-exact rung existed.
	fitUnknown fitVerdict = iota
	// fitReal: the token-exact rung MEASURED the returned transcript within
	// the real budget. The estimate must not overrule it — a pessimistic
	// estimate would otherwise report a verified-fitting request as exhausted
	// on every remaining step.
	fitReal
	// overReal: the token-exact rung measured the transcript OVER the real
	// budget even after cutting (forced keeps: preamble/pinned/signal/tail).
	// The caller must count it exhausted — the under-counting estimate would
	// otherwise hide exactly the overflow the telemetry exists to report.
	overReal
)

// compact is the estimate-compatible wrapper over compactWithVerdict for
// callers that don't consume the fit verdict (replay, tests).
func compact(ctx context.Context, msgs []Msg, budget int, keepRecent int, protectedPrefix int, opts compactOpts) []Msg {
	out, _ := compactWithVerdict(ctx, msgs, budget, keepRecent, protectedPrefix, opts)
	return out
}

func compactWithVerdict(ctx context.Context, msgs []Msg, budget int, keepRecent int, protectedPrefix int, opts compactOpts) ([]Msg, fitVerdict) {
	if estimateTokens(msgs) <= budget {
		return msgs, fitUnknown // happy path: untouched, KV cache preserved.
	}
	if keepRecent < 0 {
		keepRecent = 0
	}
	if protectedPrefix < 0 {
		protectedPrefix = 0
	}
	if protectedPrefix > len(msgs) {
		protectedPrefix = len(msgs)
	}

	// Work on a copy: never mutate the caller's slice or its backing array.
	out := make([]Msg, len(msgs))
	copy(out, msgs)

	protectedEnd := protectedPrefix      // [0,protectedEnd) is always kept.
	recentStart := len(out) - keepRecent // [recentStart,len) is always kept.
	if recentStart < protectedEnd {
		recentStart = protectedEnd
	}

	// Resolve pins TRANSITIVELY through dedupe references first: a pinned body
	// that is already a dedupe marker protects only the reference — the bytes
	// the model actually wanted live at the referenced (possibly chained) call,
	// which must be pinned too or later pressure destroys exactly the content
	// H8 exists to keep.
	pinned := opts.Pinned
	if len(pinned) > 0 {
		pinned = make(map[string]bool, len(opts.Pinned))
		for id := range opts.Pinned {
			pinned[id] = true
		}
		for changed := true; changed; {
			changed = false
			for _, m := range out {
				if m.Role == "tool" && pinned[m.ToolCallID] && isDeduped(m.Content) {
					if ref, ok := dedupeRef(m.Content); ok && !pinned[ref] {
						pinned[ref] = true
						changed = true
					}
				}
			}
		}
	}

	// Step 2-pre (always on — the cheapest rung, reference-preserving): collapse
	// OLDER tool bodies byte-identical to a LATER tool result. A model that
	// fetched the same content twice keeps its NEWEST copy authoritative; the
	// older copy becomes a reference marker naming the later call id (pairing
	// intact, information reachable via the reference). Runs before GCF so
	// per-copy re-encodings can never make duplicates unequal first. Honest
	// limits, disclosed here rather than papered over: if the later copy is
	// itself compacted afterwards the reference degrades to pointing at a
	// marker; the drop rung can remove the referenced unit entirely; and
	// repeated passes over fresh duplicates can produce short reference
	// chains. All are still no worse than the bare marker the older copy
	// would otherwise have become, and pin resolution above follows chains.
	latest := map[string]int{} // body → index of its LAST occurrence
	for i := len(out) - 1; i >= protectedEnd; i-- {
		if out[i].Role != "tool" || len(out[i].Content) < dedupeMinChars || IsCompactionArtifact(out[i].Content) {
			continue
		}
		if _, seen := latest[out[i].Content]; !seen {
			latest[out[i].Content] = i
		}
	}
	for i := protectedEnd; i < recentStart && estimateTokens(out) > budget; i++ {
		if out[i].Role != "tool" || pinned[out[i].ToolCallID] || len(out[i].Content) < dedupeMinChars || IsCompactionArtifact(out[i].Content) {
			continue
		}
		if j, ok := latest[out[i].Content]; ok && j > i {
			// Guard against GROWTH: a long call id can make the marker larger
			// than a smallish body — collapsing must always shrink.
			if m := dedupeMarker(out[j].ToolCallID, logsMarkerSize(out[i].Content, opts.LogCollapse)); len(m) < len(out[i].Content) {
				out[i].Content = m
			}
		}
	}
	if estimateTokens(out) <= budget {
		return out, fitUnknown
	}

	// Step 2a (flag-gated, LOSSLESS): re-encode older tool bodies that are
	// eligible JSON arrays into GCF columnar form — zero information loss, so
	// it always runs before any lossy rung. Deterministic and idempotent
	// (gcf.IsCompacted stops re-encoding).
	if opts.GCF {
		for i := protectedEnd; i < recentStart && estimateTokens(out) > budget; i++ {
			if out[i].Role == "tool" && !isElided(out[i].Content) && !isSkeletonized(out[i].Content) && !gcf.IsCompacted(out[i].Content) {
				if c, ok := gcf.Compact(out[i].Content); ok {
					out[i].Content = c
				}
			}
		}
		if estimateTokens(out) <= budget {
			return out, fitUnknown
		}
	}

	// Step 2a-i (flag-gated, lossy but gentler than skeleton): fold runs of
	// near-identical lines in OLDER build and test output, oldest-first, stopping
	// at the first fit (rung_logs.go; the producer gate reads the call). Same [protectedEnd, recentStart) window and pin
	// check as every rung here, so the preamble, the recent turns and pairing stay.
	// Before the search rung: it keeps every distinct line, the search rung drops most
	// of a listing, so the gentler edit gets the first chance to fit the budget.
	if opts.LogCollapse {
		logsRung(out, protectedEnd, recentStart, budget, pinned)
		if estimateTokens(out) <= budget {
			return out, fitUnknown
		}
	}

	// Step 2a-ii (flag-gated, default OFF): thin older search listings by relevance
	// to the objective (rung_search.go). Before the generic skeleton on purpose: a
	// skeletonized listing is no longer one, and this rung would never see it.
	if opts.SearchResults {
		searchRung(out, protectedEnd, recentStart, budget, pinned)
		if estimateTokens(out) <= budget {
			return out, fitUnknown
		}
	}

	// Step 2b (flag-gated): skeletonize OLDER tool bodies oldest-first — the
	// least-destructive body edit. Deterministic, so a transcript compacted
	// twice lands on identical bytes (isSkeletonized stops re-pruning).
	if opts.Skeleton {
		for i := protectedEnd; i < recentStart && estimateTokens(out) > budget; i++ {
			if out[i].Role == "tool" && !pinned[out[i].ToolCallID] {
				if sk, ok := skeletonizeAs(out[i].Content, logsMarkerSize(out[i].Content, opts.LogCollapse)); ok {
					out[i].Content = sk
				}
			}
		}
		if estimateTokens(out) <= budget {
			return out, fitUnknown
		}
	}

	// Step 3: elide OLDER tool-result bodies to markers, oldest-first. A
	// skeleton falls through to a bare marker here under harder pressure; its
	// marker reports the ORIGINAL body size (parsed from the skeleton's own
	// prefix), not the skeleton's — the marker discloses what was lost. With the
	// log rung on, that prefix is believed only when the skeleton rung could have
	// written it (logsSkeletonSize, reached through logsMarkerSize).
	// FORCE_PRESERVE guard: a body whose remaining content still carries signal
	// lines (errors/failures/warnings — the class every OmniRoute intensity
	// preserves) keeps up to a bounded residue of those lines under the marker
	// instead of losing them to a bare marker. Deduped references are skipped —
	// already tiny, and eliding one would sever the reference for no gain.
	for i := protectedEnd; i < recentStart && estimateTokens(out) > budget; i++ {
		if out[i].Role == "tool" && !isElided(out[i].Content) && !isDeduped(out[i].Content) && !pinned[out[i].ToolCallID] {
			n := logsMarkerSize(out[i].Content, opts.LogCollapse)
			if orig, ok := skeletonOriginalSize(out[i].Content); ok && !opts.LogCollapse {
				n = orig // rung on: logsMarkerSize already weighed the skeleton's header
			}
			out[i].Content = elideWithSignal(out[i].Content, n)
		}
	}
	if estimateTokens(out) <= budget {
		return out, fitUnknown
	}

	// Step 4 (token-exact, TO-4): with a Tokenizer present the whole-message
	// middle cut REPLACES the legacy estimate-driven drop rung — real token
	// counts decide what is dropped, whole messages only, head and tail always
	// survive. ok=false (endpoint has no /tokenize, or a mid-run outage) falls
	// open to the legacy rung below; the loop's tokenizer wrapper is sticky, so
	// a run pays that failed probe once.
	if opts.Tok != nil {
		if cut, fits, ok := cutMiddleTurns(ctx, opts.Tok, out, opts.RealBudget, protectedEnd, keepRecent, pinned); ok {
			if fits {
				return cut, fitReal
			}
			return cut, overReal
		}
	}

	// Step 4 (legacy fallback): drop whole OLDER turns oldest-first, keeping
	// assistant<->tool pairs together. Build a keep-mask so removal is a single
	// pass with stable order and no pairing breakage.
	keep := make([]bool, len(out))
	for i := range keep {
		keep[i] = true
	}
	for i := protectedEnd; i < recentStart && estimateTokens(masked(out, keep)) > budget; i++ {
		if !keep[i] {
			continue
		}
		if out[i].Role == "assistant" && len(out[i].ToolCalls) > 0 {
			// Drop this assistant turn AND its matching tool results as a unit —
			// UNLESS a result is pinned (the model re-requested it) or its
			// remaining content still carries signal lines (the elide rung's
			// FORCE_PRESERVE residue; bare markers/skeleton prefixes contain no
			// signal vocabulary, so this test is exactly "residue present").
			// Skipping a unit can leave the ladder exhausted over budget — the
			// caller surfaces that honestly (fit telemetry) rather than this
			// rung force-dropping preserved signal.
			ids := map[string]bool{}
			for _, c := range out[i].ToolCalls {
				ids[c.ID] = true
			}
			droppable := true
			for j := i + 1; j < recentStart; j++ {
				if out[j].Role == "tool" && ids[out[j].ToolCallID] {
					if pinned[out[j].ToolCallID] || signalLine.MatchString(out[j].Content) {
						droppable = false
						break
					}
				}
			}
			if !droppable {
				continue
			}
			keep[i] = false
			for j := i + 1; j < recentStart; j++ {
				if out[j].Role == "tool" && ids[out[j].ToolCallID] {
					keep[j] = false
				}
			}
		} else if out[i].Role != "tool" {
			// A bare assistant/other older message with no tool calls: safe to
			// drop on its own (no pairing to preserve). Never a tool result here
			// — those only exist as part of a pair handled above.
			keep[i] = false
		}
	}
	return masked(out, keep), fitUnknown
}

// masked returns the subslice of msgs whose keep flag is true, in order.
func masked(msgs []Msg, keep []bool) []Msg {
	out := make([]Msg, 0, len(msgs))
	for i, m := range msgs {
		if keep[i] {
			out = append(out, m)
		}
	}
	return out
}

// elisionMarker is the compact stand-in for an elided tool-result body. It
// records the original size so a reader (human or model) sees that content was
// dropped and how much.
func elisionMarker(origLen int) string {
	return fmt.Sprintf("[earlier result elided to fit context — %d chars]", origLen)
}

// dedupeMinChars: below this a duplicate body is not worth collapsing — the
// reference marker would be a meaningful fraction of the "savings".
const dedupeMinChars = 64

// dedupePrefix opens every dedupe reference (idempotence marker for isDeduped).
const dedupePrefix = "[identical to the later result for call "

// dedupeMarker replaces an older duplicate tool body with a reference to the
// call whose (newer) result carries the same bytes.
func dedupeMarker(callID string, origLen int) string {
	return fmt.Sprintf("%s%s — %d chars]", dedupePrefix, callID, origLen)
}

// isDeduped reports whether a body is already a dedupe reference. Pure-ASCII
// prefix — rune-safe by construction, like isElided.
func isDeduped(content string) bool {
	return len(content) >= len(dedupePrefix) && content[:len(dedupePrefix)] == dedupePrefix
}

// dedupeRef parses the referenced call id back out of a dedupe marker, so pin
// resolution can follow the reference to the bytes it stands for.
func dedupeRef(content string) (string, bool) {
	if !isDeduped(content) {
		return "", false
	}
	rest := content[len(dedupePrefix):]
	end := strings.Index(rest, " — ")
	if end <= 0 {
		return "", false
	}
	return rest[:end], true
}

// elideSignalMaxLines / elideSignalMaxChars bound the FORCE_PRESERVE residue an
// elision may keep — enough for the error that a later step needs, never a
// skeleton-sized survivor.
const elideSignalMaxLines = 5
const elideSignalMaxChars = 400

// elideWithSignal elides a body to the standard marker, keeping a bounded
// residue of its signal lines (errors/failures/warnings) UNDER the marker when
// any exist — the FORCE_PRESERVE guard: an error line survives every rung
// short of the run's own death. The marker line comes first, so isElided still
// recognizes the result and no rung reprocesses it.
func elideWithSignal(content string, origLen int) string {
	marker := elisionMarker(origLen)
	var kept []string
	total := 0 // total JOINED length of the residue, newline separators included
	for _, ln := range strings.Split(content, "\n") {
		if len(kept) >= elideSignalMaxLines {
			break
		}
		if !signalLine.MatchString(ln) {
			continue
		}
		sep := 0
		if len(kept) > 0 {
			sep = 1 // the joining newline counts against the bound too
		}
		room := elideSignalMaxChars - total - sep
		if room <= 0 {
			break
		}
		if len(ln) > room {
			ln = clip(ln, room) // rune-safe: never split a multibyte char
		}
		if ln == "" {
			break
		}
		kept = append(kept, ln)
		total += len(ln) + sep
	}
	if len(kept) == 0 {
		return marker
	}
	return marker + "\n" + strings.Join(kept, "\n")
}

// isElided reports whether a tool body has already been replaced by a marker,
// so a re-compaction does not double-elide (which would misreport the size).
// The marker line itself is pure ASCII; the two places compaction DOES
// truncate content — the FORCE_PRESERVE signal residue (elideWithSignal) and
// emergencyShrink's final trim — both go through rune-safe helpers (clip /
// contextbudget.Trim), so no multibyte rune is ever split.
func isElided(content string) bool {
	const prefix = "[earlier result elided to fit context —"
	return len(content) >= len(prefix) && content[:len(prefix)] == prefix
}

// emergencyShrink is the reactive-overflow LAST RESORT, used only after a
// server overflow rejection when compact() could not get under budget —
// typically because the oversized tool body sits inside keepRecent, where every
// ladder rung is forbidden. Observed live (flip-decision report 2026-07-24,
// finding F3): a transcript of [system, objective, assistant, tool(HUGE)]
// makes the harder-compaction retry a byte-for-byte no-op, so the retry
// re-sends the same overflow and the run dies. At that point recency
// protection has nothing left to protect — the alternative is an abort — so
// tool BODIES (never the preamble, never whole turns) are shrunk: OLDER bodies
// skeleton-first then elision markers, oldest-first; the NEWEST body is spared
// until last and then trimmed head/tail to the remaining room (a bare marker
// only if even the trim cannot fit — a degraded result beats a request the
// server will reject again). Deterministic; returns msgs unchanged when
// already within budget.
func emergencyShrink(msgs []Msg, budget, protectedPrefix int) []Msg {
	return emergencyShrinkWith(msgs, budget, protectedPrefix, compactOpts{})
}

// emergencyShrinkWith is emergencyShrink for a loop whose ladder has optional
// rungs on. Only LogCollapse matters here: it makes the markers name the size a
// folded body had before the fold (logsMarkerSize); everything else is
// emergencyShrink as documented above.
func emergencyShrinkWith(msgs []Msg, budget, protectedPrefix int, opts compactOpts) []Msg {
	if estimateTokens(msgs) <= budget {
		return msgs
	}
	out := make([]Msg, len(msgs))
	copy(out, msgs)
	if protectedPrefix < 0 {
		protectedPrefix = 0
	}
	// The NEWEST tool body — the model's just-fetched result, the content the
	// next completion most needs — is spared by passes 1-2 and degraded
	// GRACEFULLY (head/tail trim) in pass 3, never jumped straight to a marker.
	lastTool := -1
	for i := len(out) - 1; i >= protectedPrefix; i-- {
		if out[i].Role == "tool" {
			lastTool = i
			break
		}
	}
	// Pass 1: skeletonize every eligible OLDER tool body, oldest-first (only
	// useful when the skeleton rung wasn't already applied to that body).
	for i := protectedPrefix; i < len(out) && estimateTokens(out) > budget; i++ {
		if out[i].Role != "tool" || i == lastTool || isElided(out[i].Content) {
			continue
		}
		if sk, ok := skeletonizeAs(out[i].Content, logsMarkerSize(out[i].Content, opts.LogCollapse)); ok && len(sk) < len(out[i].Content) {
			out[i].Content = sk
		}
	}
	// Pass 2: elide OLDER tool bodies to bare markers, oldest-first — including
	// the keepRecent tail compact() must never touch, but NOT the newest body
	// (eliding it here would make pass 3 unreachable and destroy the one result
	// the model is about to work on). A body that is ALREADY an elision marker
	// carrying FORCE_PRESERVE signal residue is stripped to its bare first
	// line: the ladder proper never reclaims residue, so this last resort must
	// (the pin-blind rationale applies identically — the alternative is a dead
	// run), or a residue-heavy transcript could never be made to fit at all.
	for i := protectedPrefix; i < len(out) && estimateTokens(out) > budget; i++ {
		if out[i].Role != "tool" || i == lastTool {
			continue
		}
		if isElided(out[i].Content) {
			if nl := strings.IndexByte(out[i].Content, '\n'); nl >= 0 {
				out[i].Content = out[i].Content[:nl] // strip signal residue
			}
			continue
		}
		if orig, ok := skeletonOriginalSize(out[i].Content); ok && !opts.LogCollapse {
			out[i].Content = elisionMarker(orig)
		} else {
			out[i].Content = elisionMarker(logsMarkerSize(out[i].Content, opts.LogCollapse))
		}
	}
	// Pass 3: the newest tool body is what still keeps the estimate over
	// budget — trim IT to the room the rest of the transcript leaves
	// (head/tail, disclosure marker inside). If even the floored trim cannot
	// fit, fall through to a bare marker: a destroyed result beats a request
	// the server will reject again. Non-tool content (system, objective,
	// assistant text) is never touched — if THAT alone exceeds the budget the
	// transcript is returned honestly over budget, same as the preamble
	// contract.
	if lastTool >= 0 && estimateTokens(out) > budget {
		size := len(out[lastTool].Content)
		reported := logsMarkerSize(out[lastTool].Content, opts.LogCollapse)
		rest := estimateTokens(out) - size/bytesPerToken
		roomChars := (budget - rest) * bytesPerToken
		if roomChars < 256 {
			roomChars = 256
		}
		out[lastTool].Content, _ = contextbudget.Trim(out[lastTool].Content, roomChars)
		if estimateTokens(out) > budget {
			out[lastTool].Content = elisionMarker(reported)
		}
	}
	return out
}

// ---------------------------------------------------------------------------
// Replay seam (compaction eval harness). The eval harness (internal/compeval)
// replays REAL transcript slices through the EXACT production ladder — never a
// reimplementation — so measured ratios/retention are the shipping behavior.
// Exported wrappers only; the ladder itself stays package-private.
// ---------------------------------------------------------------------------

// ReplayOpts mirrors compactOpts for external replay callers.
type ReplayOpts struct {
	GCF      bool
	Skeleton bool
	// SearchResults and LogCollapse switch on the two optional rungs that are
	// default off in production; the compaction eval sets them to measure each.
	SearchResults bool
	LogCollapse   bool
}

// CompactReplay runs the production compaction ladder over msgs at budget with
// the given optional rungs. keepRecent/protectedPrefix follow compact()'s
// semantics (see above). Replay is estimate-ladder only (no Tokenizer): the
// eval harness measures offline transcripts with no serving endpoint, so it
// exercises the legacy drop rung — the same path a tokenizer-less production
// run takes.
func CompactReplay(msgs []Msg, budget, keepRecent, protectedPrefix int, opts ReplayOpts) []Msg {
	return compact(context.Background(), msgs, budget, keepRecent, protectedPrefix, compactOpts{GCF: opts.GCF, Skeleton: opts.Skeleton, SearchResults: opts.SearchResults, LogCollapse: opts.LogCollapse})
}

// EstimateTokens exposes the ladder's own token estimator so replay callers
// measure with the same yardstick the ladder budgets by.
func EstimateTokens(msgs []Msg) int { return estimateTokens(msgs) }

// DefaultKeepRecent exposes the production keep-recent default so replay
// callers (the compaction eval's trace harvest) mirror the live loop's
// pressure instead of silently replaying under a harsher setting.
const DefaultKeepRecent = defaultKeepRecent

// IsCompactionArtifact reports whether a tool body is a PRODUCT of this
// ladder — an elision marker (with or without signal residue), a skeleton, or
// a dedupe reference — rather than raw content. The trace harvest uses it to
// refuse already-compacted transcripts: replaying compaction-of-compacted text
// would measure the ladder against its own output and bias every ratio toward 1.
func IsCompactionArtifact(content string) bool {
	return isElided(content) || strings.HasPrefix(content, skeletonPrefix) || isDeduped(content)
}