Skip to content

Reconcile Google model-compatibility fixtures and scenarios #1153

Closed

Description

@mikehostetler

Summary

The Google model-compatibility replay data is not aligned with the current provider request format and model catalog.

The last recorded run from PR #1127 selected 44 models. It reported 39 active failures and 5 retired models. The failures included missing fixtures, recorded-request mismatches, and unsupported options for newer models.

One confirmed baseline mismatch is:

  • fixture: google/gemini_2_5_flash/usage.json
  • field: canonical_json.generationConfig.maxOutputTokens
  • recorded value: 500
  • current value: 10

The same mismatch was reproduced on the upstream Google provider before the text thought-signature change in PR #1127. This compatibility work is therefore separate from that fix.

Work

  • run the complete Google compatibility matrix against current main
  • classify missing fixtures, request drift, retired models, and unsupported scenarios
  • update fixtures or scenario expectations where current behavior is correct
  • add explicit exclusions only when the model or scenario is not supported
  • make the active Google compatibility matrix pass, or document each remaining expected exclusion

Reproduction

mix mc 'google:*' --max-concurrency 4
mix mc google:gemini-2.5-flash --scenario usage --debug --max-concurrency 1

Related: #1127

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions