Fix task spec vs gold mismatches in 3 tasks (AE004, AE006, F1002)#151
Open
kiwi0401 wants to merge 1 commit into
Open
Fix task spec vs gold mismatches in 3 tasks (AE004, AE006, F1002)#151kiwi0401 wants to merge 1 commit into
kiwi0401 wants to merge 1 commit into
Conversation
Three tasks ship a gold answer key that the task spec cannot produce, so a clean-room agent reasoning only from the task inputs cannot pass them. analytics_engineering004: The gold omits the attachments column and aliases the duplicate product_id as ipd, but the prompt only asks for "product details" and never mentions either. attachments is 100% NULL in the source, but so are description, purchase_order_id, and customer_order_id which the gold keeps, so null-ness is not a usable signal. Added both as prompt clarifications. analytics_engineering006: Same attachments omission and ipd alias on obt_product_inventory. In addition, fact_inventory casts transaction_created_date to DATE while leaving transaction_modified_date as a raw string, even though both are identical VARCHAR datetime columns in the source. Added prompt clarifications for the column omission and the date handling. f1002: models/stats/__stats.yml lists p1, p2, p3 columns on most_podiums that the gold excludes (they belong on finishes_by_driver). The task says to "fit the yml config," so the config and the gold contradict each other. Removed the three columns so the config matches the gold. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found these issues while testing my agent against the benchmark.
For the AE tasks setup.sh deletes the canonical model files that the task asks you to rebuild, then runs dbt. AE004 removes obt_product_inventory.sql. AE006 removes obt_product_inventory.sql, dim_products.sql, fact_inventory.sql, and obt_sales_overview.sql.
The issue is that those deleted files are the only place the gold's column and type choices ever lived. Once they're gone, nothing in the prompt, the project, or the data points to those choices, so the agent has no way to reproduce the gold. The fixes just put the missing intent back into the prompt. There are a few open issues calling out the AE tasks so this fix resolves several of them.
AE004
Gold aliases the duplicate product_id as "ipd" and drops the attachments column, neither of which is in the prompt. Dropping attachments isn't discoverable: it's 100% null, but so are description, purchase_order_id and customer_order_id, which the gold keeps. Added the ipd alias and the attachments exclusion to the prompt.
AE006
Same ipd alias and attachments exclusion, plus fact_inventory casts transaction_created_date to a date while keeping transaction_modified_date as a string, even though both are the same varchar in the source. Added the column and date handling to the prompt.
F1002
__stats.yml lists p1/p2/p3 on most_podiums but the gold doesn't include them. The task says to fit the yml, so I fixed the yml to match the gold instead. I saw a bunch of PRs open to fix this issue but figured I'd add it here as well.