Problem
WSDS automatically prefixes field names with their column directory names to avoid internal collisions. However, this causes issues when merging multi-column data into flat output dictionaries that downstream code expects to access by unprefixed names.
Example
When WSDS loads a sample from multiple column directories:
quality.in-progress/quality_score → discovered as quality.quality_scoretags_ml.in-progress/quality_score → discovered as tags_ml.quality_score
When merging these into a flat output dict:
merged_row = {}
for k in sample.keys():
merged_row[k] = sample[k]The output contains prefixed field names:
merged_row['quality.quality_score'] = 0.95
merged_row['tags_ml.quality_score'] = 0.80
But downstream processing expects unprefixed names like quality_score. This causes:
KeyError when accessing missing columns
Silent data loss when column names don't match
Cryptic failures downstream when merging data from many column directories
Root Cause
WSDS abstracts away directory structure by embedding it in field names using . as a separator. This is transparent for single-shard access, but breaks when flattening prefixed names for downstream processing.
demo_wsds_field_collision.py
Problem
WSDS automatically prefixes field names with their column directory names to avoid internal collisions. However, this causes issues when merging multi-column data into flat output dictionaries that downstream code expects to access by unprefixed names.
Example
When WSDS loads a sample from multiple column directories:
quality.in-progress/quality_score→ discovered asquality.quality_scoretags_ml.in-progress/quality_score→ discovered astags_ml.quality_scoreWhen merging these into a flat output dict:
The output contains prefixed field names:
But downstream processing expects unprefixed names like quality_score. This causes:
KeyErrorwhen accessing missing columnsSilent data loss when column names don't match
Cryptic failures downstream when merging data from many column directories
Root Cause
WSDS abstracts away directory structure by embedding it in field names using . as a separator. This is transparent for single-shard access, but breaks when flattening prefixed names for downstream processing.
demo_wsds_field_collision.py