Skip to content

validate_shards is too slow on large datasets — eats seconds before every SQL query #73

Description

@tig888

wsds.utils.validate_shards is called eagerly inside WSDataset.sql_select, to "warm up the filesystem cache" before any read. On a 7010-shard dataset on Weka with ~6 column dirs, this takes ~21s per call, dominating query latency for any workload that opens the dataset.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions