wsds.utils.validate_shards is called eagerly inside WSDataset.sql_select, to "warm up the filesystem cache" before any read. On a 7010-shard dataset on Weka with ~6 column dirs, this takes ~21s per call, dominating query latency for any workload that opens the dataset.
wsds.utils.validate_shards is called eagerly inside WSDataset.sql_select, to "warm up the filesystem cache" before any read. On a 7010-shard dataset on Weka with ~6 column dirs, this takes ~21s per call, dominating query latency for any workload that opens the dataset.