Summary
Model weights and label maps are downloaded from a single public object store whose base URL is now defined once, in trapdata/common/constants.py, as OBJECT_STORE_BASE_URL. That fixed the immediate breakage after the storage migration, but the location is still fixed at build time. A deployment that wants to serve models from somewhere else — an internal mirror, a cache close to a compute cluster, a local MinIO during development, or a future storage move — has to edit source and ship a new build.
The proposal is to let the base URL be set from the environment, keeping the current value as the default so nothing changes for anyone who does not set it.
Suggested shape
OBJECT_STORE_BASE_URL = os.environ.get("AMI_OBJECT_STORE_BASE_URL", "<current default>")
MODEL_BASE_URL = f"{OBJECT_STORE_BASE_URL}ami-models/"
IMAGE_BASE_URL = f"{OBJECT_STORE_BASE_URL}ami-trapdata/"
Points worth deciding when this is picked up:
- One variable or two. A single object-store base assumes models and trap images always live together. Separate
AMI_MODEL_BASE_URL and AMI_IMAGE_BASE_URL variables are more flexible; one variable is less to configure. A mirror is likely to carry models only, which is an argument for two.
- Trailing slash handling. Accept a value with or without a trailing slash rather than producing a silently malformed URL from a missing one.
- Where it gets documented. The worker deployment docs and the README setup section both need the variable listed, or it will only ever be discovered by reading the source.
- Startup validation. Worth considering a single reachability check at worker startup, so a wrong base URL fails immediately with a clear message instead of on the first model download deep inside a job.
Why this is a follow-up rather than part of the fix
The immediate problem was that the old base URL was written out in full 34 times, so there was no single place to change and every download failed. Consolidating to one constant solved that and is worth having on its own. Making that constant configurable is a separate, additive change with its own decisions (above), and does not need to hold up the fix.
Background
The two UK Turing model references (turing-uk_v03_resnet50_2024-05-13-10-03_state.pt and 03_uk_data_category_map.json) are absent from the bucket. The old and new buckets hold identical bytes, so these keys were already missing before the storage migration — they are a separate pre-existing issue, not something a configurable base URL would fix.
Summary
Model weights and label maps are downloaded from a single public object store whose base URL is now defined once, in
trapdata/common/constants.py, asOBJECT_STORE_BASE_URL. That fixed the immediate breakage after the storage migration, but the location is still fixed at build time. A deployment that wants to serve models from somewhere else — an internal mirror, a cache close to a compute cluster, a local MinIO during development, or a future storage move — has to edit source and ship a new build.The proposal is to let the base URL be set from the environment, keeping the current value as the default so nothing changes for anyone who does not set it.
Suggested shape
Points worth deciding when this is picked up:
AMI_MODEL_BASE_URLandAMI_IMAGE_BASE_URLvariables are more flexible; one variable is less to configure. A mirror is likely to carry models only, which is an argument for two.Why this is a follow-up rather than part of the fix
The immediate problem was that the old base URL was written out in full 34 times, so there was no single place to change and every download failed. Consolidating to one constant solved that and is worth having on its own. Making that constant configurable is a separate, additive change with its own decisions (above), and does not need to hold up the fix.
Background
The two UK Turing model references (
turing-uk_v03_resnet50_2024-05-13-10-03_state.ptand03_uk_data_category_map.json) are absent from the bucket. The old and new buckets hold identical bytes, so these keys were already missing before the storage migration — they are a separate pre-existing issue, not something a configurable base URL would fix.