
Description
Container backups are failing with disk quota exceeded errors when the account's storage usage is close to its quota limit. Ez has an mechanism designed to temporarily increase the account's storage quota before a backup operation to ensure enough headroom for snapshot/archive generation. However, this mechanism is not working correctly — backups still fail with quota errors, and manual quota increases resolve the issue.
Error Log (anonymized)
[accountId/1001] [containerId/abcd1234] CreateContainerArchiveFailed: CreateSnapshotImageFailed: {"stdErr":"Error: copying layers and metadata for container \"abcd1234...\": initializing source containers-storage:example-client.com: storing layer \"bf5090835f9a...\" to file: on copy: write /var/tmp/buildah3560078150/layer: disk quota exceeded\n","exitCode":125}
Affected Scenario
- Account quota: 80 GiB
- Account usage: ~51 GiB
- Container size: large (close to account usage)
- Backup fails with
disk quota exceeded
After manually increasing the account's storage quota by 100 GiB (temporary), the backup completed successfully. This confirms that:
- ✅ The underlying infrastructure has enough total disk space (the issue is not total disk space)
- ❌ The automatic temporary quota allocation in Ez is not taking effect
Given that manually bumping the quota by 100 GiB resolved the issue, the mechanism is clearly either not running or not applying before the disk-intensive operation starts.
Expected Behavior
- Before any backup/snapshot operation, Ez ensures the parent account has enough storage headroom
- If not enough free space, the quota is temporarily increased
- After backup completes, the quota is restored (or kept if a permanent increase is desired)
Steps to Reproduce
- Create an account with a moderate storage quota (e.g., 20 GiB)
- Push data to use ~80%+ of that quota
- Attempt to create a backup of a container whose size exceeds the remaining free space
- Observe the
disk quota exceeded error
Additional Context
/var/tmp is symlinked to a larger volume (/var/data), so the issue is not about /var/tmp being full — it's specifically an XFS quota on the account
- Affects multiple customers; likely affects any account with high storage utilization relative to quota
Severity
Medium/High — prevents backups for accounts near their quota limit, leading to data loss risk. Since Ez is expected to handle this automatically, this is a regression in expected behavior.
Description
Container backups are failing with
disk quota exceedederrors when the account's storage usage is close to its quota limit. Ez has an mechanism designed to temporarily increase the account's storage quota before a backup operation to ensure enough headroom for snapshot/archive generation. However, this mechanism is not working correctly — backups still fail with quota errors, and manual quota increases resolve the issue.Error Log (anonymized)
Affected Scenario
disk quota exceededAfter manually increasing the account's storage quota by 100 GiB (temporary), the backup completed successfully. This confirms that:
Given that manually bumping the quota by 100 GiB resolved the issue, the mechanism is clearly either not running or not applying before the disk-intensive operation starts.
Expected Behavior
Steps to Reproduce
disk quota exceedederrorAdditional Context
/var/tmpis symlinked to a larger volume (/var/data), so the issue is not about/var/tmpbeing full — it's specifically an XFS quota on the accountSeverity
Medium/High — prevents backups for accounts near their quota limit, leading to data loss risk. Since Ez is expected to handle this automatically, this is a regression in expected behavior.