Skip to content

fix: backup fails with "disk quota exceeded" when account is near its storage quota #181

Description

@alehostert

Static Badge

Description

Container backups are failing with disk quota exceeded errors when the account's storage usage is close to its quota limit. Ez has an mechanism designed to temporarily increase the account's storage quota before a backup operation to ensure enough headroom for snapshot/archive generation. However, this mechanism is not working correctly — backups still fail with quota errors, and manual quota increases resolve the issue.

Error Log (anonymized)

[accountId/1001] [containerId/abcd1234] CreateContainerArchiveFailed: CreateSnapshotImageFailed: {"stdErr":"Error: copying layers and metadata for container \"abcd1234...\": initializing source containers-storage:example-client.com: storing layer \"bf5090835f9a...\" to file: on copy: write /var/tmp/buildah3560078150/layer: disk quota exceeded\n","exitCode":125}

Affected Scenario

  • Account quota: 80 GiB
  • Account usage: ~51 GiB
  • Container size: large (close to account usage)
  • Backup fails with disk quota exceeded

After manually increasing the account's storage quota by 100 GiB (temporary), the backup completed successfully. This confirms that:

  1. ✅ The underlying infrastructure has enough total disk space (the issue is not total disk space)
  2. ❌ The automatic temporary quota allocation in Ez is not taking effect

Given that manually bumping the quota by 100 GiB resolved the issue, the mechanism is clearly either not running or not applying before the disk-intensive operation starts.

Expected Behavior

  1. Before any backup/snapshot operation, Ez ensures the parent account has enough storage headroom
  2. If not enough free space, the quota is temporarily increased
  3. After backup completes, the quota is restored (or kept if a permanent increase is desired)

Steps to Reproduce

  1. Create an account with a moderate storage quota (e.g., 20 GiB)
  2. Push data to use ~80%+ of that quota
  3. Attempt to create a backup of a container whose size exceeds the remaining free space
  4. Observe the disk quota exceeded error

Additional Context

  • /var/tmp is symlinked to a larger volume (/var/data), so the issue is not about /var/tmp being full — it's specifically an XFS quota on the account
  • Affects multiple customers; likely affects any account with high storage utilization relative to quota

Severity

Medium/High — prevents backups for accounts near their quota limit, leading to data loss risk. Since Ez is expected to handle this automatically, this is a regression in expected behavior.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    approved

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions