Support per-backend role and destination in replication engine#2743
Support per-backend role and destination in replication engine#2743maeldonn wants to merge 4 commits into
Conversation
Hello maeldonn,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Codecov Report❌ Patch coverage is
Additional details and impacted files
... and 5 files with indirect coverage changes
@@ Coverage Diff @@
## development/9.4 #2743 +/- ##
===================================================
+ Coverage 74.88% 75.08% +0.20%
===================================================
Files 200 200
Lines 13670 13681 +11
===================================================
+ Hits 10237 10273 +36
+ Misses 3423 3398 -25
Partials 10 10
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
maeldonn
force-pushed
the
improvement/BB-762/crr-multi
branch
from
36b459a to
98c878f
Compare
|
|
maeldonn
force-pushed
the
improvement/BB-762/crr-multi
branch
from
d5a834d to
83c75ef
Compare
|
|
maeldonn
force-pushed
the
improvement/BB-762/crr-multi
branch
from
3426739 to
4791b8b
Compare
|
maeldonn
force-pushed
the
improvement/BB-762/crr-multi
branch
from
4791b8b to
14ef8f1
Compare
|
PR Review Summary
Review by Claude Code |
maeldonn
force-pushed
the
improvement/BB-762/crr-multi
branch
from
14ef8f1 to
cfdd8fb
Compare
The rest of the PR looks solid. The per-backend key abstraction (site + destination + role) is well-designed, backward compatibility is properly handled via top-level fallback in the site-aware getters, and the async/await migration in QueueProcessor.processReplicationEntry is clean. Test coverage is thorough, including same-site multi-destination disambiguation, retry round-tripping, and mixed CRR + cloud backend scenarios. Review by Claude Code |
maeldonn
force-pushed
the
improvement/BB-762/crr-multi
branch
from
cfdd8fb to
34fba43
Compare
| "@smithy/node-http-handler": "^3.3.3", | ||
| "JSONStream": "^1.3.5", | ||
| "arsenal": "git+https://github.com/scality/arsenal#8.3.9", | ||
| "arsenal": "git+https://github.com/scality/arsenal#e4860ebc364db6c3cd5b3f3210339d0fa512ed24", |
There was a problem hiding this comment.
Arsenal is pinned to a commit hash instead of a tag. The yarn.lock shows this resolves to version 8.4.3 — pin to the tag instead for consistency with the project's dependency pinning convention.
— Claude Code
|
maeldonn
force-pushed
the
improvement/BB-762/crr-multi
branch
2 times, most recently
from
7a21bbc to
91a58ad
Compare
| "@smithy/node-http-handler": "^3.3.3", | ||
| "JSONStream": "^1.3.5", | ||
| "arsenal": "git+https://github.com/scality/arsenal#8.3.9", | ||
| "arsenal": "git+https://github.com/scality/arsenal#e4860ebc364db6c3cd5b3f3210339d0fa512ed24", |
There was a problem hiding this comment.
arsenal is pinned to a commit hash instead of a tag. The resolved version is 8.4.3 — if that tag exists, pin to it for consistency with the other git-based deps (breakbeat, httpagent, etc.).
— Claude Code
The code changes themselves are solid. The per-backend routing in QueueProcessor, the backendKey-based disambiguation in ObjectQueueEntry, the role-validation expansion in ReplicateObject._setupRolesOnce, and the callbackify bridge for the async migration are all well-structured. Legacy backward compatibility (top-level fallback when backends lack destination/role) is covered by tests. The Kafka message format extension (adding destination/role fields) is additive and safe for rolling upgrades. Review by Claude Code |
| const replicationEnabled = data.ReplicationConfiguration.Rules | ||
| .some(rule => rule.Status === 'Enabled' && | ||
| entry.getObjectKey().startsWith(rule.Prefix)); | ||
| entry.getObjectKey().startsWith( |
There was a problem hiding this comment.
MultipleBackendTask._setupRolesOnce still calls entry.getReplicationRoles() without passing the backend key, while the parent ReplicateObject._setupRolesOnce was updated to entry.getReplicationRoles(entry.getBackendKey()). If arsenal's getReplicationRoles(key) resolves a per-backend role when a key is provided, this cloud-backend path will silently use the top-level (legacy) role string instead of the per-backend one, defeating the purpose of this PR for cloud destinations.
— Claude Code
| Bucket: sourceEntry.getBucket(), | ||
| Key: sourceEntry.getObjectKey(), | ||
| StorageType: sourceEntry.getReplicationStorageType(), | ||
| StorageType: this._getReplicationEndpointType(), |
There was a problem hiding this comment.
Several calls in MultipleBackendTask.processQueueEntry still use this.site (a plain string) for per-backend lookups instead of the key object:\n- Line 1220: refreshedEntry.getReplicationSiteStatus(this.site) should be refreshedEntry.getReplicationSiteStatus(sourceEntry.getBackendKey())\n- Line 1240: sourceEntry.getReplicationSiteDataStoreVersionId(this.site) should also use the backend key\n\nThese were not updated to match the pattern applied in ReplicateObject and UpdateReplicationStatus. If arsenal's methods now support the key object, this will miss per-backend state.
— Claude Code
| status: 'PENDING', | ||
| backends, | ||
| content, | ||
| role: ReplicationConfiguration.resolveSourceRole(bucketRepInfo.role), |
There was a problem hiding this comment.
The _updateReplicationInfo call on line 349 sets role: ReplicationConfiguration.resolveSourceRole(bucketRepInfo.role) but doesn't set destination or storageClass in the replication info object. The old code set both destination and storageClass. If these fields are now handled inside ReplicationConfiguration.resolveBackends at the backend level, this is fine — but please confirm the downstream consumers (queue populator, status processor) don't rely on the top-level destination / storageClass fields being present.
— Claude Code
| err => err ? reject(err) : resolve(), | ||
| ); | ||
| }); | ||
| } |
There was a problem hiding this comment.
The sequential for...of loop processes backends one at a time. If one backend's taskScheduler.push rejects, the await will throw and subsequent backends will be skipped entirely (the error propagates to callbackify which calls done(err)). This means a transient failure replicating to destination A will prevent replication to destinations B and C for this Kafka entry. Consider whether each backend should be attempted independently (catching errors per-backend) so one failure doesn't block the others.
— Claude Code
There was a problem hiding this comment.
Good raised question ?
| toRetryEntry(key) { | ||
| const backend = this._findBackend(key); | ||
| return this.clone() | ||
| .setAccountId(this.getAccountId()) |
There was a problem hiding this comment.
The toRetryEntry method now uses this._findBackend(key) but the method _findBackend is not defined on ObjectQueueEntry — it must be on the parent ObjectMD in arsenal. Please confirm this method exists in the arsenal commit being pinned (c3f3bff), since it's not possible to verify from this repo alone.
— Claude Code
|
maeldonn
force-pushed
the
improvement/BB-762/crr-multi
branch
from
91a58ad to
00094e7
Compare
| "@smithy/node-http-handler": "^3.3.3", | ||
| "JSONStream": "^1.3.5", | ||
| "arsenal": "git+https://github.com/scality/arsenal#8.3.9", | ||
| "arsenal": "git+https://github.com/scality/arsenal#c3f3bffde6eeb935e03847df010ac37ac7ba8944", |
There was a problem hiding this comment.
Arsenal is pinned to a commit hash instead of a release tag. Per project policy, git-based deps must pin to a tag, not a branch or commit. This should be updated to a tagged release before merging.
— Claude Code
|
eve-ci-cd
Bot
temporarily deployed
to
zenko/improvement/ZENKO-5253/multi-crr@2.15
Read destination bucket and role from each backends[] entry instead of the shared top-level fields, so a single source object can be replicated to multiple CRR destinations with their own role. Legacy entries without per-backend fields keep working via top-level fallback in ObjectQueueEntry's site-aware getters; MongoQueueProcessor's oplog path now matches every applicable rule and dedups backends per the design's (site, destination, role) rule. Issue: BB-762
maeldonn
force-pushed
the
improvement/BB-762/crr-multi
branch
from
00094e7 to
0b2138b
Compare
| "@smithy/node-http-handler": "^3.3.3", | ||
| "JSONStream": "^1.3.5", | ||
| "arsenal": "git+https://github.com/scality/arsenal#8.3.9", | ||
| "arsenal": "git+https://github.com/scality/arsenal#724d06289e11a429dfdf9e1236f98e904f748a90", |
There was a problem hiding this comment.
Arsenal dependency is pinned to a bare commit hash instead of a version tag. Per project conventions, git-based deps (arsenal, vaultclient, etc.) should pin to a tag for clarity and reproducibility. If this commit corresponds to a release, pin to that tag instead.
— Claude Code
| .filter(b => b.status === 'PENDING' && b.site === this.site); | ||
|
|
||
| if (pendingBackends.length === 0) { | ||
| logSkip(); |
There was a problem hiding this comment.
Inconsistent indentation — these two lines use 6 extra leading spaces (14 total) instead of the file's 4-space indent (12 expected at this nesting level).
— Claude Code
|
PR Review Summary
Observations (non-blocking):
Review by Claude Code |
| const replicationEnabled = ( | ||
| data.ReplicationConfiguration.Rules.some( | ||
| rule => entry.getObjectKey().startsWith(rule.Prefix) | ||
| rule => entry.getObjectKey().startsWith( |
There was a problem hiding this comment.
Is this a change of behavior with older version because of the '' fallback ?
edit: Ok i guess startWith('') is always true, so I guess its fine unless we want to treat an empty prefix filter as invalid ?
| if (roles.length !== 2) { | ||
| log.error('expecting two roles separated by a ' + | ||
| 'comma in bucket replication configuration', | ||
| if (roles.length > 2) { |
There was a problem hiding this comment.
nit: check len == 1 || 2 ?
But also, are you sure we shouldn't just enforce === 2 ? In what case could it be 1
3 participants
Read destination bucket and role from each backends[] entry instead of the shared top-level fields, so a single source object can be replicated to multiple CRR destinations with their own role. Legacy entries without per-backend fields keep working via top-level fallback in ObjectQueueEntry's site-aware getters; MongoQueueProcessor's oplog path now matches every applicable rule and dedups backends per the design's (site, destination, role) rule.
Issue: BB-762