Model updates for bank fraud reporting and bank identifier migrations (SYN-10665)

[mikemoritz]

Summary

Model updates to support ingesting bank fraud and scam reporting from a third party feed, plus the model revision the type changes require. These bring the 2.x model closer to 3.x for threat clustering and bank identifiers.

The model-0.2.37 regression cortex that test_modelrev_0_2_37 loads landed in vertexproject/synapse-regression#33.

risk:threat:ext:ids

:ext:id gains 'alts': ('ext:ids',) plus the array property, mirroring meta:cluster:id/:ids verbatim. The vendor mints one report id per report and a scam operator spans several, so the cluster collapses onto one risk:threat and must resolve from any member id.

risk:attack:actor

Typed entity:actor, the existing 2.x ndef over ou:org, ps:person, ps:contact and risk:threat which entity:relationship:source and :target already use. This matches 3.x, where risk:attack picks up :actor from the entity:action interface typed entity:actor. 2.x risk:attack previously had no route to risk:threat at all -- :attacker is a ps:contact.

Note it sits next to the deprecated :actor:org and :actor:person. Those are unrelated and stay deprecated in favour of :attacker.

econ:bank:account:swift:bic

The econ:bank:swift:bic form existed but had nowhere to hang off an account, while :aba:rtn did. That asymmetry does not reflect real reporting, where the BIC is at least as common an account identifier as an ABA routing number.

Placed beside :aba:rtn, since both identify the bank which issued the account, where :number and :iban identify the account itself.

This is not a property-to-property copy for a future 3.x migration -- 3.x has no :swift:bic anywhere, because routing lives in the primary value of the econ:bank:account comp and econ:bank:swift:bic is one of three forms implementing econ:bank:routing:code. The gain is that the account-to-BIC pairing becomes machine-readable rather than only inferable through :issuer and the bank's :business property, which is ambiguous once a bank has more than one BIC. It makes the BIC exactly as portable as :aba:rtn already is.

Bank identifier types

All three now match 3.x exactly:

type	before	after
econ:bank:aba:rtn	[0-9]{9}	^[0-9]{9}$
econ:bank:iban	[A-Z]{2}[0-9]{2}[a-zA-Z0-9]{1,30}	^[A-Z]{2}[0-9]{2}[a-zA-Z0-9]{1,30}$
econ:bank:swift:bic	[A-Z]{6}[A-Z0-9]{5}	^[A-Z]{6}[A-Z0-9]{2}([A-Z0-9]{3})?$

The BIC change is the one motivating this work: the 8 character ISO 9362 head-office form is entirely normal and was rejected outright, so values such as BOPIPHMM and TRWIBEB1 raised BadTypeValu. The other two are pure narrowings, aligned so the two model versions agree on what these identifiers are.

Model revision 0.2.37

Str.norm calls self.regex.match() (synapse/lib/types.py:1862), which anchors only at the start. All three previous patterns therefore accepted trailing characters after an otherwise valid value -- DEUTDEFFXXXXX, 1234567890 and GB29NWBK60161331926819!! all normalized. Anchoring the end narrows released types, so data at rest may no longer re-norm.

None of these values has a derivable correct replacement, so ModelMigration_0_2_37 removes the node rather than moving it. Each removal is logged at WARNING with the node's iden, and the full record -- properties, tags, nodedata, both edge directions, and the referring nodes -- goes into the model_0_2_37:nodes queue so an operator can rebuild it. The queue is created lazily on first write, so a Cortex with nothing to migrate never gets one.

It is smaller than the two existing primary-value migrations because no node's primary value changes: every valid value the old patterns accepted is still accepted. moveNode is never called, which also leaves the base class's recursive comp-form re-key path dormant.

econ:bank:account does have properties typed econ:bank:aba:rtn and econ:bank:iban, so inbound references are cleared. Neither property is read-only, so removeNode() deletes the property and the account survives rather than being removed with the identifier. A read-only referrer is removed and queued under its own record instead; the regression cortex covers that arm with a meta:seen.

Repair library

$lib.model.migration.s.model_0_2_37 provides listNodes(), printNode() and repairNode(), matching the existing model_0_2_31 library.

This follows _0_2_31 rather than _0_2_35. The distinction is not whether a migration writes a queue but whether the queue is ever expected to be non-empty: both of _0_2_35's removeNode() calls sit inside except Exception: # pragma: no cover and its normal path is moveNode(), so its queue is expected to stay empty and no test asserts it. Removal is this migration's only outcome.

Repair matters more here than the node contents suggest. The econ:bank:account properties which pointed at a removed identifier are cleared by the migration, and repairNode() re-points them at the corrected value -- rebuilding the node by hand would restore the node but silently leave the account without its identifier. listNodes()'s form filter, inert on _0_2_31's single-form queue, is useful on a three-form one.

printNode() marks read-only references, because repairNode() cannot re-point a read-only property -- the referring node was removed and queued under its own entry, so restoring it is a second call that was previously undiscoverable. This improves the _0_2_31 library too, which has the same two-step repair.

LibModelMigrationsQueue was extracted so both libraries share the implementation and differ only by queuename. The three _meth entry points are declared on each subclass rather than inherited: registry.registerLib() stamps _storm_funcpath onto the function object, so with inherited methods both libraries resolve to the same object and the second registration relabels the first -- which would have renamed the released _0_2_31 library in its own error messages.

Tests

• test_model_risk.py -- :ext:id/:ext:ids with strip applied, the alts resolve risk:threat=({"ext:id": <peer id>}), and :actor set to both a risk:threat and an ou:org with pivots to each
• test_model_economic.py -- BIC 8 character accepted, 7/9/10/13 rejected; RTN and IBAN trailing-character cases rejected; :swift:bic set on an account with a value assertion and a -> econ:bank:swift:bic pivot
• test_lib_modelrev.py -- the repair library (form-filtered listing, printNode showing the referring property and flagging read-only refs, repairNode restoring both the node and the account property, and the queue-absent messages), plus test_modelrev_0_2_37 in the usual two-block form. The quarantined BIC spans two layers, so the record carries a sode and nodedata for each and the migration's missing-valu guard is exercised: maxvers=(0, 2, 36) asserting the pre-migration state, then a full boot asserting the survivors keep their properties, all six invalid values are gone (one from a forked view), the referencing account survives with :number intact and both identifier properties cleared, and each queued record carries the referring node plus the property which pointed at it

test_lib_modelrev.py is 36 passed / 0 skipped with SYN_REGRESSION_REPO set, and every other regression cortex migrates cleanly through the new step. The no-invalid-nodes path -- what almost every real Cortex will take -- is covered by the existing model-0.2.36 fixture.

Changelog

A migration fragment covering the removals, the cleared account properties and the read-only cascade, plus a feat fragment for the new Storm library. No fragments for the model changes themselves -- ModelDiffer picks up all of them against the v2.250.0 model ref (both new properties, the ext:id alts addkey, and all three types with old and new regexes), and they are covered by the generated model update doc.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HNhaKFd1Qi8Mkbn7AfbhPs

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.84%. Comparing base (56c0f02) to head (62a3400).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5008      +/-   ##
==========================================
- Coverage   97.84%   97.84%   -0.01%     
==========================================
  Files         308      308              
  Lines       65593    65696     +103     
==========================================
+ Hits        64180    64280     +100     
- Misses       1413     1416       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.