Row policy for Iceberg and swarm

[alexvzin]

Describe the bug
SWARM servers are declared as completely stateless, but in the case of ROW policy, this does not work.

To Reproduce
Create a database with the Iceberg engine on clickhouse swarm client server

CREATE DATABASE ice
ENGINE = DataLakeCatalog('http://ice-rest-catalog:5000')
SETTINGS catalog_type = 'rest', auth_header = '[HIDDEN]', storage_endpoint = 'http://<ENDPOINT>', warehouse = 's3://<BUCKET>'

There i create a table, equivalent to the table in this example:

https://clickhouse.com/docs/cloud/bestpractices/multi-tenancy#shared-table-example

 :) select * from ice.`default.events`

SELECT *
FROM ice.`default.events`

Query id: b19631c2-57bf-48af-b291-c482d144193b

    ┌─tenant_id─┬─UUID─────────────────────────────────┬─mcc─────────┬──timestamp─┬─user_id─┬─data────────────────────────────────────┐
 1. │         1 │ 7b7e0439-99d0-4590-a4f7-1cfea1e192d1 │ user_login  │ 1742371200 │    1001 │ {"device": "desktop", "location": "LA"} │
 2. │         1 │ 846aa71f-f631-47b4-8429-ee8af87b4182 │ purchase    │ 1742371500 │    1002 │ {"item": "phone", "amount": 799}        │
 3. │         1 │ 6b4d12e4-447d-4398-b3fa-1c1e94d71a2f │ user_logout │ 1742371800 │    1001 │ {"device": "desktop", "location": "LA"} │
 4. │         1 │ 83b5eb72-aba3-4038-bc52-6c08b6423615 │ purchase    │ 1742373900 │    1003 │ {"item": "monitor", "amount": 450}      │
 5. │         1 │ 975fb0c8-55bd-4df4-843b-34f5cfeed0a9 │ user_login  │ 1742374200 │    1004 │ {"device": "desktop", "location": "LA"} │
 6. │         2 │ 7162f8ea-8bfd-486a-a45e-edfc3398ca93 │ user_login  │ 1742371920 │    2001 │ {"device": "mobile", "location": "SF"}  │
 7. │         2 │ 6b5f3e55-5add-479e-b89d-762aa017f067 │ purchase    │ 1742372100 │    2002 │ {"item": "headphones", "amount": 199}   │
 8. │         2 │ 43ad35a1-926c-4543-a133-8672ddd504bf │ user_logout │ 1742372400 │    2001 │ {"device": "mobile", "location": "SF"}  │
 9. │         2 │ f50aa430-4898-43d0-9d82-41e7397ba9b8 │ purchase    │ 1742374500 │    2003 │ {"item": "laptop", "amount": 1200}      │
10. │         2 │ 5c150ceb-b869-4ebb-843d-ab42d3cb5410 │ user_login  │ 1742374800 │    2004 │ {"device": "mobile", "location": "SF"}  │
    └───────────┴──────────────────────────────────────┴─────────────┴────────────┴─────────┴─────────────────────────────────────────┘

10 rows in set. Elapsed: 0.315 sec.

Create users on the server for multi tenancy test:

-- Create users
CREATE USER user_1 IDENTIFIED BY '<password>'
CREATE USER user_2 IDENTIFIED BY '<password>'
-- Create row policies
CREATE ROW POLICY user_filter_1 ON ice.`default.events` USING tenant_id=1 TO user_1
CREATE ROW POLICY user_filter_2 ON ice.`default.events` USING tenant_id=2 TO user_2
-- Create role
CREATE ROLE user_role
-- Grant read only to events table.
GRANT SELECT ON default.events TO user_role
GRANT user_role TO user_1
GRANT user_role TO user_2

Select without swarm by user_1:

:) select * from ice.`default.events`

SELECT *
FROM ice.`default.events`

Query id: f035edde-95cf-46fc-83ad-afea2dd28556

   ┌─tenant_id─┬─UUID─────────────────────────────────┬─mcc─────────┬──timestamp─┬─user_id─┬─data────────────────────────────────────┐
1. │         1 │ 7b7e0439-99d0-4590-a4f7-1cfea1e192d1 │ user_login  │ 1742371200 │    1001 │ {"device": "desktop", "location": "LA"} │
2. │         1 │ 846aa71f-f631-47b4-8429-ee8af87b4182 │ purchase    │ 1742371500 │    1002 │ {"item": "phone", "amount": 799}        │
3. │         1 │ 6b4d12e4-447d-4398-b3fa-1c1e94d71a2f │ user_logout │ 1742371800 │    1001 │ {"device": "desktop", "location": "LA"} │
4. │         1 │ 83b5eb72-aba3-4038-bc52-6c08b6423615 │ purchase    │ 1742373900 │    1003 │ {"item": "monitor", "amount": 450}      │
5. │         1 │ 975fb0c8-55bd-4df4-843b-34f5cfeed0a9 │ user_login  │ 1742374200 │    1004 │ {"device": "desktop", "location": "LA"} │
   └───────────┴──────────────────────────────────────┴─────────────┴────────────┴─────────┴─────────────────────────────────────────┘

5 rows in set. Elapsed: 0.295 sec.

Select without swarm by user_2:

:) select * from ice.`default.events`

SELECT *
FROM ice.`default.events`

Query id: 828c82a6-a463-4195-b320-a1f7b956ddcf

   ┌─tenant_id─┬─UUID─────────────────────────────────┬─mcc─────────┬──timestamp─┬─user_id─┬─data───────────────────────────────────┐
1. │         2 │ 7162f8ea-8bfd-486a-a45e-edfc3398ca93 │ user_login  │ 1742371920 │    2001 │ {"device": "mobile", "location": "SF"} │
2. │         2 │ 6b5f3e55-5add-479e-b89d-762aa017f067 │ purchase    │ 1742372100 │    2002 │ {"item": "headphones", "amount": 199}  │
3. │         2 │ 43ad35a1-926c-4543-a133-8672ddd504bf │ user_logout │ 1742372400 │    2001 │ {"device": "mobile", "location": "SF"} │
4. │         2 │ f50aa430-4898-43d0-9d82-41e7397ba9b8 │ purchase    │ 1742374500 │    2003 │ {"item": "laptop", "amount": 1200}     │
5. │         2 │ 5c150ceb-b869-4ebb-843d-ab42d3cb5410 │ user_login  │ 1742374800 │    2004 │ {"device": "mobile", "location": "SF"} │
   └───────────┴──────────────────────────────────────┴─────────────┴────────────┴─────────┴────────────────────────────────────────┘

5 rows in set. Elapsed: 0.139 sec.

As we can see, row policy works

But then I do select with swarm by user_1

:) select * from ice.`default.events` SETTINGS object_storage_cluster='swarm'

SELECT *
FROM ice.`default.events`
SETTINGS object_storage_cluster = 'swarm'

Query id: 943e0b0f-304e-43ca-bc16-57acbe630062


Elapsed: 0.414 sec.

Received exception from server (version 26.1.6):
Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: Received from chi-swarm-swarm-0-0-0.chi-swarm-swarm-0-0.swarm-test.svc.cluster.local:9000. DB::Exception: user_1: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

That is, the swarm servers require user_1, and they are not completely stateless for this operation.

Adding users on swarm servers

CREATE USER user_1 IDENTIFIED BY 'pass1'
CREATE USER user_2 IDENTIFIED BY 'pass2'
CREATE ROW POLICY user_filter_1 ON ice.`default.events` USING tenant_id=1 TO user_1
CREATE ROW POLICY user_filter_2 ON ice.`default.events` USING tenant_id=2 TO user_2
CREATE ROLE user_role
GRANT SELECT ON ice.`default.events` TO user_role
GRANT user_role TO user_1
GRANT user_role TO user_2
GRANT S3 ON *.* TO user_1
GRANT S3 ON *.* TO user_2
GRANT CREATE TEMPORARY TABLE ON *.* to user_2
GRANT CREATE TEMPORARY TABLE ON *.* to user_1

Then I make a request by user_1

:) select * from ice.`default.events` SETTINGS object_storage_cluster='swarm-zav'

SELECT *
FROM ice.`default.events`
SETTINGS object_storage_cluster = 'swarm-zav'

Query id: 21f7cbee-91a8-45be-adff-6b7792d30fb5

    ┌─tenant_id─┬─UUID─────────────────────────────────┬─mcc─────────┬──timestamp─┬─user_id─┬─data────────────────────────────────────┐
 1. │         1 │ 7b7e0439-99d0-4590-a4f7-1cfea1e192d1 │ user_login  │ 1742371200 │    1001 │ {"device": "desktop", "location": "LA"} │
 2. │         1 │ 846aa71f-f631-47b4-8429-ee8af87b4182 │ purchase    │ 1742371500 │    1002 │ {"item": "phone", "amount": 799}        │
 3. │         1 │ 6b4d12e4-447d-4398-b3fa-1c1e94d71a2f │ user_logout │ 1742371800 │    1001 │ {"device": "desktop", "location": "LA"} │
 4. │         1 │ 83b5eb72-aba3-4038-bc52-6c08b6423615 │ purchase    │ 1742373900 │    1003 │ {"item": "monitor", "amount": 450}      │
 5. │         1 │ 975fb0c8-55bd-4df4-843b-34f5cfeed0a9 │ user_login  │ 1742374200 │    1004 │ {"device": "desktop", "location": "LA"} │
 6. │         2 │ 7162f8ea-8bfd-486a-a45e-edfc3398ca93 │ user_login  │ 1742371920 │    2001 │ {"device": "mobile", "location": "SF"}  │
 7. │         2 │ 6b5f3e55-5add-479e-b89d-762aa017f067 │ purchase    │ 1742372100 │    2002 │ {"item": "headphones", "amount": 199}   │
 8. │         2 │ 43ad35a1-926c-4543-a133-8672ddd504bf │ user_logout │ 1742372400 │    2001 │ {"device": "mobile", "location": "SF"}  │
 9. │         2 │ f50aa430-4898-43d0-9d82-41e7397ba9b8 │ purchase    │ 1742374500 │    2003 │ {"item": "laptop", "amount": 1200}      │
10. │         2 │ 5c150ceb-b869-4ebb-843d-ab42d3cb5410 │ user_login  │ 1742374800 │    2004 │ {"device": "mobile", "location": "SF"}  │
    └───────────┴──────────────────────────────────────┴─────────────┴────────────┴─────────┴─────────────────────────────────────────┘

As we can see, request is being successfully made, but ROW policy is not working.

Expected behavior
Completely stateless swarm and correctly working row policy.

Key information
I'm using images

altinity/clickhouse-server:26.1.6.20001.altinityantalya