From 187bf48f27811cc6caae3863016b2a532d1030ef Mon Sep 17 00:00:00 2001 From: Phil Eaton Date: Mon, 18 Aug 2025 11:38:38 -0400 Subject: [PATCH 1/8] Clarify multiple databases limitation --- product_docs/docs/pgd/6/known_issues.mdx | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/product_docs/docs/pgd/6/known_issues.mdx b/product_docs/docs/pgd/6/known_issues.mdx index 91ea635e9ab..15dd3b8c285 100644 --- a/product_docs/docs/pgd/6/known_issues.mdx +++ b/product_docs/docs/pgd/6/known_issues.mdx @@ -79,17 +79,17 @@ Take these EDB Postgres Distributed (PGD) design limitations into account when p ### Multiple databases on single instances -Support for using PGD for multiple databases on the same Postgres instance is -**deprecated** beginning with PGD 5 and will no longer be supported with PGD 6. As -we extend the capabilities of the product, the added complexity introduced -operationally and functionally is no longer viable in a multi-database design. +Because of many immediate risks and limitations, PGD 6 does not +officially support multiple databases on a single PostgreSQL +instance. A future release of PGD 6 may remove these risks and +limitations, but until then **we currently recommend a +one-database-per-instance configuration**. This best practice is also +reflected in our tooling, including the CLI and Connection Manager, +which are designed to work at the instance level. -It's best practice and we recommend that you configure only one database per PGD instance. +These immediate risks and current limitations include: -The tooling such as the CLI and Connection Manager currently codify that recommendation. - -While it's still possible to host up to 10 databases in a single instance, -doing so incurs many immediate risks and current limitations: +- A hard limit of 10 databases in a single instance. - If PGD configuration changes are needed, you must execute administrative commands for each database. Doing so increases the risk for potential From c3851a38916ac57f9e7560dc9b873b7d84649043 Mon Sep 17 00:00:00 2001 From: Phil Eaton Date: Mon, 18 Aug 2025 12:01:36 -0400 Subject: [PATCH 2/8] Simplify --- product_docs/docs/pgd/6/known_issues.mdx | 37 +++--------------------- 1 file changed, 4 insertions(+), 33 deletions(-) diff --git a/product_docs/docs/pgd/6/known_issues.mdx b/product_docs/docs/pgd/6/known_issues.mdx index 15dd3b8c285..2f09761c7d7 100644 --- a/product_docs/docs/pgd/6/known_issues.mdx +++ b/product_docs/docs/pgd/6/known_issues.mdx @@ -79,39 +79,10 @@ Take these EDB Postgres Distributed (PGD) design limitations into account when p ### Multiple databases on single instances -Because of many immediate risks and limitations, PGD 6 does not -officially support multiple databases on a single PostgreSQL -instance. A future release of PGD 6 may remove these risks and -limitations, but until then **we currently recommend a -one-database-per-instance configuration**. This best practice is also -reflected in our tooling, including the CLI and Connection Manager, -which are designed to work at the instance level. - -These immediate risks and current limitations include: - -- A hard limit of 10 databases in a single instance. - -- If PGD configuration changes are needed, you must execute administrative commands - for each database. Doing so increases the risk for potential - inconsistencies and errors. - -- You must monitor each database separately, adding overhead. - -- Connection Manager works at the Postgres instance level, not at the database level, - meaning the leader node is the same for all databases. - -- Each additional database increases the resource requirements on the server. - Each one needs its own set of worker processes maintaining replication, for example, - logical workers, WAL senders, and WAL receivers. Each one also needs its own - set of connections to other instances in the replication cluster. These needs might - severely impact performance of all databases. - -- Synchronous replication methods, for example, CAMO and Group Commit, won’t work as - expected. Since the Postgres WAL is shared between the databases, a - synchronous commit confirmation can come from any database, not necessarily in - the right order of commits. - -- CLI integration assumes one database. +Because of many immediate risks and limitations, we currently +recommend a one-database-per-instance configuration. This best +practice is also reflected in our tooling, including the CLI and +Connection Manager, which are designed to work at the instance level. ### Durability options (Group Commit/CAMO) From 7a4349aa6f8ee91fa3593f86e208182f353dfb10 Mon Sep 17 00:00:00 2001 From: Phil Eaton Date: Tue, 19 Aug 2025 18:48:05 -0400 Subject: [PATCH 3/8] More changes --- product_docs/docs/pgd/6/known_issues.mdx | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/product_docs/docs/pgd/6/known_issues.mdx b/product_docs/docs/pgd/6/known_issues.mdx index 2f09761c7d7..bea89470ee1 100644 --- a/product_docs/docs/pgd/6/known_issues.mdx +++ b/product_docs/docs/pgd/6/known_issues.mdx @@ -202,6 +202,23 @@ PGD was developed to [enable rolling upgrades of PGD](/pgd/latest/reference/upgr We expect users to run mixed versions only during upgrades and, once an upgrade starts, that they complete that upgrade. We don't support running mixed versions of PGD except during an upgrade. +### Large object replication + +Logical decoding and replication, which PGD is built upon, does not +support large objects. As an alternative, use TEXT, BYTEA, JSON, or +JSONB data types, as appropriate for the application, for values up to +1 GB in size. + +You can check for existing large objects with: + +```sql +SELECT count(*) FROM pg_largeobject; +``` + +### Slow performance for tables without primary keys or unique constraints + +For updates and deletes to replicate on other nodes, PGD must be able to identify the unique rows affected. If no primary key or unique index is defined for a table, performance of replication and conflict detection can be severely affected. See: https://www.enterprisedb.com/docs/pgd/latest/reference/appusage/behavior/#replication-behavior and https://www.enterprisedb.com/docs/pgd/latest/reference/tables-views-functions/pgd-settings/#bdrdefault_replica_identity + ### Other limitations This noncomprehensive list includes other limitations that are expected and From 2bfa29696d0b886ad492db2626ee9ee43f0e5bcf Mon Sep 17 00:00:00 2001 From: Phil Eaton Date: Wed, 27 Aug 2025 14:49:08 -0400 Subject: [PATCH 4/8] clarify --- product_docs/docs/pgd/6/known_issues.mdx | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/product_docs/docs/pgd/6/known_issues.mdx b/product_docs/docs/pgd/6/known_issues.mdx index bea89470ee1..274bcd48aa4 100644 --- a/product_docs/docs/pgd/6/known_issues.mdx +++ b/product_docs/docs/pgd/6/known_issues.mdx @@ -215,9 +215,27 @@ You can check for existing large objects with: SELECT count(*) FROM pg_largeobject; ``` -### Slow performance for tables without primary keys or unique constraints - -For updates and deletes to replicate on other nodes, PGD must be able to identify the unique rows affected. If no primary key or unique index is defined for a table, performance of replication and conflict detection can be severely affected. See: https://www.enterprisedb.com/docs/pgd/latest/reference/appusage/behavior/#replication-behavior and https://www.enterprisedb.com/docs/pgd/latest/reference/tables-views-functions/pgd-settings/#bdrdefault_replica_identity +### Slow performance for UPDATE/DELETE heavy workloads on large tables without primary keys or unique constraints + +For UPDATEs and DELETEs to replicate on other nodes, PGD must be able +to identify the unique rows affected. If no primary key or unique +index is defined for a table, performance of replication and conflict +detection can be severely affected. See: +https://www.enterprisedb.com/docs/pgd/latest/reference/appusage/behavior/#replication-behavior +and +https://www.enterprisedb.com/docs/pgd/latest/reference/tables-views-functions/pgd-settings/#bdrdefault_replica_identity. + +This is particularly something to keep in mind for making +UPDATEs/DELETEs of only a small number of rows in a very large table, +which would otherwise be cheap to replicate. + +!!! Note + For very small tables (a common use for non-indexed tables) things + can be actually faster without index. + +!!! Note + Replication of large UPDATE/DELETE query that affected millions of + rows will still be slow even with indexes. ### Other limitations From 83c96f8bd2c056c28a640262ccfaf8a9e0db7a14 Mon Sep 17 00:00:00 2001 From: Phil Eaton Date: Wed, 27 Aug 2025 15:46:10 -0400 Subject: [PATCH 5/8] More limitations --- product_docs/docs/pgd/6/known_issues.mdx | 115 +++++++++++++++++++++++ 1 file changed, 115 insertions(+) diff --git a/product_docs/docs/pgd/6/known_issues.mdx b/product_docs/docs/pgd/6/known_issues.mdx index 274bcd48aa4..029f4e04ac9 100644 --- a/product_docs/docs/pgd/6/known_issues.mdx +++ b/product_docs/docs/pgd/6/known_issues.mdx @@ -237,6 +237,121 @@ which would otherwise be cheap to replicate. Replication of large UPDATE/DELETE query that affected millions of rows will still be slow even with indexes. +### EPAS Queue Tables not replicated + +EPAS [Queue +Tables](https://www.enterprisedb.com/docs/epas/latest/reference/oracle_compatibility_reference/epas_compat_sql/31_create_queue_table/) +are not replicated. Check for their existence with: + +```sql +SELECT * FROM pg_catalog.edb_queue_table; +``` + +### Explicit row-level locks not replicated + +Explicit row-level locks like `SELECT ... FOR UPDATE` / `SELECT +... FOR SHARE` are not replicated. As long as you only write to the +PGD write leader this will not be a problem. + +### Advisory locks not replicated + +Postgres advisory locks are not replicated by PGD or physical +streaming replication, and will not work across the replication +failover. + +Check your application code for uses of `pg_advisory*` and +`pg_try_advisory*`. + +### Avoid DELETE/INSERT of replica identity values + +Avoid modification of the replica identity values, such as primary +keys, especially in quick succession, as it may lead to replication +conflicts that cannot be automatically resolved, and subsequently to +divergent data errors. + +### TRIGGER and REFERENCES privileges + +TRIGGER and REFERENCES privileges are the two types of privileges that +are not commonly used with PostgreSQL and can cause problems with PGD. + +PGD supports triggers, but for security reasons blocks the case when +the trigger is not owned by the same user as the table on which it is +defined. That situation can occur when a non-table owner is explicitly +granted the TRIGGER privilege on a table. + +Trigger privileges granted explicitly to non-table-owners may be +present as part of trigger-based replication solutions, such as Slony +or Bucardo. PGD generally supersedes functionality of such solutions; +they are considered insecure and should be avoided. + +PGD supports foreign keys but for security and performance reasons +blocks the case when the owner of the referencing table does not have +the SELECT privilege on the referenced table. That situation can occur +when a user is explicitly granted the REFERENCES privilege on a table. + +Explicit GRANTs of TRIGGER or REFERENCES privileges can be detected +using the following query. These two cases are not related to each +other, but it is convenient to detect both at the same time with one +query. + +```sql +SELECT * FROM ( + SELECT relname + ,relowner::regrole as relowner + ,split_part(unnest(relacl)::text,'=', 1) as grantee + ,split_part(split_part(unnest(relacl)::text,'=', 2), '/', 1) as acl + FROM pg_class where relacl is not null) as a /* Get the raw ACL string(s) */ + WHERE relowner::regrole::text != grantee /* NOT table owner */ + AND ((strpos(acl, 'x') > 0 /* REFERENCES priv */ AND strpos(acl, 'r') = 0) + OR strpos(acl, 't') > 0 /* TRIGGER priv */ AND acl != 'arwdDxt' /* NOT ALL */); +``` + +Alternatively, you can look for insecure trigger functions using this query: + +```sql +SELECT relname, tgname, proname, relowner, proowner, prosecdef; +FROM pg_trigger t +JOIN pg_class c ON t.tgrelid = c.oid +JOIN pg_proc f ON t.tgfoid = f.oid +WHERE (relowner != proowner /* Different owner */ + OR prosecdef) /* SECURITY DEFINER */ + AND proowner != 10 /* Non-catalog functions */; +``` + +Resolve in the database by modifying trigger ownership and table +privileges as necessary. Note: if the affected objects are routinely +created by the application then this should be resolved in the +application. + +### Replica rules are ignored on replicas + +Rules, if present, are executed on the origin node, and ignored on replicas. + +To check for replica rules, run: + +```sql +SELECT rulename, ev_class, ev_type, ev_enabled +FROM pg_rewrite +WHERE ev_type != '1' + AND NOT is_instead + AND ev_enabled in ('R', 'A'); +``` + +Resolve in application by eliminating any use of rules that are +expected to fire on replication, if they exist. + +### LISTEN and NOTIFY not replicated + +These commands are not replicated and work on each cluster node +individually. Consider that notifications can be lost in case of a +failover. + +Triggers that send out NOTIFY on commit will work with PGD, as long as +they are defined as REPLICA TRIGGER. + +Instead of LISTEN and NOTIFY consider using replication slots +directly. + ### Other limitations This noncomprehensive list includes other limitations that are expected and From ebe14dc23de6a200ad2270cd6e25c1d94701b33b Mon Sep 17 00:00:00 2001 From: Phil Eaton Date: Wed, 27 Aug 2025 16:35:15 -0400 Subject: [PATCH 6/8] again --- product_docs/docs/pgd/6/known_issues.mdx | 27 +++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/product_docs/docs/pgd/6/known_issues.mdx b/product_docs/docs/pgd/6/known_issues.mdx index 029f4e04ac9..be49b7508ff 100644 --- a/product_docs/docs/pgd/6/known_issues.mdx +++ b/product_docs/docs/pgd/6/known_issues.mdx @@ -260,7 +260,17 @@ streaming replication, and will not work across the replication failover. Check your application code for uses of `pg_advisory*` and -`pg_try_advisory*`. +`pg_try_advisory*`. Consider using `bdr.global_advisory_lock()` +instead. + +### LOCK TABLE commands + +`LOCK TABLE` causes a global DML lock to be taken on the table across +the entire PGD cluster, which requires participation of all PGD +nodes. This behaviour is similar to +[bdr.global_lock_table()](https://www.enterprisedb.com/docs/pgd/latest/reference/tables-views-functions/functions/#bdrglobal_lock_table) +and can be disabled by setting the configuration variable +`bdr.lock_table_locking` to "off" (default is "on"). ### Avoid DELETE/INSERT of replica identity values @@ -352,6 +362,21 @@ they are defined as REPLICA TRIGGER. Instead of LISTEN and NOTIFY consider using replication slots directly. +### Sequences + +Standard sequences in PostgreSQL aren't multi-node aware and produce +values that are unique only on the local node. For this reason, PGD +provides an application-transparent way to generate unique identifiers +using sequences of integer datatypes across the whole PGD cluster, +called [global +sequences](https://www.enterprisedb.com/docs/pgd/latest/reference/sequences/). + +To find existing sequences, run: + +```sql +SELECT count(*) FROM pg_sequences; +``` + ### Other limitations This noncomprehensive list includes other limitations that are expected and From 290225a44f0bf0d3d76b383b624db1b41b6410d3 Mon Sep 17 00:00:00 2001 From: Phil Eaton Date: Wed, 27 Aug 2025 16:37:11 -0400 Subject: [PATCH 7/8] clarify --- product_docs/docs/pgd/6/known_issues.mdx | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/product_docs/docs/pgd/6/known_issues.mdx b/product_docs/docs/pgd/6/known_issues.mdx index be49b7508ff..d38daba391d 100644 --- a/product_docs/docs/pgd/6/known_issues.mdx +++ b/product_docs/docs/pgd/6/known_issues.mdx @@ -377,6 +377,17 @@ To find existing sequences, run: SELECT count(*) FROM pg_sequences; ``` +For sequences created on an existing PGD cluster, +bdr.default_sequence_kind is "distributed" by default, which means +that int8 sequences (i.e. BIGINT) will be created with the type +“snowflakeid”, while int4 (i.e. INTEGER) will be of the type +“galloc”. + +Set `bdr.default_sequence_kind` as appropriate for application +requirements and EDB recommendations. Any sequences that exist before +converting a Postgres database to a PGD node are converted +automatically when a PGD node group is first created. + ### Other limitations This noncomprehensive list includes other limitations that are expected and From 5ac842e0112cbdd1edf2d38d00ecce64ef85c985 Mon Sep 17 00:00:00 2001 From: Phil Eaton Date: Wed, 27 Aug 2025 16:41:49 -0400 Subject: [PATCH 8/8] assess command --- product_docs/docs/pgd/6/known_issues.mdx | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/product_docs/docs/pgd/6/known_issues.mdx b/product_docs/docs/pgd/6/known_issues.mdx index d38daba391d..19f6ef62397 100644 --- a/product_docs/docs/pgd/6/known_issues.mdx +++ b/product_docs/docs/pgd/6/known_issues.mdx @@ -60,7 +60,20 @@ To modify a commit scope safely, use [`bdr.alter_commit_scope`](/pgd/latest/refe ## Limitations -Take these EDB Postgres Distributed (PGD) design limitations into account when planning your deployment. +If you have an existing application that works with PostgreSQL, you +can quickly assess its compatibility with EDB Postgres Distributed by +using the +[assess](https://www.enterprisedb.com/docs/pgd/latest/reference/cli/command_ref/assess/) +function of the PGD command-line interface (CLI). + +The PGD CLI can be installed independently on any Linux or Mac OS +machine and will generate an assessment report after connecting to the +source PostgreSQL database remotely. + +We have also written out limitations the "assess" command checks here +for your reference. We have also noted additional limitations that +must be checked through static scanning and are thus beyond the scope +of the "assess" command. ### Nodes