Skip to content

[FLINK-40108] Adding snapshot capability with split enumerator#209

Open
eskabetxe wants to merge 1 commit into
apache:mainfrom
eskabetxe:postgres_splitter_db
Open

[FLINK-40108] Adding snapshot capability with split enumerator#209
eskabetxe wants to merge 1 commit into
apache:mainfrom
eskabetxe:postgres_splitter_db

Conversation

@eskabetxe

Copy link
Copy Markdown
Member

This pull request introduces a new connection management layer for the Flink JDBC connector, adding support for pluggable connection providers and improved connection configuration. The changes include new classes for connection options, a builder for those options, a minimal DataSource implementation for integration with connection pools, and interfaces for connection and statement handling. Additionally, HikariCP is added as a dependency (with SLF4J excluded), and a new async enumerator is provided for snapshot splitting.

Connection Management and Configuration:

  • Added ConnectionOptions and ConnectionOptionsBuilder classes to encapsulate and build JDBC connection configuration, including timeouts and properties. [1] [2]
  • Introduced ConnectionDataSource, a minimal DataSource implementation that delegates driver resolution, enabling proper classloader handling with connection pools like HikariCP.
  • Added ConnectionException for consistent error handling around connection issues.
  • Added ConnectionProvider interface, extending the base JDBC connection provider with methods for table/column metadata discovery and query building for snapshot splitters.

Statement and Result Handling:

  • Introduced ResultSetMapper and StatementPreparer functional interfaces for mapping JDBC results and preparing statements, respectively. [1] [2]

Async Snapshot Splitting:

  • Added AsyncSnapshotSplitterEnumerator, a background-threaded enumerator for computing and emitting snapshot splits asynchronously, improving scalability and responsiveness during snapshotting.

Dependency Management:

  • Added HikariCP as a dependency in pom.xml, with SLF4J excluded to avoid conflicts.

@eskabetxe

Copy link
Copy Markdown
Member Author

@RocMarshal @snuyanzin can you check please

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant