Skip to content

Fix intermittent Thrift port bind failures - #726

Open
gtrettenero wants to merge 2 commits into
masterfrom
gtret/fix-thrift-port-bind
Open

gtrettenero wants to merge 2 commits into
masterfrom
gtret/fix-thrift-port-bind

Conversation

@gtrettenero

Copy link
Copy Markdown
Collaborator

Summary

  • correct the Thrift socket retry delay so exponential backoff uses seconds rather than raw milliseconds
  • preserve the underlying bind exception and bind before allocating worker pools
  • retain the actual Thrift server instances for health checks and shutdown
  • validate duplicate listener ports and roll back partially started servers
  • continue core-service shutdown when Thrift cleanup fails

Why

Metacat nodes intermittently fail startup when port 12001 is temporarily unavailable. The existing five retries run only milliseconds apart because guava-retrying treats the exponential multiplier as milliseconds. Thrift 0.9.2 also removes the underlying IOException, making the bind failure difficult to diagnose.

The service lifecycle additionally recreated server objects for health and shutdown instead of retaining the instances that own the sockets.

Testing

  • :metacat-thrift:test
  • targeted MetacatThriftServiceSpec and MetacatThriftInitServiceSpec
  • :metacat-thrift:checkstyleMain
  • :metacat-main:checkstyleMain
  • RAILGUARD diff scan: no findings

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant