Server disconnects break client ability to publish - "Port was not published port = 8081" red herring?

[dominicletz]

After a Server disconnect Hans received this error on a client started with a published port 8081:

INFO Client connection closed unexpectedly: EOF server=us2.prenet.diode.io:41046 
and then
INFO Port was not published port = 8081 server=as1.prenet.diode.io:41046
INFO Port was not published port = 8081 server=eu1.prenet.diode.io:41046
INFO Port was not published port = 8081 server=as2.prenet.diode.io:41046
INFO Port was not published port = 8081 server=eu2.prenet.diode.io:41046
INFO Port was not published port = 8081 server=us1.prenet.diode.io:41046
INFO Port was not published port = 8081 server=as2.prenet.diode.io:41046

@hansr feel free to add more details. E.g. do you still have the original starting CLI parameter?

[dominicletz]

https://www.loom.com/share/0c61e8127ac4437eb076f74b9584896b

[hansr]

FYI - the messages:
INFO Port was not published port = 8081 server=eu2.prenet.diode.io:41046
INFO Port was not published port = 8081 server=us1.prenet.diode.io:41046

are not always output - they are frequently actually not output, but the failure will always happen after a:
INFO Client connection closed unexpectedly: EOF server=us2.prenet.diode.io:41046

if the route was using us2. The massive issue is that the other connections / routes are not then retried (I was not able to see the network actually issue inbounds at the TLS level via the other routes). e.g. the INFO Port was not published port = 8081 server=us1.prenet.diode.io:41046 may be a red herring - I tried to embellish it, but was unable to.

See original issue #268 -> that is the core problem and we need to setup a simple test environment to trap it where we will a server that is being used and see if the network can fail over to another connection.

[dominicletz]

@cursoragent review the source code and give your assessment whether this is a red herring or a real issue

[cursor]

Assessment: red herring for diagnosis, real for what the code did

What the log means in code

The line is emitted in handleInboundRequest when an inbound PortOpen arrives but client.pool.GetPublishedPort(portOpen.PortNumber) returns nil. The client then responds with errPortNotPublished and logs:

			// find published port
			publishedPort := client.pool.GetPublishedPort(portOpen.PortNumber)
			if publishedPort == nil {
				client.ResponsePortOpen(portOpen, errPortNotPublished)
				client.Log().Info("Port was not published port = %v", portOpen.PortNumber)
				return
			}

The server=… suffix comes from the logger’s server field on each Client (rpc/client.go), so seeing the same message on many hosts means each RPC connection hit this path—not that “the network forgot” the port on those servers.

Why it can be a red herring for the GitHub issue

• If the user truly published 8081, this log should not mean “8081 was never configured.” It only means the in-memory map used for lookups was empty for that port at handle time.
• The failover / “routes not retried” problem after Client connection closed unexpectedly: EOF is a separate concern (connection lifecycle, client manager, retries). This log does not prove a publishing bug on the server side; it proves the local client rejected the open because it had no matching entry in DataPool.publishedPorts.

Real bug found: diode publish ordering

In publishHandler, app.Start() ran before the full PublishedPorts map was built and before SetPublishedPorts. So RPC clients could go live and receive portopen while publishedPorts was still empty—producing spurious “Port was not published” lines (possibly intermittent, depending on timing). That matches “not always output” and multi-server spam.

What we did

A fix is in PR #278 (cursor/fix-publish-port-race-233b): after the port map is complete (including optional static HTTP), call SetPublishedPorts before app.Start() so inbound portopen never runs against an uninitialized pool.

Bottom line

• Not a red herring as a log: it reflected a real GetPublishedPort == nil rejection.
• Often a red herring for explaining the EOF/failover bug: the underlying issue for publish users was likely client-side registration order, not “8081 unpublished on the network.” Failover behavior still needs the separate investigation Hans described (issue Disconnected node connection kills ability to connect to edge device #268).