External Storage Integration: Lazy resolving references, general refactoring by cconstable · Pull Request #3016 · temporalio/sdk-java

cconstable · 2026-08-18T20:24:39Z

What was changed

Renames ExternalStorageMessageTransformer to ExternalStorage.
Added a ExternalStorageDataConverter: an internal data converter used consolidating extstore usage.

Why?

These changes support the following PRs:

Workflow worker integration External Storage Integration: WorkflowWorker, replay, history #3017
Nexus worker integration External Storage Integration: NexusWorker #3018
Activity worker and client integration External Storage Integration: Activity worker, client #3020
E2E extstore tests

Checklist

Added tests

cconstable · 2026-08-19T16:01:02Z

    private boolean allowActivityHeartbeatDuringShutdown;
    private String workerControlTaskQueue;
    private PreferredVersionProvider preferredVersionProvider;
+    private @Nullable ExternalStorage externalStorage;


This doesn't have any callers in this PR but three of the PRs that build off this will use it. Included it here since those PRs are being up together.

jmaeagle99 · 2026-08-20T17:33:03Z

    }

+    if (ExternalStorageReferences.isReference(payload)) {
+      throw new ExternalStorageNotConfiguredException();


IIUC, this looks to be a DataConverter, which means it runs within the workflow code context. This means that this exception is handleable by user code. I think we need to move this to somewhere before the workflow code executes so we can fail the workflow task without allowing the user code to compensate.

The context here is that if we missed integrating external storage into some piece of the SDK or a new feature was added that forgot to integrate external storage, this would catch it and throw a not configured error. It's purely a defensive fallback to make sure that SDK bugs don't appear as "your payloads can't be decoded/transformed" errors.

I do think ExternalStorageNotConfiguredException is probably the wrong error to be throwing here because it's likely that it was configured right and the sdk just isn't using external storage. Should this be a "some feature you are using has not integrated external storage. please file a bug report" type thing?

I added a new ExternalStorageUnhandledReferenceException for when a reference payload makes it to fromPayload without being retrieved. This is a guard against SDK features that fail or fail to correctly integrate external storage. Normal misconfiguration errors should be caught at a level above fromPayload and are done so in the PRs that build off this.

jmaeagle99 · 2026-08-20T17:34:14Z

+/**
+ * A {@link DataConverter} that resolves external storage reference payloads before deserialization.
+ */
+public final class ExternalStorageResolvingDataConverter implements DataConverter {


Same concern in https://github.com/temporalio/sdk-java/pull/3016/changes#r3823884199

Removed this entirely in favor of just doing the lazy retrieving in the few spots we needed explicitly. After chatting this felt like too much abstraction.

jmaeagle99 · 2026-08-20T17:42:01Z

@@ -84,10 +98,21 @@ public Builder setPayloadSizeThreshold(int payloadSizeThreshold) {
      return this;
    }

+    /**
+     * Maximum number of payload lists visited concurrently while offloading or restoring the
+     * payloads of a single message. Must be at least 1. Defaults to 3.


Not sure a user is going to understand what is meant by "a single message". And "a single message" isn't quite the right scope from a implementation perspective. Maybe should describe this in terms of client operations and worker tasks.

I think this will get cleaned up in the concurrency pass. I believe (technically) a single message is correct but I agree it's not a helpful way to describe it.

jmaeagle99 · 2026-08-20T17:52:31Z

+    }
+  }
+
+  <T extends Message> CompletableFuture<T> store(


Are there legitimate places where we need to visit on the fully constructed message instead of the build (the next overload)? I presume that the caller already created a builder, constructed the message, then this would effective recreate another builder, and reconstruct the message again. Might be perf issues. I would check to see if we can drop the message overloads and only use the builder overloads to force callers into the better performing algorithm.

jmaeagle99 · 2026-08-20T17:55:10Z

+    }
+  }
+
+  <T extends Message> CompletableFuture<T> store(


This will not work for workflow task completions because the nested commands need to change the the context when they are encountered. Having an outer visitor doing that determination and then calling this method is probably okay.

Correct but workflow task completions are routed through a different store overload that includes the MessageVisitor so that the context can be changed (it's the one earlier in the file).

public <T extends Message> T storeBlocking( T message, @Nullable StorageDriverTargetInfo target, @Nullable MessageVisitor<StorageDriverTargetInfo> targetVisitor) { ... }

Regardless, it is confusing that there are so many overloads. I'll take a pass at consolidating them. I think we could just have one or two public ones. Some of the overloads above organically grew during the refactors and can be removed.

jmaeagle99 · 2026-08-21T16:10:22Z

+/**
+ * A {@link DataConverter} that resolves external storage reference payloads before deserialization.
+ */
+public final class ExternalStorageResolvingDataConverter implements DataConverter {


Maybe add a comment that this is used in non-workflow contexts.

…former to ExternalStorage, create a lazy extstore resolving data converter.

cconstable · 2026-08-21T21:35:55Z

Pushed a few updates:

I realized the names were a little off between this SDK and the Python/TS ones so I renamed ExternalStorageOptions to ExternalStorage and the previous ExternalStorage to ExternalStorageRunner (this feels a bit TS-flavored)
I moved ExternalStorage config on to the DataConverter and added a default null value for it. Matches the other SDKs.
Moved the ExternalStorageGenericWorkflowClient to the activity/client PR https://github.com/temporalio/sdk-java/pull/3020/changes. It's still awkward that some things in there eagerly fetch payloads and others just pass through but I'm not sure of a good path forward there.
I removed the lazy data converter decorator in favor of just doing the lazy external storage retrieving for describe and listWorkflowExecutions inline. Seemed like the decorator was more abstraction than necessary.

…e to match other sdks.

…h to workers. we've got the dataconverter already so we can just derive it where its needed.

…for configuring external storage.

…ce payload makes it to fromPayload without being retrieved. This is a guard against SDK features that fail or fail to correctly integrate external storage. Normal misconfiguration errors should be caught at a level above fromPayload.

jmaeagle99 · 2026-08-25T19:50:17Z

+   */
+  @Experimental
+  @Nullable
+  default ExternalStorage getExternalStorage() {


External storage should not be on DataConverter but instead on WorkflowClientOptions. The IDataConverter instance is accessible from within the workflow context, where we do not want external storage be accessible, let alone executable.

We already have externalStorage on the data converter in both the Python and TS implementations. Also WorkflowClientOptions wouldn't be sufficient. I think we would need to independently configure ActivityClientOptions, NexusClientOptions, etc. I also don't think there is anything preventing a user from just doing S3Client().getObject(...) inside a "workflow".

After chatting: will move to WorkflowClientOptions, ActivityClientOptions, NexusClientOptions, ScheduleClientOptions.

jmaeagle99 · 2026-08-25T19:52:26Z

      return (T) new RawValue(payload);
    }

+    if (ExternalStorageReferences.isReference(payload)) {


I'm not really convinced this is necessary. If we forget to plumb through retrieval somewhere, then it should likely fail at deserialization time explaining that the ExternalStorageReference message couldn't be deserialized into the target type. I also don't think we should give customers something that they can handle and attempt to compensate for in their executions.

will chat with @Quinn-With-Two-Ns or @maciejdudko

Potential directions:

wrap the final data converter with a "reference payload checking" data converter. ensure it fails the workflow task.

remove check entirely

…flow client options

… make sure users don't see a misleading error is nice it could also cause some nondeterminism if we fix the error and came with its own set of problems. maybe in the future we can find a better way to catch missing external storage integration in a better way.

jmaeagle99

Most are nits except the ExternalStorageDataConverter decorator should implement all methods and some more test coverage. I also pointed out some naming inconsistencies, but please do take a pass over them.

jmaeagle99 · 2026-08-27T22:25:36Z

+     * <p>Defaults to null.
+     */
+    @Experimental
+    public Builder setExternalStorage(@Nullable ExternalStorage externalStorage) {


nit: Is there a way we could possibly mark this hidden from API / doc generation for now and adjust the comment to effectively say this is no-op at this time? I know we have the other PRs as follow ups to enable it, but just in case we need to pause and release Java in between. Obviously remove all of that when it does actually get attached in some usable manner.

jmaeagle99 · 2026-08-27T22:32:24Z

 import javax.annotation.Nullable;

-final class ExternalStorageReferences {
+public final class ExternalStorageReferences {


This can be non-public again, correct?

jmaeagle99 · 2026-08-27T22:34:24Z

    }

+    /**
+     * External storage configuration uses to store/retrieve large payloads.


Suggested change

* External storage configuration uses to store/retrieve large payloads.

* External storage configuration used to store/retrieve large payloads.

jmaeagle99 · 2026-08-27T22:47:07Z

+ * <p>This This is an internal class that is not exposed to users or workflow code. The intent is to
+ * use this data converter to consolidate extstore usage within the SDK.
+ */
+public final class ExternalStorageDataConverter implements DataConverter {


Need to implement Object[] fromPayloads(Optional<Payloads> content, Class<?>[] parameterTypes, Type[] genericParameterTypes).

jmaeagle99 · 2026-08-27T22:49:36Z

+
+  private Payloads store(Payloads payloads) {
+    Payloads.Builder builder = payloads.toBuilder();
+    externalStorage.store(builder, storageTarget, null, CancellationToken.none());


nit: I think this needs to be lifted to be an instance field so that the owner of the data converter can cancel in flight operations, or maybe provided via something like a thread local so each call site can cancel. Then again, not sure if there is anything on any of the clients that would enable cancellation. Take this as a todo rather than fixing unless others object.

jmaeagle99 · 2026-08-27T22:57:34Z

  private final boolean allowActivityHeartbeatDuringShutdown;
  private final String workerControlTaskQueue;
  private final PreferredVersionProvider preferredVersionProvider;
+  private final @Nullable ExternalStorageRunner externalStorage;


nit: storageRunner or externalStorageRunner

jmaeagle99 · 2026-08-27T23:00:01Z

  HeartbeatManager getHeartbeatManager();
+
+  @Nullable
+  ExternalStorageRunner getExternalStorage();


getStorageRunner or getExternalStorageRunner

jmaeagle99 · 2026-08-27T23:04:24Z

+import java.util.concurrent.CompletableFuture;
+import org.junit.Test;
+
+public class ExternalStorageDataConverterTest {


When you do implement the array form of fromPayloads, please add a test.

jmaeagle99 · 2026-08-27T23:05:39Z

+import java.util.concurrent.CompletableFuture;
+import org.junit.Test;
+
+public class ExternalStorageDataConverterTest {


This we could add a test that wraps CodecDataConverter with a basic "encrypting/decrypting" codec to validate that drivers receive "encrypted" payloads?

jmaeagle99 · 2026-08-27T23:07:24Z

+  }
+
+  @Test
+  public void throwIfContainsReferenceThrowsOnReference() throws Exception {


While this is a worthwhile test, I think there should be a test that does the same thing but for a non-Payload/Payloads message type. Maybe a WFT completion with a start activity task command that has a storage reference input.

cconstable force-pushed the extstore/foundation branch from 3fe47fb to 831c284 Compare August 18, 2026 21:03

cconstable commented Aug 19, 2026

View reviewed changes

cconstable force-pushed the extstore/foundation branch from 831c284 to 460bfbf Compare August 19, 2026 17:40

cconstable changed the title ~~refactor(extstore): general extstore refactoring. rename MessageTransformer to ExternalStorage, create a lazy extstore resolving data converter.~~ External Storage refactoring for workflow, activity, client, and nexus integration Aug 19, 2026

cconstable changed the title ~~External Storage refactoring for workflow, activity, client, and nexus integration~~ External Storage Integration: general refactoring Aug 19, 2026

cconstable changed the title ~~External Storage Integration: general refactoring~~ External Storage Integration: Lazy resolving references, general refactoring Aug 19, 2026

cconstable marked this pull request as ready for review August 19, 2026 18:57

cconstable requested a review from a team as a code owner August 19, 2026 18:57

jmaeagle99 requested changes Aug 20, 2026

View reviewed changes

jmaeagle99 reviewed Aug 21, 2026

View reviewed changes

refactor(extstore): general extstore refactoring. rename MessageTrans…

eee31a3

…former to ExternalStorage, create a lazy extstore resolving data converter.

cconstable force-pushed the extstore/foundation branch from 460bfbf to b3804da Compare August 21, 2026 20:35

refactor(extstore): move external storage to data converter and renam…

ca09b50

…e to match other sdks.

cconstable force-pushed the extstore/foundation branch from b3804da to ca09b50 Compare August 24, 2026 01:23

cconstable added 4 commits August 24, 2026 15:03

refactor(extstore): stop threading the external storage runner throug…

af36622

…h to workers. we've got the dataconverter already so we can just derive it where its needed.

Update ExternalStorageNotConfiguredException to refer to correct API …

5ebc8ee

…for configuring external storage.

lint: fix whitespace formatting issue

66f31c3

jmaeagle99 reviewed Aug 25, 2026

View reviewed changes

cconstable added 6 commits August 26, 2026 15:10

address PR feedback

d51e348

refactor(extstore): move external storage from data converter to work…

36224c3

…flow client options

remove unused tests and code

1261294

small fixes

024ec40

move ExternalStorageDataConverter to foundation PR.

8df7ce1

jmaeagle99 approved these changes Aug 27, 2026

View reviewed changes

	* External storage configuration uses to store/retrieve large payloads.
	* External storage configuration used to store/retrieve large payloads.

Uh oh!

Conversation

cconstable commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was changed

Why?

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cconstable commented Aug 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jmaeagle99 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

cconstable commented Aug 18, 2026 •

edited

Loading