Add new BID_VALUATION intent and additional payload for intermediate transformation

[arpadmiklos-ttd]

This PR adds a new BID_VALUATION intent as well as the ability to pass object IDs pertaining to the bidding process into an ARTF-compliant container, then associate result values with them that are then used by the orchestrator in further stages of bid request evaluation.

Broadly speaking, the scenario these additions cover is the capability to represent and operate on an intermediate state (for a DSP) between the bid request and the bid response, in particular when using containers that want to influence the valuation of an impression, but will not be in a position to provide an authoritative bid price (using BID_SHADE).

In addition to passing object IDs, the proposed new AdjustBidValuation message can be extended in the future with embedding additional information traveling in the bid request envelope as necessary.

[adrian-vaca-humanes-wt]

With this we can one can take the deals or segments from the RTB section and interpret the host wants those scored.

For discussion:

What should the semantic be if the caller/host passes imp[].pmp.deals[] or user.data[].segment[] as empty lists? Invalid request, or would the expectation be for it to still return something? Because for example, if the container receives an embedding it could issue ACTIVATE_DEAL and ACTIVATE_SEGMENT and at the same time a ADJUST_BID_VALUATION for those. While applicable_intents could be a signal of the type of types of IDs expected back in this scenario, I wonder if intents should support inputs other than what one may shoehorn as an extension, for example, in this empty list scenario where the container is supposed to give it its best shot, a topK input could be useful. And yes, this could also just be default configuration of the container or come through an extension.

[arpadmiklos-ttd]

These IDs are representations of an advertiser's entities on the buyer side, and the purpose of this BID_VALUATION addition is to allow third parties to contribute to the advertiser asset (say, campaign or ad group) valuation decision made by the DSP against the incoming bid request. As such, these do not correlate to individual deals or contextual segments or any other field in the bid request.

IMO if we want to add functionality to attribute relative value/worth to deals or segments, that should be covered by another intent separate from this PR, and the discussion point you raised would be germane to that.

On a closely related note, I think the standard could be improved in terms of either explicitly associating intents and payloads more unequivocally, or at least clarifying expectations. For example, a container responding with mutations using the ACTIVATE_SEGMENTS intent could well supply IDsPayload as sets of flat segment IDs, or DataPayload instances that have segments plus and additional name and ID or even content IDs, and I'm not sure what the expectation from an orchestrator is in this case.

[adrian-vaca-humanes-wt]

I got ahead of myself by thinking of these object IDs as references to objects in a partial OpenRTB payload. From there, I moved toward semantics that could be implemented today using deal IDs and segment IDs as examples. More generally, though, I was exploring whether this could be generalized regardless of ID type—particularly the semantics of an empty list, where the orchestrator could treat the container as a recommender. But you may be right that recommendation or candidate discovery belongs in a separate intent specification altogether.

Setting that discussion aside, do you envision distinct ID-type-specific valuation containers, where each container can score only one type of ID? If so, would there be an implicit, out-of-band agreement about what the opaque IDs represent for that intent?

[arpadmiklos-ttd]

The current 1.0 version of this standard doesn't specify such ID types even though IDsPayload is already used in different contexts, that's why I didn't add ID types here. When someone works with us to implement a container that supports ADJUST_BID_VALUATION, they will know what to expect based on out-of-band conventions (not part of this spec).

Indeed we probably should add an ID type field, or create different variants of the IDsPayload message to ensure these payload IDs align with the expected type and that the container can actually work with them. Unfortunately, both would be a breaking change.

[adrian-vaca-humanes-wt]

To prevent mismatches at population time (because serialization will work regardless), I am thinking this should be a repeated object instead, that additionally has an id_type e.g. {deal, segment, ...} and value_type {multiplier, score} so that the machine gets to know about the semantic, rather than being implicit.

[arpadmiklos-ttd]

Indeed it would be more elegant and I'm not against a new repeated object that ensures alignment, which was my original idea as well.

My angle has been that ARTF is already slow to work with (our infra at TTD currently handles 20M+ qps) vs an optimized custom interface, and parallel running arrays are less error prone when encapsulated in a payload object, so this may be acceptable from a clarity point of view.

If you feel strongly about it, I'm happy to add a new object.

[adrian-vaca-humanes-wt]

That makes sense. Just curious, what would you do if ever there is a mismatch in elements? Ignore that part of the response completely?

Also, what about the point of qualifying the type for the id at score at response time?

[arpadmiklos-ttd]

Element mismatch indicates message corruption in this setup, so we should drop the requested mutation altogether and update logs/metrics.

The ID type in the mutation response must correspond to that of the IDs in the request, so I wouldn't repeat this information in the mutation.

[adrian-vaca-humanes-wt]

Would you add an additional Lifecycle stage and additional Originator Type?

[arpadmiklos-ttd]

Would you add an additional Lifecycle stage and additional Originator Type?

@adrian-vaca-humanes-wt why do you think it's necessary? We would introduce an arbitrary new lifecycle stage compared to existing programmatic auction definitions, and the originator is clearly TYPE_DSP in this case as it's part of the internal bid evaluation process after receiving a bid request and prior to producing a bid response, so I'm not even sure what else one would add.

[adrian-vaca-humanes-wt]

Would you add an additional Lifecycle stage and additional Originator Type?

@adrian-vaca-humanes-wt why do you think it's necessary? We would introduce an arbitrary new lifecycle stage compared to existing programmatic auction definitions, and the originator is clearly TYPE_DSP in this case as it's part of the internal bid evaluation process after receiving a bid request and prior to producing a bid response, so I'm not even sure what else one would add.

Forget about the Originator, I thought it was part of the response while I wrote that, but alas I remembered wrongly.

As for the lifecycle stage, not needed at all. I had the incomplete idea that for a machine reasoning about where it's at, a more granular state machine could be useful, but admittedly it is also implicit in the intent.