Adds settings-only option to use Outbox in SendsAtomicWithReceiveMode

[SzymonPobiega]

• Option requires explicit opt-in via settings

  • This will be done by proper APIs in a downstream package

• In SendsAtomicWithReceiveMode mode:

  • Outgoing operations are enlisted in the receive transaction
  • Outbox record is not marked as Dispatched immediately after dispatching the outgoing operations (because the receive transaction can still roll back so they are not really dispatched)
  • Outbox batch contains a control message sent to self to mark the record as Dispatched
  • There is a behavior in the beginning of the pipeline to process it

• Added 4 tests to verify the behavior

  • Happy path (that a message is delivered)

  • That when a failure happens just before ACKing the received message, the outgoing messages are not released (and are indeed part of the receive transaction)

  • That after such a failure ☝️ a retry pushes out the messages

  • That the self-sent control message indeed marks the record as Dispatched, preventing duplicate amplification

  • Added strict sequential mode to AT transport in order to make sure the tests verify what they are supposed to. This mode consists of LimitMessageProcessingConcurrencyTo(1); and b.ConfigureTransport<AcceptanceTestingTransport>().FifoMode = true;

[danielmarbach]

I have a few concerns that I think we have to discuss

• How does that affect the transactional session? The transactional session already uses a similar dispatch message approach and I wonder if those cases should be aligned or uniformed somehow in case we are doing this change. It might allow us to make tradeoffs in the transactional session to directly dispatch the transport messages but use the mechanism here to set it as dispatched or some other fancy clever combination I cannot come up right now by only spending a few minutes looking at the code
• Another thing that I find concerning is that today I think we have designed and implemented quite a few assumptions around the interaction with the persister that also the transactional session leverages. Some of the persisters could technically rely on the context being passed to Get also being passed to SetAsDispatched to achieve, for example, concurrency control by storing information in the context bag. This information would get lost and can lead to all sorts of issues.
• If we further think about the transactional session use cases, we might also consider the local processing vs remote processing of the "control message" and then maybe the hard coded assumption of SendLocal might not be a good fit.

[danielmarbach]

I think for consistency it should be using the outbox seam and pass context.Extensions as the bag.

[SzymonPobiega]

We can't use IOutboxSeam here because for the SAWR mode it _noopstheSetAsDispatched. My idea for introducing IOutboxSeamwas that theTransportReceiveToBlaBlaBehaviorhas no extensibility points to tweak what is executed andIOutboxSeam` provides that.

So IOutboxSeam is a contract between that behavior and the Outbox feature. Within the Outbox feature (like in SetAsDispatched behavior), we interact with IOutboxStore directly.

[SzymonPobiega]

How does that affect the transactional session? The transactional session already uses a similar dispatch message approach and I wonder if those cases should be aligned or uniformed somehow in case we are doing this change. It might allow us to make tradeoffs in the transactional session to directly dispatch the transport messages but use the mechanism here to set it as dispatched or some other fancy clever combination I cannot come up right now by only spending a few minutes looking at the code

My mental model of the transactional session is based on the split between sync part (while handling the web request) and async part (when the control message arrives), connected by that control message. If I understand correctly the current behavior is that the BeginTransaction, Store and CommitTransaction operations happen in the sync part and the dispatching and SetAsDispatched in the async one via the TransportReceiveBlaBlaConnector.

This is a good question but I don't see for now much opportunity for alignment. The significant difference is that for the SendAtomicWithReceive mode to work, the control message needs to be part of the outbox while for TS it cannot. For TS it drives the outbox dispatch while for SAWR it drives the outbox cleanup.

I think that the we can look at this space like this. In order to implement atomic send and update the sends need to be stored and there needs to be a specific sequence of steps executed: Get, Store, Dispatch, SetAsDispatched. TS and SAWR outbox are two other (besides the classic outbox) variations on how these steps get connected together to guarantee that they are executed reliably in that order,

Another thing that I find concerning is that today I think we have designed and implemented quite a few assumptions around the interaction with the persister that also the transactional session leverages. Some of the persisters could technically rely on the context being passed to Get also being passed to SetAsDispatched to achieve, for example, concurrency control by storing information in the context bag. This information would get lost and can lead to all sorts of issues.

Compared to the SendAtomicWithReceive mode, the TS does Get and SetAsDispatched (because the actual dispatching happens there) while SAWR only does SetAsDispatched. I agree that there might potentially be some assumptions that Get is always called first but the issue with that would be mitigated by the fact that SAWR requires settings-based opt in, meaning that the persistence/transport combination that supports it would be verified to not have that Get-SetAsDispatched dependency via automated tests.

If we further think about the transactional session use cases, we might also consider the local processing vs remote processing of the "control message" and then maybe the hard coded assumption of SendLocal might not be a good fit.

I think that the local processing does not apply to SAWR outbox because by definition it requires an incoming message. This means that we can safely assume that the control message that cleans the outbox can be sent to the local queue.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.