limited support of atomic transaction in redis cluster. #4
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
multi-exec around consecutive redis commands that are for the same
slot. Redis cluster does not support multi-exec for different slots,
even if the slots are on the same shard.
This feature provides limited support (*1) of atomic transaction using
multi-exec on redis cluster.
Step1: add optional arg use_multi=False to pipeline.execute(), so the
default behavior is what it used to be. ie. this feature is off by
default.
*1: limited support: only for consecutive commands in pipeline for the
same slot. eg. for commands
incr a{1} #1
decr b{1} #2
incr a{2} #3
decr b{2} #4
incr a{3} Grokzen#5
Since #1,#2 are for the same slot, we will add multi-exec around them
(multi before #1, and exec after #2). #3,#4 are in their own one
slot, but different from that of #1,#2, so another set of multi-exec.
Grokzen#5 is in its own slot, no multi-exec added.
Thus the commands sent to the cluster are:
multi
incr a{1} #1
decr b{1} #2
exec
multi
incr a{2} #3
decr b{2} #4
exec
incr a{3} Grokzen#5
The result of the added multi-exec are stripped from the response, so
the client will see no change in response format.
commands are for the same slot.
- scan thru all the commands for one node. Add multi-exec around
consecutive commands for the same slot.
- add NodeCommands.iRsp[] to record the indices of the expected
responses. eg. for [1], iRsp will be
[ None, "0", "1", [ 0, 1 ], None, "2", "3", [ 2, 3 ], 4 ]
multi #1 #2 exec multi #3 #4 exec Grokzen#5
- the rsp to MULTI is "OK", so we want to discard that rsp.
If the index is None, we will discard the rsp.
- the rsp for queued commands is "QUEUED" (good case) or error.
If good case, we want to discard the placeholder rsp "QUEUED".
An index with type str indicates we should discard good case rsp.
In case of error, we use that index (str->int) to record error rsp.
- the rsp for exec is a list of rsp for queued commands (good
case) or an error.
If good case, we have the indices of the commands corresponding to
the rsp.
If error, we recorded the errors already, so we can discard this
error (error of exec says "see previous errors" anyway).
- A plain int index is for commands outside multi-exec, eg. Grokzen#5.
[1]
multi
incr a{1} #1
decr b{1} #2
exec
multi
incr a{2} #3
decr b{2} #4
exec
incr a{3} Grokzen#5
callback here, because we call
self.parse_response( connection, '_' )
with no specific commands ('_' underscore is the placeholder command
with no callback). (copied from
redis.client.Pipeline._execute_transaction)
…tr indices). Also ignore the error of 'exec'; it says "see previous errors."
if use_multi == True, it means we want the commands for the same slot to be atomic. Automatic retry spoils the atomic semantic. So no automatic retry and let the caller handle the retry, to ensure atomic semantic.
a harmless command to force fetch of new node-slot table.
timliim
requested review from
hendrikdemolder,
echaz,
javkhlan-g and
letshriramin
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The plan is to update to the latest redis client and see if this (atomic transaction
in redis cluster) is already supported.
This PR is more for the record, so the local changes are not lost.