use redis as persistent queue in gearman (v1.1.16)

[soooldier]

If somebody use redis as persistent queue in gearman，be aware of configuration hash_max_ziplist_value about redis, otherwise you may get some errors when you restart gearmand.

Begin stack trace, frames found: 17
?0  0x7f6a6cdaa9b3 in /usr/local/gearman/sbin/gearmand(+0x1b9b3) [0x7f6a6cdaa9b3]
?1  0x7f6a6cd99bd7 in /usr/local/gearman/sbin/gearmand(+0xabd7) [0x7f6a6cd99bd7]
?2  0x7f6a6b5007e0 in /lib64/libpthread.so.0(+0xf7e0) [0x7f6a6b5007e0]
#3  0x7f6a6a3965e5 in gsignal at /lib64/libc.so.6
#4  0x7f6a6a397dc5 in abort at /lib64/libc.so.6
---> demangled: __gnu_cxx::__verbose_terminate_handler() -> __gnu_cxx::__verbose_terminate_handler()
#5  0x7f6a6ac50a7d in __gnu_cxx::__verbose_terminate_handler() at /usr/lib64/libstdc++.so.6
?6  0x7f6a6ac4ebd6 in /usr/lib64/libstdc++.so.6(+0xbcbd6) [0x7f6a6ac4ebd6]
?7  0x7f6a6ac4ec03 in /usr/lib64/libstdc++.so.6(+0xbcc03) [0x7f6a6ac4ec03]
?8  0x7f6a6ac4ed22 in /usr/lib64/libstdc++.so.6(+0xbcd22) [0x7f6a6ac4ed22]
---> demangled: std::__throw_invalid_argument(char const*) -> std::__throw_invalid_argument(char const*)
#9  0x7f6a6abf3f57 in std::__throw_invalid_argument(char const*) at /usr/lib64/libstdc++.so.6
?10  0x7f6a6cdb817e in /usr/local/gearman/sbin/gearmand(+0x2917e) [0x7f6a6cdb817e]
?11  0x7f6a6cdb835c in /usr/local/gearman/sbin/gearmand(+0x2935c) [0x7f6a6cdb835c]
?12  0x7f6a6cdb5f3c in /usr/local/gearman/sbin/gearmand(+0x26f3c) [0x7f6a6cdb5f3c]
?13  0x7f6a6cdad4e7 in /usr/local/gearman/sbin/gearmand(+0x1e4e7) [0x7f6a6cdad4e7]
?14  0x7f6a6cda54ec in /usr/local/gearman/sbin/gearmand(+0x164ec) [0x7f6a6cda54ec]
#15  0x7f6a6a382d1d in __libc_start_main at /lib64/libc.so.6
?16  0x7f6a6cd99a05 in /usr/local/gearman/sbin/gearmand(+0xaa05) [0x7f6a6cd99a05]
terminate called after throwing an instance of 'std::invalid_argument'
  what():  stoi

[p-alik]

thanks for reporting @soooldier. Could you provide little more information to improve the issue:

• redis version?
• redis setting before gearmand crash
• changes of redis settings

[soooldier]

@p-alik Any version of redis may cause this issue. i think the reason is gearmand using redis data structure hash set for persistent queue, while the structure has no guarantee of the order about these keys in hash set.

In line queue.cc#L90 ,the field priority is setted after the data; and in line queue.cc#L158, the same order. but in fact, we could get a wrong order when using hgetall.

we could breaking the rules by increacing the value of hash-max-ziplist-value (default 64 bytes). see more about hash-max-ziplist-value

sorry for my poor english ... :(

[p-alik]

@soooldier, it works fine in my environment.

• Ubuntu 16.04.2 LTS
• Linux 4.4.0-83-generic
• libhiredis-dev 0.13.3-2
• redis-server 2:3.0.6-1

I tested it as follow:

• run gearmand --queue-type redis --verbose DEBUG
• add some background jobs (redis.conf has default hash-max-ziplist-value 64)
• set hash-max-ziplist-value 128
• restart redis
• restart gearmand

By changing hash-max-ziplist-value` from 128 back to 64 no issue to me.
Maybe it's your environment matter?

[soooldier]

@p-alik does the job data size more than 64 bytes？I did it as follow:

• run gearmand --queue-type redis --verbose DEBUG

• start redis (redis.conf has default hash-max-ziplist-value 64)

• add 1000 background jobs (the job data size more than 64 bytes)

• check the data type in redis (OBJECT ENCODING _gear_function-name-object-id),it output as hashtable.

• restart gearmand and the error occurs

[p-alik]

Thank you @soooldier.
I could reproduce the issue with a single background job only.
It occurs for jobs with payload >= 64 without any changes of redis configuration.

• run gearmand --queue-type redis --verbose DEBUG
• add a job:

STR=`cat /dev/urandom | tr -dc a-zA-Z0-9 | fold -w 64 | head -n 1`
echo $STR | gearman -b -f foo

• stop gearmand
• start gearmand ends in exception.

[SpamapS]

We just merged a fix to master, so if you can build from master and confirm, that would be great. Otherwise I expect a release with the fix soon-ish.

[soooldier]

@SpamapS that's great ! thanks very much.