test: flaky app-tap/tarantoolctl.test.lua test

[avtikhon]

Tarantool version:
Tarantool 2.5.0-87-g943b507a3
Target: Linux-x86_64-RelWithDebInfo
Build options: cmake . -DCMAKE_INSTALL_PREFIX=/usr/local -DENABLE_BACKTRACE=ON
Compiler: /usr/bin/clang-8 /usr/bin/clang++-8
C_FLAGS: -Wno-unknown-pragmas -fexceptions -funwind-tables -fno-omit-frame-pointer -fno-stack-protector -fno-common -msse2 -fsanitize=address -fsanitize-blacklist=/builds/5Q-f3C3h/0/tarantool/tarantool/asan/asan.supp -std=c11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Werror
CXX_FLAGS: -Wno-unknown-pragmas -fexceptions -funwind-tables -fno-omit-frame-pointer -fno-stack-protector -fno-common -msse2 -fsanitize=address -fsanitize-blacklist=/builds/5Q-f3C3h/0/tarantool/tarantool/asan/asan.supp -std=c++11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Werror

OS version:
Debian 10 + ASAN + Clang8

Bug description:
https://gitlab.com/tarantool/tarantool/-/jobs/574913068
https://gitlab.com/tarantool/tarantool/-/jobs/574429004

 [023] Last 15 lines of Tarantool Log file [Instance "app_server"][/builds/5Q-f3C3h/0/tarantool/tarantool/test/var/023_app-tap/tarantoolctl.test.lua.tarantool.log]:
 [023] Netbox console API is deprecated since 1.10, please use require('console').connect() instead
 [023] Netbox text protocol support is deprecated since 1.10, please use require('console').connect() instead
 [023] Netbox console API is deprecated since 1.10, please use require('console').connect() instead
 [023] Netbox text protocol support is deprecated since 1.10, please use require('console').connect() instead
 [023] Netbox console API is deprecated since 1.10, please use require('console').connect() instead
 [023] Netbox text protocol support is deprecated since 1.10, please use require('console').connect() instead
 [023] Netbox console API is deprecated since 1.10, please use require('console').connect() instead
 [023] Netbox console API is deprecated since 1.10, please use require('console').connect() instead
 [023] Netbox text protocol support is deprecated since 1.10, please use require('console').connect() instead

Steps to reproduce:
cd test && ASAN=ON
LSAN_OPTIONS=suppressions=/builds/5Q-f3C3h/0/tarantool/tarantool/asan/lsan.supp
ASAN_OPTIONS=heap_profile=0:unmap_shadow_on_exit=1:detect_invalid_pointer_pairs=1:symbolize=1:detect_leaks=1:dump_instruction_bytes=1:print_suppressions=0
./test-run.py --force

Same fail at:

1. https://gitlab.com/tarantool/tarantool/-/jobs/856670673#L3152
   artifacts.zip

results file checksum: 1eea8234b18bf107271b2a973f32a10e

2. https://gitlab.com/tarantool/tarantool/-/jobs/844055813#L3176
   artifacts.zip

results file checksum: 5e9064e8640a5106967c4737523364ee

Optional (but very desirable):

• coredump
• backtrace
• netstat

[Totktonada]

Related output from the first link:

[023] ok - basic test for bad script
[023] # check answers in case of call
[023] 1..5
[023] ok - check 'start' command status for 'good_script'
[023] not ok - check 'eval' command status for 'good_script bad_script.lua'
<...>
[023] expected: 3
[023] got: 2
[023] ...
[023] 2 Control socket exists, but Tarantool doesn't listen on it

It seems there is some time period after successul tarantoolctl start until tarantool will actually listen on a socket. Maybe we should spin for a while (with trying to execute a command) until tarantool will actually ready to process commands after tarantooctl start.

Origin of the error message: https://github.com/tarantool/tarantool/blob/194d40dbc55218aff9172e6e811d448293bc2896/extra/dist/tarantoolctl.in#L818-L822

How tarantool-java testing harness overcomes this: https://github.com/tarantool/tarantool-java/blob/9936054e82f0ef109e5637ec64f6d0424c61e799/src/test/java/org/tarantool/TarantoolControl.java#L169-L208

[Totktonada]

NB: It does not look related to OS / build options.

[ligurio]

bug can be easily reproduced with following command-line: ../../test/test-run.py --builddir=/home/s.bronnikov/tarantool/build --vardir=/home/s.bronnikov/tarantool/build/test/var -j 500 $(yes app-tap/tarantoolctl.test.lua | head -n 150) with interesting note: hw with > 4 CPU cores is required. I failed to reproduce a bug on virtual machine with 4 CPU cores and succeed on HW with 80 CPU cores.

[128] 
[128]     not ok - check 'eval' command status for 'good_script bad_script.lua'
[128]       ---
[128]       filename: /home/s.bronnikov/tarantool/build/tarantool/test/app-tap/tarantoolctl.test.lua
[128]       trace:
[128]       - line: 136
[128]         source: '@/home/s.bronnikov/tarantool/build/tarantool/test/app-tap/tarantoolctl.test.lua'
[128]         filename: /home/s.bronnikov/tarantool/build/tarantool/test/app-tap/tarantoolctl.test.lua
[128]   

Tarantool 2.6.0-39-ga102123