test-run sometimes doesn't kill server started by test

[locker]

If a test starts a server, it should stop it on completion. However, if it doesn't, the server should still be stopped by test-run or luatest. Normally, this is what happens, but sometimes, the test server somehow survives.

How to reproduce:

1. Revert test: stop server started by vinyl-luatest/update_optimize test tarantool#7359.
   The PR added server stop to vinyl-luatest/update_optimize_test.lua.
2. Run the test in a loop:

   yes vinyl-luatest/update_optimize_test.lua | head -n 30 | xargs ./test-run.py --builddir ../build/debug
   

3. Check if there Tarantool instances left after the test:

   ps ax | grep tarantool
   

(You'll probably need to try a few times before you catch it.)

A stray instance is running normally: it can be connected to or killed with SIGTERM.

Output

vlad@esperanza:~/src/tarantool/tarantool/test$ ps ax | grep tarantool
 240930 pts/3    S+     0:00 grep --color=auto tarantool
vlad@esperanza:~/src/tarantool/tarantool/test$ yes vinyl-luatest/update_optimize_test.lua | head -n 30 | xargs ./test-run.py --builddir ../build/debug
Started ./test-run.py --builddir ../build/debug vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua vinyl-luatest/update_optimize_test.lua
Running in parallel with 16 workers

Timeout options:
-------------------
SERVER_START_TIMEOUT:     90
REPLICATION_SYNC_TIMEOUT: 100
TEST_TIMEOUT:             110
NO_OUTPUT_TIMEOUT:        120

Collecting tests in 'app'            (Found 0   tests): application server tests.
Collecting tests in 'app-luatest'    (Found 0   tests): application server tests on luatest.
Collecting tests in 'app-tap'        (Found 0   tests): application server tests (TAP).
Collecting tests in 'box'            (Found 0   tests): Database tests.
Collecting tests in 'box-luatest'    (Found 0   tests): Database tests.
Collecting tests in 'box-py'         (Found 0   tests): legacy python tests.
Collecting tests in 'box-tap'        (Found 0   tests): Database tests with #! using TAP.
Collecting tests in 'engine'         (Found 0   tests): tarantool multiengine tests.
Collecting tests in 'engine-luatest' (Found 0   tests): Database tests.
Collecting tests in 'engine-tap'     (Found 0   tests): tarantool multiengine tap tests.
Collecting tests in 'engine_long'    (Found 0   tests): tarantool engine stress tests.
Collecting tests in 'long_run-py'    (Found 0   tests): long running tests.
Collecting tests in 'replication'    (Found 0   tests): tarantool/box, replication.
Collecting tests in 'replication-luatest' (Found 0   tests): replication luatests.
Collecting tests in 'replication-py' (Found 0   tests): tarantool/box, replication.
Collecting tests in 'small'          (Found 0   tests): libsmall unit tests.
Collecting tests in 'sql'            (Found 0   tests): sql tests.
Collecting tests in 'sql-luatest'    (Found 0   tests): SQL tests on luatest.
Collecting tests in 'sql-tap'        (Found 0   tests): Database tests with #! using TAP.
Collecting tests in 'swim'           (Found 0   tests): SWIM tests.
Collecting tests in 'unit'           (Found 0   tests): unit tests.
Collecting tests in 'vinyl'          (Found 0   tests): vinyl integration tests.
Collecting tests in 'vinyl-luatest'  (Found 30  tests): vinyl space engine luatests.
Collecting tests in 'wal_off'        (Found 0   tests): tarantool/box, wal_mode = none.
Collecting tests in 'xlog'           (Found 0   tests): tarantool write ahead log tests.
Collecting tests in 'xlog-py'        (Found 0   tests): legacy python tests.

Tarantool server information
 | Found executable at /home/vlad/src/tarantool/tarantool/build/debug/src/tarantool
 | Found tarantoolctl at /home/vlad/src/tarantool/tarantool/build/debug/extra/dist/tarantoolctl

 | Tarantool 2.11.0-entrypoint-201-g1fa9b6648725
 | Target: Linux-x86_64-Debug
 | Build options: cmake . -DCMAKE_INSTALL_PREFIX=/usr/local -DENABLE_BACKTRACE=ON
 | Compiler: /usr/bin/cc /usr/bin/c++
 | C_FLAGS: -fexceptions -funwind-tables -fno-common -fopenmp -msse2 -std=c11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Wno-format-truncation -Wno-gnu-alignof-expression -fno-gnu89-inline -Wno-cast-function-type -Werror
 | CXX_FLAGS: -fexceptions -funwind-tables -fno-common -fopenmp -msse2 -std=c++11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Wno-format-truncation -Wno-invalid-offsetof -Wno-gnu-alignof-expression -Wno-cast-function-type -Werror


======================================================================================
WORKR TEST                                            PARAMS          RESULT
---------------------------------------------------------------------------------
[007] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[003] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[004] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[010] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[009] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[001] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[013] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[008] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[014] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[006] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[002] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[011] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[016] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[005] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[012] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[015] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[007] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[001] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[004] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[010] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[006] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[014] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[003] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[005] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[002] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[009] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[013] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[016] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[011] vinyl-luatest/update_optimize_test.lua                          [ pass ]
[008] vinyl-luatest/update_optimize_test.lua                          [ pass ]
---------------------------------------------------------------------------------
Top 10 tests by occupied memory (RSS, Mb):
*  401.0 vinyl-luatest/update_optimize_test.lua

(Tests quicker than 0.1 seconds may be missed.)
---------------------------------------------------------------------------------
Top 10 longest tests (seconds):
*   1.81 vinyl-luatest/update_optimize_test.lua
---------------------------------------------------------------------------------
Statistics:
* pass: 30
vlad@esperanza:~/src/tarantool/tarantool/test$ ps ax | grep tarantool
 241803 pts/3    Sl     0:00 tarantool default.lua <running>
 242301 pts/3    S+     0:00 grep --color=auto tarantool
vlad@esperanza:~/src/tarantool/tarantool/test$ kill 241803
vlad@esperanza:~/src/tarantool/tarantool/test$ ps ax | grep tarantool
 242497 pts/3    S+     0:00 grep --color=auto tarantool