Skip to content

test: flaky address already in use in app-tap and box-tap suites tests #186

Closed
tarantool/tarantool
#7551
@avtikhon

Description

@avtikhon

Tarantool version:
Tarantool 2.6.0-114-g6c04687566
Target: Linux-x86_64-RelWithDebInfo
Build options: cmake . -DCMAKE_INSTALL_PREFIX=/builds/M4RrgQZ3/0/tarantool/tarantool/static-build/tarantool-prefix -DENABLE_BACKTRACE=TRUE
Compiler: /usr/bin/cc /usr/bin/c++
C_FLAGS: -static-libstdc++ -fexceptions -funwind-tables -fno-omit-frame-pointer -fno-stack-protector -fno-common -fopenmp -msse2 -std=c11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Wno-format-truncation -Wno-gnu-alignof-expression -fno-gnu89-inline -Wno-cast-function-type -Werror
CXX_FLAGS: -static-libstdc++ -fexceptions -funwind-tables -fno-omit-frame-pointer -fno-stack-protector -fno-common -fopenmp -msse2 -std=c++11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Wno-format-truncation -Wno-invalid-offsetof -Wno-gnu-alignof-expression -Wno-cast-function-type -Werror

OS version:

  1. Fedora 29
  2. CentOS 8
  3. Debian 9
  4. CentOS 7
  5. Fedora 29

Bug description:

  1. https://gitlab.com/tarantool/tarantool/-/jobs/759541066#L1982

artifacts.zip

Results files checksum seems not stable:

results file checksum: d41d8cd98f00b204e9800998ecf8427e

[015] app-tap/debug.test.lua                                          
[015] [Instance "server"] Tarantool server failed to start
[015] 
[015] Last 15 lines of Tarantool Log file [Instance "server"][/build/usr/src/debug/tarantool-2.6.0.114/test/var/015_app-tap/server.log]:
[015] 2020-09-27 14:12:49.059 [24403] main/103/server I> set 'log_level' configuration option to 5
[015] 2020-09-27 14:12:49.059 [24403] main/106/checkpoint_daemon I> scheduled next checkpoint for Sun Sep 27 16:06:18 2020
[015] 2020-09-27 14:12:49.060 [24403] main/103/server I> set 'listen' configuration option to "\/build\/usr\/src\/debug\/tarantool-2.6.0.114\/test\/var\/015_app-tap\/server.socket-iproto"
[015] 2020-09-27 14:12:49.060 [24403] main/103/server I> set 'log_format' configuration option to "plain"
[015] 2020-09-27 14:12:49.061 [24403] main/103/server tarantoolctl:549 E> Start failed: builtin/box/console.lua:865: failed to create server localhost:53542: Address already in use
[015] 2020-09-27 14:12:49.037 [24403] main/103/server I> assigned id 1 to replica ed9406a2-e73c-4199-a89d-56a2499d03ef
[015] 2020-09-27 14:12:49.037 [24403] main/103/server I> cluster uuid bc171f6c-0696-480c-a9f2-731b0f4bb916
[015] 2020-09-27 14:12:49.054 [24403] snapshot/101/main I> saving snapshot `/build/usr/src/debug/tarantool-2.6.0.114/test/var/015_app-tap/server/00000000000000000000.snap.inprogress'
[015] 2020-09-27 14:12:49.058 [24403] snapshot/101/main I> done
[015] 2020-09-27 14:12:49.059 [24403] main/103/server I> ready to accept requests
[015] 2020-09-27 14:12:49.059 [24403] main/103/server I> set 'log_level' configuration option to 5
[015] 2020-09-27 14:12:49.059 [24403] main/106/checkpoint_daemon I> scheduled next checkpoint for Sun Sep 27 16:06:18 2020
[015] 2020-09-27 14:12:49.060 [24403] main/103/server I> set 'listen' configuration option to "\/build\/usr\/src\/debug\/tarantool-2.6.0.114\/test\/var\/015_app-tap\/server.socket-iproto"
[015] 2020-09-27 14:12:49.060 [24403] main/103/server I> set 'log_format' configuration option to "plain"
[015] 2020-09-27 14:12:49.061 [24403] main/103/server tarantoolctl:549 E> Start failed: builtin/box/console.lua:865: failed to create server localhost:53542: Address already in use
  1. https://gitlab.com/tarantool/tarantool/-/jobs/759207732#L2048

artifacts.zip

Results files checksum seems not stable:

results file checksum: d41d8cd98f00b204e9800998ecf8427e

[080] app-tap/inspector.test.lua                                      
[080] [Instance "box"] Tarantool server failed to start
[080] 
[080] Last 15 lines of Tarantool Log file [Instance "box"][/build/usr/src/debug/tarantool-2.6.0.114/test/var/080_app-tap/box.log]:
[080] 2020-09-27 01:37:47.248 [26361] main/106/checkpoint_daemon I> scheduled next checkpoint for Sun Sep 27 03:29:54 2020
[080] 2020-09-27 01:37:47.248 [26361] main/103/box I> set 'memtx_memory' configuration option to 107374182
[080] 2020-09-27 01:37:47.248 [26361] main/103/box I> set 'listen' configuration option to "\/build\/usr\/src\/debug\/tarantool-2.6.0.114\/test\/var\/080_app-tap\/box.socket-iproto"
[080] 2020-09-27 01:37:47.248 [26361] main/103/box I> set 'log_format' configuration option to "plain"
[080] 2020-09-27 01:37:47.250 [26361] main/103/box tarantoolctl:549 E> Start failed: builtin/box/console.lua:865: failed to create server localhost:55208: Address already in use
[080] 2020-09-27 01:37:47.237 [26361] main/103/box I> cluster uuid 7a79955d-0418-4a55-af5d-0af84fede406
[080] 2020-09-27 01:37:47.240 [26361] snapshot/101/main I> saving snapshot `/build/usr/src/debug/tarantool-2.6.0.114/test/var/080_app-tap/box/00000000000000000000.snap.inprogress'
[080] 2020-09-27 01:37:47.245 [26361] snapshot/101/main I> done
[080] 2020-09-27 01:37:47.247 [26361] main/103/box I> ready to accept requests
[080] 2020-09-27 01:37:47.248 [26361] main/103/box I> set 'log_level' configuration option to 5
[080] 2020-09-27 01:37:47.248 [26361] main/106/checkpoint_daemon I> scheduled next checkpoint for Sun Sep 27 03:29:54 2020
[080] 2020-09-27 01:37:47.248 [26361] main/103/box I> set 'memtx_memory' configuration option to 107374182
[080] 2020-09-27 01:37:47.248 [26361] main/103/box I> set 'listen' configuration option to "\/build\/usr\/src\/debug\/tarantool-2.6.0.114\/test\/var\/080_app-tap\/box.socket-iproto"
[080] 2020-09-27 01:37:47.248 [26361] main/103/box I> set 'log_format' configuration option to "plain"
[080] 2020-09-27 01:37:47.250 [26361] main/103/box tarantoolctl:549 E> Start failed: builtin/box/console.lua:865: failed to create server localhost:55208: Address already in use
[080] [ fail ]
  1. https://gitlab.com/tarantool/tarantool/-/jobs/758615152#L4183

artifacts.zip

results file checksum: d41d8cd98f00b204e9800998ecf8427e

[042] box-tap/net.box.test.lua                                        
[042] [Instance "box"] Tarantool server failed to start
[042] 
[042] Last 15 lines of Tarantool Log file [Instance "box"][/builds/nZUxDh2c/0/tarantool/tarantool/test/var/042_box-tap/box.log]:
[042] 2020-09-26 00:08:22.333 [27175] main/106/checkpoint_daemon I> scheduled next checkpoint for Sat Sep 26 01:47:49 2020
[042] 2020-09-26 00:08:22.334 [27175] main/103/box I> set 'memtx_memory' configuration option to 107374182
[042] 2020-09-26 00:08:22.334 [27175] main/103/box I> set 'listen' configuration option to "\/builds\/nZUxDh2c\/0\/tarantool\/tarantool\/test\/var\/042_box-tap\/box.socket-iproto"
[042] 2020-09-26 00:08:22.334 [27175] main/103/box I> set 'log_format' configuration option to "plain"
[042] 2020-09-26 00:08:22.339 [27175] main/103/box tarantoolctl:549 E> Start failed: builtin/box/console.lua:865: failed to create server localhost:56234: Address already in use
[042] 2020-09-26 00:08:22.330 [27175] main/103/box I> cluster uuid c1c284d4-7f87-40e3-a97e-b4eb27d3682a
[042] 2020-09-26 00:08:22.331 [27175] snapshot/101/main I> saving snapshot `/builds/nZUxDh2c/0/tarantool/tarantool/test/var/042_box-tap/box/00000000000000000000.snap.inprogress'
[042] 2020-09-26 00:08:22.332 [27175] snapshot/101/main I> done
[042] 2020-09-26 00:08:22.333 [27175] main/103/box I> ready to accept requests
[042] 2020-09-26 00:08:22.333 [27175] main/103/box I> set 'log_level' configuration option to 5
[042] 2020-09-26 00:08:22.333 [27175] main/106/checkpoint_daemon I> scheduled next checkpoint for Sat Sep 26 01:47:49 2020
[042] 2020-09-26 00:08:22.334 [27175] main/103/box I> set 'memtx_memory' configuration option to 107374182
[042] 2020-09-26 00:08:22.334 [27175] main/103/box I> set 'listen' configuration option to "\/builds\/nZUxDh2c\/0\/tarantool\/tarantool\/test\/var\/042_box-tap\/box.socket-iproto"
[042] 2020-09-26 00:08:22.334 [27175] main/103/box I> set 'log_format' configuration option to "plain"
[042] 2020-09-26 00:08:22.339 [27175] main/103/box tarantoolctl:549 E> Start failed: builtin/box/console.lua:865: failed to create server localhost:56234: Address already in use

https://gitlab.com/tarantool/tarantool/-/jobs/759207861#L4888
https://gitlab.com/tarantool/tarantool/-/jobs/759236803#L2293
https://gitlab.com/tarantool/tarantool/-/jobs/759476184#L2185

artifacts.zip

Not possible to use results file for checksum creation due to a lot of changing information printing.

[034] app-tap/http_client.test.lua                                    [ fail ]
[034] Test failed! Output from reject file app-tap/http_client.reject:
[034] TAP version 13
[034] # TARANTOOL_SRC_DIR=/builds/M4RrgQZ3/0/tarantool/tarantool
[034] 1..2
[034]     # http over AF_INET
[034]     1..11
[034]     # starting HTTP server on 127.0.0.1:34324...
[034]     not ok - server started
[034]       ---
[034]       filename: /builds/M4RrgQZ3/0/tarantool/tarantool/test/app-tap/http_client.test.lua
[034]       line: 0
[034]       expected: heartbeat
[034]       trace:
[034]       - line: 25
[034]         source: '@/builds/M4RrgQZ3/0/tarantool/tarantool/test/app-tap/http_client.test.lua'
[034]         filename: /builds/M4RrgQZ3/0/tarantool/tarantool/test/app-tap/http_client.test.lua
[034]         what: Lua
[034]         namewhat: upvalue
[034]         name: start_server
[034]         src: '.../0/tarantool/tarantool/test/app-tap/http_client.test.lua'
[034]       - line: 585
[034]         source: '@/builds/M4RrgQZ3/0/tarantool/tarantool/test/app-tap/http_client.test.lua'
[034]         filename: /builds/M4RrgQZ3/0/tarantool/tarantool/test/app-tap/http_client.test.lua
[034]         what: Lua
[034]         namewhat: global
[034]         name: run_tests
[034]         src: '.../0/tarantool/tarantool/test/app-tap/http_client.test.lua'
[034]       - line: 616
[034]         source: '@/builds/M4RrgQZ3/0/tarantool/tarantool/test/app-tap/http_client.test.lua'
[034]         filename: /builds/M4RrgQZ3/0/tarantool/tarantool/test/app-tap/http_client.test.lua
[034]         what: Lua
[034]         namewhat: local
[034]         name: fun
[034]         src: '.../0/tarantool/tarantool/test/app-tap/http_client.test.lua'
[034]       - line: 218
[034]         source: '@builtin/tap.lua'
[034]         filename: builtin/tap.lua
[034]         what: Lua
[034]         namewhat: method
[034]         name: test
[034]         src: builtin/tap.lua
[034]       - line: 0
[034]         source: '@/builds/M4RrgQZ3/0/tarantool/tarantool/test/app-tap/http_client.test.lua'
[034]         filename: /builds/M4RrgQZ3/0/tarantool/tarantool/test/app-tap/http_client.test.lua
[034]         what: main
[034]         namewhat: 
[034]         src: '.../0/tarantool/tarantool/test/app-tap/http_client.test.lua'
[034]       ...
[034]     # trying to connect to http://127.0.0.1:34324/
[034]     not ok - connection is ok
[034]       ---
[034]       filename: /builds/M4RrgQZ3/0/tarantool/tarantool/test/app-tap/http_client.test.lua
[034]       trace:
[034]       - line: 25
[034]         source: '@/builds/M4RrgQZ3/0/tarantool/tarantool/test/app-tap/http_client.test.lua'
[034]         filename: /builds/M4RrgQZ3/0/tarantool/tarantool/test/app-tap/http_client.test.lua
[034]         what: Lua
[034]         namewhat: upvalue
[034]         name: start_server
[034]         src: '.../0/tarantool/tarantool/test/app-tap/http_client.test.lua'
[034]       - line: 585
[034]         source: '@/builds/M4RrgQZ3/0/tarantool/tarantool/test/app-tap/http_client.test.lua'
[034]         filename: /builds/M4RrgQZ3/0/tarantool/tarantool/test/app-tap/http_client.test.lua
[034]         what: Lua
[034]         namewhat: global
[034]         name: run_tests
[034]         src: '.../0/tarantool/tarantool/test/app-tap/http_client.test.lua'
[034]       - line: 616
[034]         source: '@/builds/M4RrgQZ3/0/tarantool/tarantool/test/app-tap/http_client.test.lua'
[034]         filename: /builds/M4RrgQZ3/0/tarantool/tarantool/test/app-tap/http_client.test.lua
[034]         what: Lua
[034]         namewhat: local
[034]         name: fun
[034]         src: '.../0/tarantool/tarantool/test/app-tap/http_client.test.lua'
[034]       - line: 218
[034]         source: '@builtin/tap.lua'
[034]         filename: builtin/tap.lua
[034]         what: Lua
[034]         namewhat: method
[034]         name: test
[034]         src: builtin/tap.lua
[034]       - line: 0
[034]         source: '@/builds/M4RrgQZ3/0/tarantool/tarantool/test/app-tap/http_client.test.lua'
[034]         filename: /builds/M4RrgQZ3/0/tarantool/tarantool/test/app-tap/http_client.test.lua
[034]         what: main
[034]         namewhat: 
[034]         src: '.../0/tarantool/tarantool/test/app-tap/http_client.test.lua'
[034]       line: 0
[034]       expected: 200
[034]       got: 595
[034]       ...
[034] 
[034] Last 15 lines of Tarantool Log file [Instance "app_server"][/builds/M4RrgQZ3/0/tarantool/tarantool/test/var/034_app-tap/http_client.test.lua.tarantool.log]:
[034] Traceback (most recent call last):
[034]   File "/builds/M4RrgQZ3/0/tarantool/tarantool/test/app-tap/httpd.py", line 141, in <module>
[034]     sock.bind(sock_addr)
[034]   File "<string>", line 1, in bind
[034] socket.error: [Errno 98] Address already in use
  1. https://gitlab.com/tarantool/tarantool/-/jobs/822743309#L2071

artifacts.zip

results file checksum: d41d8cd98f00b204e9800998ecf8427e

[057] box-tap/session.storage.test.lua                                
[057] [Instance "tiny"] Tarantool server failed to start
[057] 
[057] Last 15 lines of Tarantool Log file [Instance "tiny"][/build/usr/src/debug/tarantool-2.7.0.10/test/var/057_box-tap/tiny.log]:[057] 2020-10-31 10:57:20.082 [13166] main/106/checkpoint_daemon I> scheduled next checkpoint for Sat Oct 31 11:57:47 2020
[057] 2020-10-31 10:57:20.082 [13166] main/103/tiny I> set 'memtx_memory' configuration option to 52428800
[057] 2020-10-31 10:57:20.082 [13166] main/103/tiny I> set 'listen' configuration option to "\/build\/usr\/src\/debug\/tarantool-2.7.0.10\/test\/var\/057_box-tap\/tiny.socket-iproto"
[057] 2020-10-31 10:57:20.083 [13166] main/103/tiny I> set 'log_format' configuration option to "plain"
[057] 2020-10-31 10:57:20.084 [13166] main/103/tiny tarantoolctl:549 E> Start failed: builtin/box/console.lua:865: failed to create server localhost:33666: Address already in use
[057] 2020-10-31 10:57:20.079 [13166] main/103/tiny I> cluster uuid 43a6d250-290d-458c-b4f9-91170966ff89
[057] 2020-10-31 10:57:20.080 [13166] snapshot/101/main I> saving snapshot `/build/usr/src/debug/tarantool-2.7.0.10/test/var/057_box-tap/tiny/00000000000000000000.snap.inprogress'
[057] 2020-10-31 10:57:20.081 [13166] snapshot/101/main I> done
[057] 2020-10-31 10:57:20.082 [13166] main/103/tiny I> ready to accept requests
[057] 2020-10-31 10:57:20.082 [13166] main/103/tiny I> set 'log_level' configuration option to 5
[057] 2020-10-31 10:57:20.082 [13166] main/106/checkpoint_daemon I> scheduled next checkpoint for Sat Oct 31 11:57:47 2020
[057] 2020-10-31 10:57:20.082 [13166] main/103/tiny I> set 'memtx_memory' configuration option to 52428800
[057] 2020-10-31 10:57:20.082 [13166] main/103/tiny I> set 'listen' configuration option to "\/build\/usr\/src\/debug\/tarantool-2.7.0.10\/test\/var\/057_box-tap\/tiny.socket-iproto"
[057] 2020-10-31 10:57:20.083 [13166] main/103/tiny I> set 'log_format' configuration option to "plain"
[057] 2020-10-31 10:57:20.084 [13166] main/103/tiny tarantoolctl:549 E> Start failed: builtin/box/console.lua:865: failed to create server localhost:33666: Address already in use

Steps to reproduce:

Optional (but very desirable):

  • coredump
  • backtrace
  • netstat

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions