Improve log output piping to programs such as less

[stastnypremysl]

Bug Report

In the given folder, there is a strategy, its output, the catalog data in zip file and a script.

Expected Behavior

./strategy.py 2>&1 | less prints the whole log of the NT.

Actual Behavior

./strategy.py 2>&1 | less prints log output contracted at the end.

Further comment

./strategy.py 2>&1 prints the whole log of the NT.
./strategy.py 2>&1 | cat prints the whole log of the NT.
./strategy.py >./strategy.out 2>&1 prints the whole log of the NT.
./strategy.py 2>&1 | cat >./strategy.out prints the whole log of the NT.
./strategy.py 2>&1 | dd bs=32 >./strategy.out prints the whole log of the NT.

./strategy.py 2>&1 | more prints the contracted log of the NT.
./strategy.py 2>&1 | cat | less prints the contracted log of the NT.
./strategy.py 2>&1 | dd bs=32 | less prints the contracted log of the NT.

If bug can't be reproduced on the MacOS, please try it on the GNU/Linux.

Steps to Reproduce the Problem

1. Download the data
2. Extract the zip file and change extracted catalog path in strategy.py to the catalog
   3a. Run ./run.sh for a positive result
   3b. Run ./strategy.py 2>&1 for a negative result

Specifications

• OS platform: Gentoo Linux (native build, amd64)
• Python version: Python 3.12.9 (native build)
• nautilus_trader version: 1.217.0 (revision 95a671e; make build-wheel)

Additional notes

• The simulation is running on aggregated trade ticks from Binance data collection
• The instrument information was downloaded using the NT Binance client
• The vectorization and other micro-architecture processor instructions and customized optimization might have caused the bug to behave differently on my system
• Ignore the 1.217.6 version in the log - no changes in the source codes was made other then change of version

[cjdsellers]

Hi @stastnypremysl

Thanks for the report.

The logging system is using a separate thread (probably causal to this) which correctly outputs to stdout, stderr and files.

We actually made some improvements recently to log buffer flushing, and I think the commit you are on here should include those changes.

I wouldn't expect piping into the less program to necessarily work due to the threading. This ticket should be turned into an enhancement request for improving piping, as the main logging use cases of stdout, stderr and files are well handled.

If you (or anyone else) is able to figure out a fix for the issue then we'd definitely accept a PR.