You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If we have long, synchronous blocks of code that run at some stage of the pipeline, it will block one of the worker threads, possibly for longer than we want.
One approach to this, is to make the synchronous block async and yield during the execution.
Another approach would be to break up one large batch into smaller batches, and call the synchronous code asynchronously for each batch in a stream, and reassemble the bactches at completion.
We should evaluate both these options in the future
The text was updated successfully, but these errors were encountered:
This is motivated by a discussion here:
#455 (comment)
If we have long, synchronous blocks of code that run at some stage of the pipeline, it will block one of the worker threads, possibly for longer than we want.
One approach to this, is to make the synchronous block async and yield during the execution.
Another approach would be to break up one large batch into smaller batches, and call the synchronous code asynchronously for each batch in a stream, and reassemble the bactches at completion.
We should evaluate both these options in the future
The text was updated successfully, but these errors were encountered: