Closed
Description
The flash (and future FEL) loader currently does the conversion to parquet in parallel with all files simultaneously. This can consume huge amounts of memory, and lead to very inefficient conversion for systems with few cores and low memory (such as the github action workers).
The solution would be to do the conversion in chunks determined by the number of available cores and/or the num_cores
config parameter we already define. Maybe this should be moved from [binning]
to [core]
then.
Metadata
Metadata
Assignees
Labels
No labels