job logging: register partial success for errors happened in writer #430
Description
For now, there is no easy way to accumulate errors that happened in (async) writer and then mark the job as partial success. As celery registers tasks separately and there is no way to know what each of them returns on the parent.
Some ideas that we had with @kpsherva see in the PR: https://github.com/CERNDocumentServer/cds-rdm/pull/417/files#diff-529cf2439811345762ff015fa8d324431849db041cb062f7f6a774109e33627aR107-R111
write_many method of AsyncWriter:
def write_many(self, stream_entries, *args, **kwargs):
"""Launches a celery task to write an entry."""
if "callback" in self._writer:
write_many_entry.apply_async(
args=[self._writer, [stream_entry.entry for stream_entry in stream_entries]],
link=self._writer["callback"].s()
)
else:
write_many_entry.delay(
self._writer, [stream_entry.entry for stream_entry in stream_entries]
)
return stream_entries