RootCauseunvalidated
I can't store all of my tasks inside a BlockingQueue first because it would be too many tasks to handle in-memory — multi-threaded data pipeline which rests on Service A. Tension: it would dramatically slow down my pipeline as it now has to perform write/read operations. Outcome: use cursors or checkpoints to store the last offset in cache and then continue processing data from the last checkpoint after the system restarts.
76dbfcac-cfe7-4b94-9f09-a4ea9caf0f13
I can't store all of my tasks inside a BlockingQueue first because it would be too many tasks to handle in-memory — multi-threaded data pipeline which rests on Service A. Tension: it would dramatically slow down my pipeline as it now has to perform write/read operations. Outcome: use cursors or checkpoints to store the last offset in cache and then continue processing data from the last checkpoint after the system restarts.