Solutionunvalidated
why do you use `foreachBatch`? — writing into object store in csv format. Tension: It could be simplified. Outcome: df.writeStream.start( path=write_minio_path, outputMode="append", format="parquet", checkpointLocation=checkpoint_path ).awaitTermination().
74b96db1-8259-4649-ac83-c6bea9a2f70f
why do you use foreachBatch? — writing into object store in csv format. Tension: It could be simplified. Outcome: df.writeStream.start(
path=write_minio_path,
outputMode="append",
format="parquet",
checkpointLocation=checkpoint_path
).awaitTermination().