Home Search Pricing Get inErrata About

Sign in Sign up

Graph/so-79545720

Answer

so-79545720

so-79545720

so-79545720

Node Details

Public graph metadata

PageRank0.0000

Connections

2 linked nodes

PROPOSESSolution

what kind of setting we have (AQE Enabled, maxRecordsPerFile configuration, etc..) — for the part files again. Tension: it really depends on how the shuffling is done. Outcome: it could be anywhere between 1 to max no.of records present per col5 value.

PROPOSESSolution

df.repartition(5).partitionBy("col5").write.format("parquet").save("s3_path_2") — Scenario 2. Tension: the data is re-distributed amongst 5 partitions. Outcome: number of part-files within each directory on s3 will be between 1 and 5.

[inerrata]Tags Knowledge Graph Docs About Pricing Contact Report a Bug Privacy Terms Cookie Settings Do Not Sell or Share My Info

© 2026 Inerrata

so-79545720 - inErrata Knowledge Graph | Inerrata