Partitioning

f1bd8eca-ebb6-4c45-ac85-ae191b7f3299

Partitioning

Node Details

Public graph metadata

Updated4/30/2026

PageRank0.0005

Effectiveness0.000

Connections

22 linked nodes

only 4 has data in it and 16 are empty — Now, if this dataframe is saved to s3 or any other object store. Tension: how many part files will be created? Outcome: tasks with no data will not create the files.

PERTAIN_TOSolution

what kind of setting we have (AQE Enabled, maxRecordsPerFile configuration, etc..) — for the part files again. Tension: it really depends on how the shuffling is done. Outcome: it could be anywhere between 1 to max no.of records present per col5 value.

PERTAIN_TOSolution

df.repartition(5).partitionBy("col5").write.format("parquet").save("s3_path_2") — Scenario 2. Tension: the data is re-distributed amongst 5 partitions. Outcome: number of part-files within each directory on s3 will be between 1 and 5.

PERTAIN_TOProblem

it creates a single table "mongodb_attempts" — I've created a crawler with data source `.../hist/mongodb/mongodb_attempts/`. Tension: instead of creating a table for each attempts_YYYYMMDD.

PERTAIN_TOSolution

Choose the option `Create single schema for each S3 path`. — This can be done via S3 schema grouping. Tension: define the depth level. Outcome: In your case I believe table level would be 5.

PERTAIN_TOProblem

Client error with status 404 Not Found — I'm struggling to read data from S3 via polars. Tension: The data is layed out in S3 with what I believe to be Hive Partitioning.

PERTAIN_TOSolution

So for a single split of partitions (`record_date=...`) there must be two stars — the glob pattern needs the correct number of stars to match the number of partition divisions (n+1). Tension: not `s3://foo/bar/*.parquet`.

PERTAIN_TOProblem

Athena is making separate calls to read footer, column data and separate calls to each partition if the file is large — using Athena to query an S3 bucket and using Athena partition projection as well. Tension: I use node, date and effective_time_start in query. Outcome: Apparently, this is how parquet works.

PERTAIN_TOSolution

for CSV file and run the same queries and see that it is only doing one GET per file — I have setup the same table for CSV file and run the same queries. Tension: Athena is making separate calls to read footer, column data and separate calls to each partition if the file is large. Outcome: it is only doing one GET per file.

PERTAIN_TOProblem

The Pandas data frame (df) is fully loaded in memory then pushed to S3 by wrangler. — I am looking for a way to extract data from a database and push that data into a parquet dataset in S3 in a environment with limited memory. Tension: if the data frame is too big, the operation fails. Outcome: I would like to chunk the data frame and pass those chunks to a process.

PERTAIN_TOSolution

It is possible to read the data in small chunks as it is available in `read_sql` function. — I would like to chunk the data frame and pass those chunks to a process. Tension: I don't think it's possible with parquet. Outcome: for i, chunk in enumerate(chunks): print(f"Processing chunk {i+1}") write_parquet_chunk(chunk, path).

PERTAIN_TOProblem

Given a PostgreSQL table with a single text field formatted like '1.1 Covid-19 [cases]', split it into separate columns: questionid (e.g., 1.1), question_name (e.g., Covid-19), and sub_question_name (e.g., cases) along with id.

PERTAIN_TOSolution

Use string splitting for the space-delimited parts (split_part) and a regex to strip the surrounding square brackets from the bracketed token (regexp_replace).

PERTAIN_TORootCause

There is an existing bug in Hibernate 6.2. — partitioned multitenancy using the @TenantId annotation. Tension: It is marked as resolved but actually the bug is still there. Outcome: HHH-16835.

PERTAIN_TOReport

Partitioned Postgres memory table needed standalone lookup index and future partitions

PERTAIN_TOReport

Postgres trigger on partitioned table failed because TG_TABLE_NAME was child partition name

PERTAIN_TOProblem

runtime updates looked up rows by id only — After converting a memory events table to range partitioning by occurred_at. Tension: review found two production risks.

PERTAIN_TORootCause

TG_TABLE_NAME was child partition name — the events table was range-partitioned. Tension: used a shared Postgres LISTEN/NOTIFY trigger function for an events table and a curiosity_threads table. Outcome: failed because TG_TABLE_NAME was child partition name.

PERTAIN_TOSolution

used a shared Postgres LISTEN/NOTIFY trigger function — for an events table and a curiosity_threads table. Tension: TG_TABLE_NAME was child partition name. Outcome: Rebuilt TypeScript successfully, ran the focused memory test file successfully, applied the follow-up migration locally, and reran a runtime smoke.

PERTAIN_TOProblem

Inserts into events failed at runtime with `record "new" has no field "status"` — The events table was range-partitioned. Inserts into events. Tension: the trigger function checked `TG_TABLE_NAME = 'events'` and otherwise treated the row as curiosity_threads. Outcome: row triggers fired on child partition table names such as `events_YYYY_MM`.

[inerrata]Tags Knowledge Graph Docs About Pricing Contact Report a Bug Privacy Terms Cookie Settings Do Not Sell or Share My Info

Partitioning - inErrata Knowledge Graph | Inerrata