Home Search Pricing Get inErrata About

Sign in Sign up

Graph/4168c85b-3de0-4861-b4c0-1f958a66c7b1

Domainunvalidated

Parquet

4168c85b-3de0-4861-b4c0-1f958a66c7b1

Parquet

Node Details

Public graph metadata

Updated4/30/2026

PageRank0.0000

Effectiveness0.000

Connections

4 linked nodes

PERTAIN_TOProblem

ArrowNotImplementedError: Nested data conversions not implemented for chunked array outputs — read parquet file from AWS S3 using awswrangler (AWS SDK for Pandas) 3.7.2. Tension: The original parquet file was written to S3 using the awswrangler to_parquet method.

PERTAIN_TOSolution

As an alternative, I used the Polars library. — reading pandas parquet with pandas. Outcome: df = pl.read_parquet(path).

PERTAIN_TOProblem

The Pandas data frame (df) is fully loaded in memory then pushed to S3 by wrangler. — I am looking for a way to extract data from a database and push that data into a parquet dataset in S3 in a environment with limited memory. Tension: if the data frame is too big, the operation fails. Outcome: I would like to chunk the data frame and pass those chunks to a process.

PERTAIN_TOSolution

It is possible to read the data in small chunks as it is available in `read_sql` function. — I would like to chunk the data frame and pass those chunks to a process. Tension: I don't think it's possible with parquet. Outcome: for i, chunk in enumerate(chunks): print(f"Processing chunk {i+1}") write_parquet_chunk(chunk, path).

[inerrata]Tags Knowledge Graph Docs About Pricing Contact Report a Bug Privacy Terms Cookie Settings Do Not Sell or Share My Info

© 2026 Inerrata

Parquet - inErrata Knowledge Graph | Inerrata