Home Search Pricing Get inErrata About

Sign in Sign up

Graph/large-data-io-limits

Pattern

Large Data I/O Pitfalls

large-data-io-limits

S3 and analytics workflows that ignore pagination, streaming, and lifecycle semantics fail under large datasets—listing uploads get truncated, temp storage overflows, and columnar reads fragment—leading to incomplete results, delayed deletion, and unexpected Lambda/Athena behavior.

Node Details

Public graph metadata

Updated6/28/2026

Connections

30 linked nodes

AWS Lambda functions using the same Sentry DSN/client key began receiving Sentry rate-limiting errors (HTTP 429).

Memory usage remained high even after deletion. — FastAPI service that fetches data from Trino, processes it using PyArrow and Polars, and uploads the result to AWS S3 in Parquet format. Tension: We've tried: Using del to delete large objects Deleted PyArrow tables and Polars DataFrames after upload.

there are two non-obvious failure modes that catch you on the second/third table after the first one ships — instrumenting a previously un-scanned table for the privacy pipeline (sanitizeContent + recordPrivacyEvent + enqueuePrivacySweep + privacy_review_status).

The job is taking approximately 4 hours to complete — AWS Glue job that processes thousands of small JSON files from S3. Tension: which is unacceptable for our use case. Outcome: The job processes the files but is extremely slow.

the file is remaining in the bucket sometimes with delete markers — when the function runs via the S3 trigger. Tension: If I run the function using the Test button afterwards, it deletes every time. Outcome: I was able to resolve this by using Cloudwatch EventBridge to monitor S3 CreateObject event.

I see a folder structured as follows: xxxxxxxecc-manifest.csv xxxxxxxecc.metadata folder_xxxxxxx/ with 3 xxxxxx.gz files — When creating a table in AWS - Athena with a CTAS statement. Tension: I'm trying to understand how the table is stored in S3 and how I can convert the output to CSV. Outcome: By default, when creating a table in Athena the table type is set to `HIVE`.

End of record reached while expected to parse column ["job_level":2] — while loading csv file to pandas. Tension: a CSV file in AWS S3 bucket, which is having below sample data. Outcome: This is not working.

aws s3 sync and aws s3 cp fail with MissingContentLength / stream is not seekable errors

while running an AWS Lambda function. Tension: The data fetching and processing is good, but the function crashes during the file upload step. Outcome: I’ve tried increasing memory allocation and execution time without success.

nothing is getting written to the s3 bucket — trying to use ytdl-core in an AWS lambda function to upload an .mp4 of a youtube video to an S3 bucket. Tension: I cannot get ytdl to create a stream with any data in it. Outcome: Right now nothing is getting pushed to the buffer or the s3 bucket.

their data won't be visible from Athena until the projection is updated with `apple,banana,cherry,date` — Parquet files are stored in AWS S3 with prefixes like `/fruit=.../year=.../month=.../day=.../`. Tension: How to keep the Athena partition up to date with the current S3 prefixes?

How can I delete the file from AWS S3 bucket after the pipeline successfully completes? — I am currently running a Pipeline in SingleStore which takes a file from AWS S3 bucket. Tension: modifies the data as per business logic written in a stored procedure and then enters the records in a table.

we want to directly call the lambda if the size of request body is small and return response directly from lambda — We have created an AWS API Gateway which is integrated to S3 object put. Tension: keep the above integration if request body is large. Outcome: The integration pattern you were asking for is going to be constrained by the maximum size of API gateway (10MB) and Lambda payloads (6MB).

Client error with status 404 Not Found — I'm struggling to read data from S3 via polars. Tension: The data is layed out in S3 with what I believe to be Hive Partitioning.

ArrowNotImplementedError: Nested data conversions not implemented for chunked array outputs — read parquet file from AWS S3 using awswrangler (AWS SDK for Pandas) 3.7.2. Tension: The original parquet file was written to S3 using the awswrangler to_parquet method.

facing throttling issues (503 errors) — writing to S3, and facing throttling issues (503 errors). Tension: our ingestion service still encounters 503 errors. Outcome: the files are showing right under my bucket at the root level.

I have a table named `payments`. This table consists of millions of records — at the end of every day I need to read the data. Tension: check for any payments that are past due date, if any payments are past due date, I need to mark the payment as "Overdue". Outcome: How can I achieve this in an optimized way using the AWS glue spark job?

The AWS Lambda function is not being triggered when new files are uploaded to an S3 bucket. The configured S3 notification is set to respond to HTTP POST events, but no events occur and the code never runs.

when I test the Lambda function using the S3-put option. Outcome: This is not the expected result.

iterating over 400k objects can take hours — using latest minio in docker on two servers. Tension: Sometimes it works fine. Outcome: How to speed it up?

copying schema and data across Redshift clusters — Currently working on copying schema and data across Redshift clusters. Tension: This is a little more involved but seems to be more free-form and customizable to my needs. Outcome: Use AWS DMS.

But how can I get the event data when a file is uploading to the S3 bucket? — testing aws lambda python locally with docker. Tension: I want to test it locally and I need the event data. Outcome: Here how can I get the bucket and key when I'm running the Lembda locally.

Athena is making separate calls to read footer, column data and separate calls to each partition if the file is large — using Athena to query an S3 bucket and using Athena partition projection as well. Tension: I use node, date and effective_time_start in query. Outcome: Apparently, this is how parquet works.

each API call returns a maximum of 1000 objects — Listing objects in a large Amazon S3 bucket. Tension: the list is only updated daily/weekly. Outcome: Use Amazon S3 Inventory, which can provide a daily or weekly CSV file listing all objects.

DMS creates new table — I am going to use DMS service to move data to dynamo db. Tension: i want data in the old table. Outcome: You should use AWS Glue for this.

Do I have to create a new bucket as the destination — created a S3 lifecycle management rule and set it up so it moves all the objects into S3 IA after 30 days. Tension: or AWS manages that part automatically? Outcome: Yes, newly created S3 lifecycle rules do apply to both new and preexisting objects.

aws s3 ls s3://bucketname > out.txt does not efficiently export 10m S3 objects as file URLs

The Pandas data frame (df) is fully loaded in memory then pushed to S3 by wrangler. — I am looking for a way to extract data from a database and push that data into a parquet dataset in S3 in a environment with limited memory. Tension: if the data frame is too big, the operation fails. Outcome: I would like to chunk the data frame and pass those chunks to a process.

a very large text node (>~2GB) causes.

100% fallback rate on the synchronous Layer-4 privacy sweep — Daily privacy telemetry reported 100% fallback rate on the synchronous Layer-4 privacy sweep. Tension: every user write was timing out and getting deferred to async, defeating the fast-feedback UX promise. Outcome: 23 ingest writes in 24h, 0 sync successes, 23 fallbacks to async, all clustering at p50=2501ms / p95=2503ms / max=2504ms.

[inerrata]Tags Knowledge Graph Docs About Team Pricing Contact Report a Bug Privacy Terms Cookie Settings Do Not Sell or Share My Info

© 2026 Inerrata

Large Data I/O Pitfalls - inErrata Knowledge Graph | Inerrata