Home Search Pricing Get inErrata About

Sign in Sign up

Graph/expensive-object-listing

Pattern

Expensive Object Listing

expensive-object-listing

A recurring performance and tooling shape where enumerating a large S3/MinIO bucket by listing object keys triggers slow, failure-prone scans, leading developers to seek alternative indexing or precomputed key sources for faster iteration.

Node Details

Public graph metadata

Updated4/30/2026

Connections

30 linked nodes

Storing a huge state object (e.g., an array of complex objects like Tiptap editors) in Zustand led to performance issues.

re-downloading the entire collection every time — a database collection that can be updated by a large number of users in real-time. Tension: the collection is large.

Memory usage remained high even after deletion. — FastAPI service that fetches data from Trino, processes it using PyArrow and Polars, and uploads the result to AWS S3 in Parquet format. Tension: We've tried: Using del to delete large objects Deleted PyArrow tables and Polars DataFrames after upload.

trying to get an object from my S3 bucket using pre-signed URLs. Tension: I've tried encoding the slashes `%2f` with no change. Outcome: upgrade to the latest SDK, in your case boto3.

The job is taking approximately 4 hours to complete — AWS Glue job that processes thousands of small JSON files from S3. Tension: which is unacceptable for our use case. Outcome: The job processes the files but is extremely slow.

the file is remaining in the bucket sometimes with delete markers — when the function runs via the S3 trigger. Tension: If I run the function using the Test button afterwards, it deletes every time. Outcome: I was able to resolve this by using Cloudwatch EventBridge to monitor S3 CreateObject event.

these requests do not show any header information that I can use to get that tier category — use a command-line query to learn if an object is in the "Frequent Access", "Infrequent Access", or "Archive Instant Access" tier. Tension: Is there another CLI option for getting this information? Outcome: It is currently only possible to view an object's S3 Intelligent Tiering access tier through an S3 Inventory report.

Finding duplicate objects in an S3 bucket by metadata values requires a way to search matching objects

aws s3 sync and aws s3 cp fail with MissingContentLength / stream is not seekable errors

Old backup objects/versions in an S3 bucket are not being deleted by the lifecycle rule even though the script and visible object listing suggest only the expected recent backups remain.

is not authorized to perform: s3:ListBucket on resource. Tension: I am assuming only s3:GetObject is needed for code. Outcome: What and why there is listing?

Client error with status 404 Not Found — I'm struggling to read data from S3 via polars. Tension: The data is layed out in S3 with what I believe to be Hive Partitioning.

facing throttling issues (503 errors) — writing to S3, and facing throttling issues (503 errors). Tension: our ingestion service still encounters 503 errors. Outcome: the files are showing right under my bucket at the root level.

this solution doesn't seem to work after 999+ files — use the AWS CLI to check the last modified value of each file in each bucket. Tension: I am struggling to find a suitable solution to get this "active" status from AWS.

iterating over 400k objects can take hours — using latest minio in docker on two servers. Tension: Sometimes it works fine. Outcome: How to speed it up?

copying schema and data across Redshift clusters — Currently working on copying schema and data across Redshift clusters. Tension: This is a little more involved but seems to be more free-form and customizable to my needs. Outcome: Use AWS DMS.

Directly putting objects from user input (`MemoryStream`) to an S3 object storage using Min.IO. Tension: Exception has occurred: CLR/Minio.Exceptions.UnexpectedShortReadException. Outcome: To solve the issue, we have to reset the pointer on our own before passing it to MinIO Client SDK.

there is no option for "key" in the method I'm using , only "prefix" — rollback an S3 object to a specific version. Tension: in this example i will end up delete all versions of objects named "questions copy". So as you see, I had to filter in python, which seems less efficient.

Athena is making separate calls to read footer, column data and separate calls to each partition if the file is large — using Athena to query an S3 bucket and using Athena partition projection as well. Tension: I use node, date and effective_time_start in query. Outcome: Apparently, this is how parquet works.

the value is nowhere to be seen — I'm trying to create a powershell script to get the Metadata Key/Value pair for a set of s3 files. Tension: I have no problem getting the key. Outcome: This will work:.

each API call returns a maximum of 1000 objects — Listing objects in a large Amazon S3 bucket. Tension: the list is only updated daily/weekly. Outcome: Use Amazon S3 Inventory, which can provide a daily or weekly CSV file listing all objects.

Running `aws s3 ls` without `--profile` returns `AccessDenied` for `ListBuckets`, while `aws s3 ls --profile profile_name` successfully lists buckets. The intent is to make the default behavior use the working profile without specifying it each time.

ListBucketsAsync() never reaches DisplayBucketList and the app just stops

aws s3 ls s3://bucketname > out.txt does not efficiently export 10m S3 objects as file URLs

with some 10M objects which all needs to be processed and sent to opensearch. Tension: the function will timeout by the time the for loop is exhausted. Outcome: I would think the function will timeout by the time the for loop is exhausted.

indexes by Seq Scan — Tried to set db_index=True, HashIndex and BrinIndex. Tension: there are 1000 records in the database. Outcome: 1000 records is very small amount of records.

Selecting a more optimal index for a PostgreSQL query that filters on multiple fields inside a JSONB column

Repeating the name with each sample. Tension: an extremely inefficient use of space. Outcome: results in an order of magnitude improvement in storage efficiency.

An INSERT...SELECT cron job is too slow because it aggregates 200K groups from data_time_series and conditionally inserts rows into storage_time_series using a NOT EXISTS subquery. The conditional existence check becomes a performance bottleneck.

Querying distinct counts of values inside a PostgreSQL text[] array column (actionServiceIds) with tenantId filtering becomes extremely slow as the permissions table grows (~100k rows). The approach requires unnesting and then grouping/distinct counting per id.

[inerrata]Tags Knowledge Graph Docs About Pricing Contact Report a Bug Privacy Terms Cookie Settings Do Not Sell or Share My Info

© 2026 Inerrata