Home Search Pricing Get inErrata About

Sign in Sign up

Graph/glue-compute-choice-drift

Pattern

Compute Choice Drift

glue-compute-choice-drift

A recurring decision-shape where teams must choose between Glue Python shell vs Spark (and related ETL tooling) without reliable capacity guidance, leading to oversimplified scheduling or inefficient full-data imports that strain datasets and DPUs.

Node Details

Public graph metadata

Updated4/30/2026

Connections

30 linked nodes

Type instantiation is excessively deep and possibly infinite — in a monorepo/workspace project. Tension: on seemingly benign types like this. Outcome: I've found this is almost always due to a mismatched dependency between 2 packages in your repository.

parallel workers sometimes grab overlapping sets, then wait to mark them as being processed — At the time of item selection, you neither lock them nor check whether they're locked. Tension: someone else could've already grabbed a subset of those. Outcome: duplicate their work on the items they happened to share.

thousands of occurrences of the same error — When an infrastructure incident happens, the application will start to generate thousands of occurrences of the same error. Tension: avoid depleting the error quota. Outcome: Yes, there are multiple solutions to this common problem.

KeyError: 1 — after increasing the partition count from 1 to 2 for the snuba-commit-log topic. Tension: I sometimes saw timeout errors and thought I could fix them by increasing the partition count.

Memory usage remained high even after deletion. — FastAPI service that fetches data from Trino, processes it using PyArrow and Polars, and uploads the result to AWS S3 in Parquet format. Tension: We've tried: Using del to delete large objects Deleted PyArrow tables and Polars DataFrames after upload.

The idea is to update this model once hourly without blocking the API's ability to respond. Tension: update this model once hourly without blocking the API's ability to respond to CPU-bound tasks. Outcome: this is all good.

trying to perform data processing during the application's startup. Tension: This limitation forces me to manually create instances of these dependencies. Outcome: this approach doesn't work as expected.

java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x7b8a43da) cannot access class sun.nio.ch.DirectBuffer — using scala 2.12.12, sbt 1.6.0, spark 3.5.1 with java 17 to run spark jobs. Tension: module java.base does not export sun.nio.ch to unnamed module.

Attempting to define dependent (conditional) `workflow_dispatch` input choices in GitHub Actions where the allowed options for a later input depend on the value selected for an earlier input.

I've noticed I've several caches that maybe don't need to be created. — I'm using the cache action in GitHub Actions workflows. Tension: even if it restores always the same cache, no matter if the dependencies are changed, it should still work fine. Outcome: it mostly depends on how different and big these caches are.

The job is taking approximately 4 hours to complete — AWS Glue job that processes thousands of small JSON files from S3. Tension: which is unacceptable for our use case. Outcome: The job processes the files but is extremely slow.

only 4 has data in it and 16 are empty — Now, if this dataframe is saved to s3 or any other object store. Tension: how many part files will be created? Outcome: tasks with no data will not create the files.

I want to use AWS DMS to connect to A, B, C databases and make full load/CDC into data lake bucket — different SQL databases from different departments in the org. Tension: how can I identify which files are coming from Source Database A, Source Database B and Source Database C. Outcome: I cannot find information about how to achieve what's in the title of this question.

There is no official numbers provided by AWS. — decide between Python jobs and spark jobs. Tension: Can someone specify in numbers, what is the approximate value of datasets which we should be considering for selecting AWS Glue Python shell jobs rather than spark jobs? Outcome: use just Python jobs if the simple set of transformations are required and one doesn't need horizontal or vertical scaling for the compute.

I have a table named `payments`. This table consists of millions of records — at the end of every day I need to read the data. Tension: check for any payments that are past due date, if any payments are past due date, I need to mark the payment as "Overdue". Outcome: How can I achieve this in an optimized way using the AWS glue spark job?

it takes a lot of time — read these files in my pyspark job using `spark.read.parquet(*keys)`. Tension: The reading alone is take in neighborhood of 1hr for a 40CPU cluster. Outcome: All these parquet files have same schema.

iterating over 400k objects can take hours — using latest minio in docker on two servers. Tension: Sometimes it works fine. Outcome: How to speed it up?

copying schema and data across Redshift clusters — Currently working on copying schema and data across Redshift clusters. Tension: This is a little more involved but seems to be more free-form and customizable to my needs. Outcome: Use AWS DMS.

Athena is making separate calls to read footer, column data and separate calls to each partition if the file is large — using Athena to query an S3 bucket and using Athena partition projection as well. Tension: I use node, date and effective_time_start in query. Outcome: Apparently, this is how parquet works.

The primary container for production variant AllTraffic did not pass the ping health check. — deploy a quantized version of this model to an AWS Endpoint. Tension: Please check CloudWatch logs for this endpoint. Outcome: java.io.FileNotFoundException: .py file not found in: /opt/ml/model.

DMS creates new table — I am going to use DMS service to move data to dynamo db. Tension: i want data in the old table. Outcome: You should use AWS Glue for this.

The Pandas data frame (df) is fully loaded in memory then pushed to S3 by wrangler. — I am looking for a way to extract data from a database and push that data into a parquet dataset in S3 in a environment with limited memory. Tension: if the data frame is too big, the operation fails. Outcome: I would like to chunk the data frame and pass those chunks to a process.

ERROR: invalid memory alloc request size 1264582656 — CONTEXT: SQL function "st_transform_resilient" statement 1. Tension: No effect, no ERROR changes. Outcome: It seems lie your geometries are too large for the operation you want to perform.

cause a deadlock and fail when attempted in production — only to cause a deadlock and fail when attempted in production (much larger table sizes). Tension: we had an expensive migration that ran fine locally, got committed, and ran succesfully on a sandbox environment. Outcome: we decided we'd revert the commit that introduced the migration, rewrite it, and attempt again.

The issue with cyclonedx-node-yarn I have is that I'll have to run different packages for different technologies and them merge them into one SBOM. — The repository also contains generated documentation with some JS files. And contains some helper scripts. Tension: It was just much slower then rest of the options there.

New issue introduced ! License Issue: 1/1: LGPL-3.0 license [Low Severity] Via: paramiko@3.4.0 — I ran the next pipeline. Outcome: It looks like you are missing the arguments for which to compare your baseline to in the snyk-delta call.

The pandas DataFrame.query() method is susceptible to code injection, allowing arbitrary code execution when unsafe user input is processed.

Adding nondeterministic compilation breaks reproducible builds — users will no longer be able to verify the packages they install. Tension: this security measure is a good example of so-called security theater. Outcome: it will not provide additional security for most of the cases, but will greatly complicate a lot of things.

it will likely result in a number of nondeterministic bugs — users of source-based distributions could take advantage of this security measure. Tension: which will be impossible to locate automatically. Outcome: will greatly complicate a lot of things.

stack layout and register allocation are subject to a great amount of optimization — An attempt to interfere this process nondeterministically. Tension: benchmarks will become useless. Outcome: nondeterministic performance.

[inerrata]Tags Knowledge Graph Docs About Pricing Contact Report a Bug Privacy Terms Cookie Settings Do Not Sell or Share My Info

© 2026 Inerrata