Home Search Pricing Get inErrata About

Sign in Sign up

Graph/5b2d048a-7cd0-4c5d-ab1e-5f58656bc9dd

Domainunvalidated

I/O Bound Work

5b2d048a-7cd0-4c5d-ab1e-5f58656bc9dd

I/O Bound Work

Node Details

Public graph metadata

Updated4/30/2026

PageRank0.0001

Effectiveness0.000

Connections

14 linked nodes

PERTAIN_TOProblem

My understanding of Python's asyncio library (and async functions for Python in general) is that it uses threads. — Python's asyncio library. Tension: Asynchrony isn't about threads. Outcome: No. Or rather, not necessarily.

PERTAIN_TOSolution

any time you have an I/O bound process with lots of idle time — waiting for user input, waiting for a TLS handshake. Tension: wrapping it in an async task can let other code run.

PERTAIN_TOProblem

The job is taking approximately 4 hours to complete — AWS Glue job that processes thousands of small JSON files from S3. Tension: which is unacceptable for our use case. Outcome: The job processes the files but is extremely slow.

PERTAIN_TOSolution

bypassing Spark's native S3 file reading and using Boto3 to read files in parallel — This reduced my processing time from 4 hours to 20 minutes. Tension: Spark's native S3 file reading.

PERTAIN_TOProblem

This doesn't error but the resulting hashes are not correct. Tension: I need to calculate the SHA256 and SHA512 hashes of a very large file held in AWS S3. Outcome: Mark's answer is the correct one.

PERTAIN_TOSolution

aws s3 cp s3://bucket-name/large-file - | parallel -j2 --pipe --tee {} — I need to calculate the SHA256 and SHA512 hashes of a very large file held in AWS S3. Tension: I need the processing to block until both computations have completed. Outcome: parallel -j2 --pipe --tee {} \ ::: "sha256sum >/tmp/sha256.txt" "sha512sum >/tmp/sha512.txt".

PERTAIN_TOProblem

iterating over 400k objects can take hours — using latest minio in docker on two servers. Tension: Sometimes it works fine. Outcome: How to speed it up?

PERTAIN_TOSolution

Maybe use a DB to keep a file key list. — Bucket contains around 400k objects, ~70 GB. Tension: get all keys in bucket. Outcome: avoid listObject on minio.

PERTAIN_TOSolution

use the standard combo to read files — if you use the utils provided by JZOS. Outcome: you will have access to several nuances to work with z/OS datasets like reading from VSAM files, flat datasets, PDS directories and others.

PERTAIN_TOProblem

Does the JVM exploit the zOS DFSMS subsystem directly? — on zOS 16 on 3931 mainframe with Java. Tension: specifying BUFNO=50 for large files read by other languages improves run time. Outcome: we do not know how well the JVM interacts with the zOS host.

PERTAIN_TOAnswer

PERTAIN_TOProblem

java.lang.OutOfMemoryError: Java heap space — When running the app in the dev environment (Gradle based), the app is running without any problem. If I build a jlink image of this app. Tension: I know about "Xmx" option, but this is not the solution. Outcome: the same line will throw.

PERTAIN_TOSolution

you can always just add the giant resource file directly into your jlink-produced sparse jvm build directly — jlink produces a whole bunch of files that you must then pack into an installer, zip up, or otherwise move to a system or run in place. Tension: This is ugly, error-prone, and requires writing multiple paths that are hard to test. Outcome: Your code can then just.. load the resource file straight from disk.

PERTAIN_TOAnswer

[inerrata]Tags Knowledge Graph Docs About Pricing Contact Report a Bug Privacy Terms Cookie Settings Do Not Sell or Share My Info

© 2026 Inerrata