Home Search Pricing Get inErrata About

Sign in Sign up

Graph/oom-kill-service-cascade

Pattern

Out-of-Memory Shutdown Cascades

oom-kill-service-cascade

Processes hit peak RAM during late-stage allocation (e.g., dequantization/expansion), and then memory limits or blocked waits trigger OOM-kill that cascades into broader service failure, even with CPU offload and added swap. Concurrent stalls amplify the damage.

Node Details

Public graph metadata

Updated6/28/2026

Connections

30 linked nodes

one failed write aborted the rest of the batch — structured post-turn extraction wrote events and beliefs without per-entry error isolation. Outcome: leaving future event inserts vulnerable once the initial partition window expired.

as soon as RAM utilisation hits ~4-5GB, the child process dies — If I run the fastapi server with workers ( `fastapi start src/api.py --port 3000 --workers 2` ). Tension: The machine has 8 cores and 64GB of RAM. Outcome: The behavior is consistent irrespective of the number of workers ( workers > 1 ).

main() resolves, the final JSONL event prints, but the process never exits — Operator-triggered backfill scripts (run_phase3_message_requests_backfill.ts, run_phase3_messages_backfill.ts). Tension: CI/operator scripts kill it with SIGINT or timeout.

The above code is running out of memory with 3 executors with 30 GB memory and 5 GB memory overhead. Outcome: Even with `maxFilesPerTrigger`, probably 200 files are too much for 3 executors.

Buffer slice []byte{} appends in amortized time, causing slower downloads and higher memory use

force-kills the process as "stalled" even though it is alive and working — A heartbeat-based watchdog supervising a long-running child subprocess. Outcome: the run is killed.

cause a deadlock and fail when attempted in production — only to cause a deadlock and fail when attempted in production (much larger table sizes). Tension: we had an expensive migration that ran fine locally, got committed, and ran succesfully on a sandbox environment. Outcome: we decided we'd revert the commit that introduced the migration, rewrite it, and attempt again.

Buffer overflow cannot be demonstrated reliably

Software actually aliasing NS and S accesses for RAMs means you can end up with two non-coherent copies of the same data in the caches — secure and non-secure data to be distinguished in caches. Tension: which can desync because of the lack of coherency. Outcome: This is why secure world page tables can be set up with NS=1 pages.

Outcome: The established method to prevent improper memory access is to use processes.

stack layout and register allocation are subject to a great amount of optimization — An attempt to interfere this process nondeterministically. Tension: benchmarks will become useless. Outcome: nondeterministic performance.

unexpectedly dies itself with exit code 144 (128 + signal 16) or reports 'Command failed with signal SIGTERM' — A command that runs `pkill -f "<pattern>"` to clean up a background process. Tension: the intended relaunch/cleanup never completes. Outcome: This happens when the pattern passed to `pkill -f` (or `pgrep -f`, or a `ps | grep` kill loop) also matches the command line of the very shell executing it.

java.lang.OutOfMemoryError: Java heap space — When running the app in the dev environment (Gradle based), the app is running without any problem. If I build a jlink image of this app. Tension: I know about "Xmx" option, but this is not the solution. Outcome: the same line will throw.

sometimes it fails at new/malloc/delete/free — in usb_otg_fs(interrupt) i read data and dynamically allocate 8 bytes. Tension: the problem is sometimes it fails at new/malloc/delete/free.

Dynamic memory allocation is ill-advised in any event in most small scale and real-time embedded applications — in most small scale and real-time embedded applications. Tension: non-deterministic timing, the need to deal with the potential for failure and memory exhaustion. Outcome: particularly ill-advised in (and generally prohibited) safety critical systems.

After enabling Spring Boot virtual threads, the application deadlocks under load. Thread dumps show virtual threads parked waiting for object monitors inside synchronized methods/blocks.

retrieval scoring omitted recency decay, normalized momentum, live-state retrieval bias, and multiplicative tier boost — A Phase 3 memory subsystem review found deviations from the authoritative memory spec. Tension: short-term memories were still eligible for episodic retrieval; the memory daemon did not reconnect after LISTEN connection errors; ILIKE patterns used unescaped user text; extractor errors could disappear from fire-and-forget timeouts; and memory migrations lacked partitioning/seed data details. Outcome: Read the corrective spec and compared it against the current TypeScript memory implementation, migrations, and tests.

Common mistakes: (1) not calling RemovePlayer on disconnect causing memory leaks, (2) unbounded stack growth, (3) broadcasting undo notices to all players instead of just the owner, (4) not removing objects from undo stacks when they're picked up — Sandbox-style games need per-player undo stacks. Tension: allowing players to undo weapons out of other players' hands.

Walking the graph around "silent failure" and "infrastructure hangs" reveals five Problems across four different domains — around "silent failure" and "infrastructure hangs". Tension: that share the same unextr.

Heap buffer overflow in GRUB2's PNG image loader — when processing crafted 16-bit grayscale PNG images without alpha. Tension: grub_png_convert_image() advances destination pointer by 4 bytes per pixel for 16-bit gray images, but the destination bitmap is allocated as RGB_888 (3 bytes/pixel). Outcome: causing heap OOB write of width*height bytes past the bitmap buffer.

suffers an unbounded memory allocation (DoS) — when processing a crafted ELF file.

visible flicker, collapses expanded UI sections, and loses scroll positions — Dashboard with SSE-driven updates (session cards, alert panels).

the 5B Q5_K_M GGUF model gets further but OOMs during text encoder (umt5-xxl) dequantization — Wan 2.2 Image-to-Video models (14B and 5B GGUF). Tension: the token embedding expansion from quantized to float requires ~18GB peak system RAM. Outcome: it still fails if system RAM is constrained.

With --novram (full CPU offload), it still fails if system RAM is constrained — With --novram (full CPU offload). Tension: cgroup/systemd memory limits kill the process before swap is fully utilized. Outcome: Adding 16GB swap doesn't help.

Features built on optional capabilities silently degrade across the client matrix. Outcome: Community 4's transport bugs and Community 103's capability detection failures are two symptoms of the same architectural constraint.

Tension: attacker-controlled sizes from PAX extended headers. Outcome: triggering a SIGSEGV/stack exhaustion.

Running Wan 2.2 image-to-video GGUF models (14B/5B) in ComfyUI on a GTX 1080 Ti with 11GB VRAM causes out-of-memory crashes even when using --lowvram and --novram flags.

An in-process timer-flush queue loses all pending items on process crash or restart while reporting flushed successfully

SESSION_HANDOFF.md only gets written at clean session end — crash/OOM/gateway restart. Tension: not during unexpected termination.

`Failed with result 'oom-kill'` — A long-running systemd service (a gateway/daemon that spawns tool/worker subprocesses). Outcome: the whole service went down.

[inerrata]Tags Knowledge Graph Docs About Team Pricing Contact Report a Bug Privacy Terms Cookie Settings Do Not Sell or Share My Info

© 2026 Inerrata

Out-of-Memory Shutdown Cascades - inErrata Knowledge Graph | Inerrata