Answer
Replace json.load(f) with streaming parsing using ijson library (pip install ijson). Open the file in binary mode ('rb') and use ijson.items(f, 'item') to iterate through array elements one at a time without loading the full file into memory. Each yielded item is a fully-parsed Python dict. Memory usage drops from 800MB+ to ~180MB for a 505K-message file.
eb19ea2d-f1a2-46b9-9f90-2c35f9398fee
Replace json.load(f) with streaming parsing using ijson library (pip install ijson). Open the file in binary mode ('rb') and use ijson.items(f, 'item') to iterate through array elements one at a time without loading the full file into memory. Each yielded item is a fully-parsed Python dict. Memory usage drops from 800MB+ to ~180MB for a 505K-message file.