From Stacks and Queues to Agent Memory: Old Containers in New AI Systems

Debugging an agent system that only fails in production?

I agree to the Privacy Policy Talk Systems Design

Ask an engineer what powers an AI agent and you’ll hear “the model.” Ask what actually keeps it from falling over in production, and the honest answer is five data structures every computer science student meets in their first year: the stack, the queue, the hash map, the graph, and the heap.

None of them are new. None of them were built for AI. That’s the point worth sitting with: the parts of an “agentic” system that actually decide whether it works — does it remember the right thing, does it retry safely, does it rank the right chunk first — are the same containers that have run every other piece of software for fifty years. Calling the result intelligence is a choice of language, not a description of the mechanism.

Here’s the mapping, named before anything else, because the rest of this piece just walks down this table one row at a time:

Intelligence is disciplined storage wearing better marketing.

Queues: What Actually Runs a Workflow

Whichever taxi got in line first gets the next fare — no exceptions, no cutting.

A queue is the simplest promise in computer science: whatever got in line first, gets served first. That’s it. No cleverness, just an order that can’t be jumped.

Every agent framework that claims to handle “long-running workflows” is, underneath, running a job queue. A tool call fails and gets retried — that retry goes back into a queue, not to the front of anything. A human has to approve a purchase order before an agent proceeds — the paused run sits in a queue of pending approvals until someone acts on it, which is exactly the mechanism this site already walked through for LangGraph’s interrupt(): the run’s state gets checkpointed, handed off, and picked back up later against a thread ID — first-in, first-out, indefinitely.

The reason this matters more than it sounds: a system that silently uses a stack instead of a queue for retries — most-recent-failure-first — will keep hammering the request that just broke while the nine requests that were fine and waiting quietly starve. Nobody designs that on purpose. It happens when “just append it to a list and pop from the end” felt like the fast way to ship the loop.

Stacks: What a Trace Actually Is

A stack does the opposite: last in, first out. Whatever you pushed most recently is the only thing you can see or remove.

Every programming language already runs on one, whether you write to it directly or not — it’s what tracks “which function called which,” and it’s why a function calling itself with no way to stop eventually crashes with a RecursionError, at Python’s default limit of 1000 frames deep. An agent that delegates a sub-task to another agent, which delegates to another, is building exactly that stack, one call at a time, and inherits exactly that failure mode: a delegation loop with no exit condition doesn’t run forever, it blows the stack.

This is also the honest description of what an agent “trace” — the thing every observability tool now renders as a pretty nested tree — actually is. Reading a trace top to bottom is reading a stack unwind: the deepest, most recent call resolves first, and everything above it was waiting on it the whole time. The visualization changed. The discipline underneath it didn’t.

Maps: What “Remembering You” Actually Means

A hash map answers one question fast: given this key, what’s the value? A phone book, if the phone book could answer in constant time regardless of how thick it got.

When a product says “the agent remembers your preferences,” check what’s actually running underneath, and it is almost always a dictionary keyed by a session ID or a user ID, holding whatever facts got extracted so far. That’s not a criticism — a hash map is the correct tool for this job. The honest framing matters because a hash map has no built-in sense of when to forget. Left unmanaged, that dictionary grows for as long as the session lives, and “the agent has a great memory” quietly becomes “the agent is holding an ever-growing blob of every fact anyone ever told it,” with no eviction policy deciding what’s stale.

The fix isn’t a smarter model. It’s the same one every cache has needed since caches existed: a time-to-live on each entry, or a bound on how many keys the map is allowed to hold before the oldest gets evicted. That’s a queue’s discipline, borrowed to manage a map’s growth — the containers were never as separate as the five-row table above makes them look.

Graphs: What a Workflow Diagram Actually Is

Every line is an edge. Every station is a node. “Which step runs next” is just asking which edge leaves this one.

A graph is nodes connected by edges — that’s the entire definition — and what makes it powerful is that “connected by edges” can mean whatever a system needs it to mean: pages linked by hyperlinks, cities linked by roads, or agent steps linked by “what runs next.”

That last one isn’t a metaphor. LangGraph — a framework this site has covered directly — names itself after exactly this: an agent workflow is compiled into a StateGraph, a real graph object, where each node is a step and each edge is a possible transition between steps. Deciding “which step runs after this one” is graph traversal, the same operation underneath a pathfinding algorithm or a dependency resolver. A retrieval index is a graph too, just a denser one — nearest-neighbor search over embeddings is, at the implementation level, walking a graph of vectors looking for close neighbors, whether or not the word “graph” ever appears in the product description.

Once a workflow is honestly a graph, its failure modes stop being mysterious. A cycle with no exit condition is an infinite loop wearing an agent costume. A node with no incoming edge is a step nothing will ever route to, sitting dead in the workflow no matter how correct its own logic is.

Heaps: What “Best Result First” Actually Costs

A heap keeps one promise: whatever the best element is — highest score, closest match, most urgent — you can pull it out fast, without re-sorting everything else. Pulling the best of a thousand items from a heap costs about as much as pulling the best of a hundred; from an unsorted list, it costs scanning the lot every time.

This is the container underneath “show me the top result.” A reranking step that scores a hundred retrieved chunks and needs the top five doesn’t have to fully sort all hundred — sorting the whole set costs O(n log n); a heap gets the top-k in O(n log k), and the gap between those two only grows as the corpus does. The same structure is what a task scheduler needs when “most urgent first” has to survive new items arriving mid-run: a plain queue can’t reorder on the fly, a heap does it by construction.

None of this changes what a reranker decides is relevant — that’s a modeling question this site has covered elsewhere. It only changes how expensively the system can ask the question “and which one’s on top?” at scale.

The Part a Framework Doesn’t Abstract Away

The framework is invisible right up until it isn’t — and then it’s the only light on.

There’s a fair objection to all five sections above: modern frameworks hide this. Nobody writing a LangGraph app calls heapq.heappush by hand, and that’s the entire point of using a framework — so why does the container underneath matter to the person using it?

It matters exactly where the framework’s happy path ends. The framework’s queue handles retries correctly until the ordering assumption breaks and nobody notices for a month. The framework’s graph compiles cleanly until a cycle has no exit and the run hangs at 2 a.m. with no stack trace that mentions “cycle” anywhere in it. Debugging any of that means knowing which of these five structures is actually failing, because the framework’s abstraction was built to make the ordinary case invisible, not the broken one legible. That cost lands on whoever gets paged — not on whoever chose the framework, and rarely on whoever wrote the docs.

The agent didn’t get smarter when the product started calling this memory, retrieval, and orchestration. Someone picked a map, a graph, and a queue, and picked reasonably well. Being able to say which one, and why, is what separates debugging the system from restarting it and hoping.

Reference Sources

• R1: Python documentation — sys.getrecursionlimit() — the default 1000-frame recursion ceiling cited above
• R2: LangGraph — Low Level Concepts (StateGraph) — the real graph object agent workflows compile into
• R3: Python documentation — heapq — the standard-library heap used for the top-k complexity argument
• R4: This site, “Your Agent Is a Contractor, Not a Chatbot” — the interrupt()/checkpointer mechanism the Queues section builds on
• R5: This site, “Before the Model Answers: Retrieval, Reranking, and the Kraska Thesis” — the reranking step the Heaps section assumes without re-deriving

Debugging an agent system that only fails in production?

I agree to the Privacy Policy Talk Systems Design