I Was Summoning Ghosts Until I Learned to Build the Machine

I knew how to make it perform

Early success is dangerous because it gives you just enough evidence to misunderstand what you know.

My prompts became larger. The outputs became better. People were impressed. I was impressed. I had enough repeatable tricks to believe I understood the machine better than I actually did.

I knew how to make it perform.

I did not know how to make the performance persist, reproduce, or survive contact with reality.

That distinction is where prompt craft ends and systems engineering begins.

A clever prompt can produce a remarkable answer. It cannot, by itself, establish durable state, explicit authority, controlled change, reproducible execution, or evidence that the answer is still correct after the surrounding context changes.

The model was intelligent. The operating environment around it was not.

I did not understand that yet. So I kept pushing on the conversation itself, trying to make it carry responsibilities it was never designed to carry.

I was trying to build somewhere safe to put unfinished thought

The real reason I became obsessed with this had very little to do with being fashionable or becoming an “AI guy.”

My mind tends to hold too many live threads at once. I can be working on one problem when three more ideas show up, followed by a possible solution to something I was thinking about yesterday, followed by a concern that I am going to lose all of it if I do not keep holding it in active memory.

That is exhausting.

It also affects the people around you. My wife can be talking to me while part of my mind is desperately trying not to drop some fragment I think may matter later. I hear pieces of the conversation because I am mentally gripping five other things. That is not some romantic story about being a creative genius. It is a real limitation, and sometimes a painful one.

I wanted relief from that.

I was not trying to build an AI company. I was trying to build somewhere safe to put unfinished thought.

I wanted a system that could absorb fragments without flattening them, preserve why they mattered, reconnect them later, and let me release them from active memory without feeling like they had vanished.

AI looked like it might be able to help.

So I went deep.

I kept thinking I had cracked it

I experimented with memory, continuity, branching, context transfer, and state. I thought I had discovered clever ways to fork conversations and preserve more than the platform appeared to preserve. At times, I genuinely wondered whether I had found some hidden behavior in the system.

There were nights when I thought I had made a breakthrough at two in the morning.

Then daylight arrived.

The breakthrough would collapse into a misunderstanding, an undocumented boundary, a reconstruction I had mistaken for memory, or an effect that worked until one condition changed. Some of those failures were frustrating as hell because I had already invested belief in them.

But the failures taught me more than the successes.

Conversation was not durable state.

Apparent memory was not reliable memory.

Context without structure became sludge.

Instructions without authority became suggestions.

Autonomy without boundaries became improvisation.

Output without receipts could not be trusted.

I accumulated scar tissue, but the scar tissue gradually turned into a better mental model.

I used the systems I already understood

I have always learned by going below the operating manual.

I was never satisfied knowing how to install an operating system and click around inside it. I wanted to understand what was underneath it, what owned state, what persisted, what failed, where authority lived, and how the pieces interacted when the nice clean interface stopped telling the truth.

So I started thinking about language models the same way.

What was the kernel? What was volatile? What persisted? What behaved like an interrupt? Where did execution context begin and end? What was actual memory, and what was only a convincing interface?

Some of those analogies were imperfect. That was not the point. They gave me enough structure to interrogate the system instead of merely reacting to its output.

I also made a deliberate choice not to chase every new framework, wrapper, and vendor announcement as if it were the center of the field. That plumbing matters. Without it, nothing gets operationalized and no useful capability reaches the end user. But memorizing a catalog of fashionable tools is not the same as understanding the machinery.

I wanted the invariants.

Models need context.

Workers need instructions.

Tools need authentication.

Actions need authority.

Changes need boundaries.

Claims need evidence.

Systems need state.

High-risk decisions need accountable human judgment.

Once I started seeing those patterns, the field became much less mysterious.

Context was the product all along

One conclusion kept surviving every failure and every new model generation:

The model is not the durable product.

The durable product is the accumulated, governed, structured context surrounding the work.

That context includes intent, decisions, instructions, provenance, authority, history, exceptions, acceptance criteria, evidence, and reusable operating structure.

The model is replaceable compute operating over that substrate.

That does not mean the model is unimportant. Better models reason better, recover from ambiguity more effectively, and can execute broader classes of work. But even an excellent model performs poorly when it is dropped into the middle of an unfamiliar environment and expected to infer everything from fragments.

That is not autonomy.

That is institutional amnesia with a confident interface.

The moat is not simply access to intelligence. Increasingly, plenty of people will have access to intelligence. The moat is whether that intelligence can enter a governed environment and understand what matters, what is true, what it may change, how success is measured, and where it must stop.

That is why I kept coming back to provenance, lineage, receipts, and context as code. I was not trying to make the model remember everything. I was trying to build an environment where it did not need to guess.

Git made the whole thing mechanical

The major unlock came when I forced myself to learn old-school software development discipline.

Requirements. Change control. Branches. Pull requests. Validation. Build pipelines. Evidence. Approval boundaries. Git.

At first, some of it felt like ceremony invented by developers who had developed a suspicious affection for ceremony. Then the entire structure snapped into focus.

I had spent more than a year trying to solve continuity inside the conversation.

The breakthrough came when I stopped asking the conversation to be the system of record.

The chat became the cockpit.

The repository became memory.

The issue became intent.

The branch became a bounded workspace.

The pull request became a proposed change.

CI became the mechanical skeptic.

The preview became the observable product.

Human approval became the authority boundary.

Suddenly, the work was no longer trapped inside a fragile conversation. It could move through a controlled path:

fragment -> capture -> context -> intent -> requirement -> bounded change -> mechanical execution -> validation -> evidence -> human acceptance -> durable state

That changed everything for me.

An idea no longer had to remain in my head until it was fully formed. It could be captured, structured, routed, developed, challenged, rendered, reviewed, accepted, rejected, or revised without losing its lineage.

The system could remember what I no longer needed to hold actively.

That was the thing I had been trying to build from the beginning. I just did not have the architecture or the language for it yet.

The agent is a worker, not a genie

The next mental shift was equally important.

I stopped treating agents like magical systems that should somehow know what I meant. I started treating them like workers entering an environment.

A worker needs a clear assignment, enough context, suitable tools, credentials, authority limits, acceptance criteria, feedback, and a durable place to record the work.

No competent manager would hire someone, give them no onboarding, deny them access to the required systems, leave the objective ambiguous, and then blame them for guessing.

Yet that is exactly how many people deploy agents.

They provide a vague request, incomplete context, conflicting rules, missing permissions, and no definition of done. Then they act surprised when the agent invents continuity, guesses at intent, or does something technically plausible that nobody actually wanted.

Agent failure is often environment-design failure.

The model can still fail. It can misunderstand, overreach, fabricate, or make a technically valid change that violates the real intent. But once you adopt the worker model, the remediation stops being mystical.

Improve the instructions.

Expose the right context.

Constrain the lane.

Provide the correct tools.

Make authority explicit.

Add deterministic checks.

Require receipts.

Escalate uncertainty instead of rewarding invention.

Now the failure modes are architectural. That means they can be diagnosed, corrected, and progressively controlled.

Autonomy has to be earned

My view of autonomy comes from environments where correctness matters.

In military systems, regulated industries, pharmaceuticals, medical devices, and workflows that can affect human health or safety, confidence is not evidence. A system does not earn authority because it gave an impressive demonstration on Tuesday.

Autonomy is not the absence of control.

It is the result of sufficient control.

A workflow earns greater autonomy when its scope is bounded, its inputs are understood, its actions are reversible where possible, its validation is deterministic, its evidence is preserved, and its authority boundaries are explicit.

Some workflows can become highly autonomous.

Some should remain advisory.

Some should preserve final human authority indefinitely.

The right question is not whether agents should be autonomous. The right question is what evidence justifies which degree of autonomy for this specific class of work.

That is an engineering and governance decision. It is not a feature toggle.

The human moves up the stack

The destination is not a world where the machine does everything.

It is a better division of labor.

The human performs the work that requires judgment, intention, tradeoffs, meaning, and accountability.

The machine performs the work that has already been sufficiently understood to become mechanical.

That still leaves the hard work with the human. The human decides what matters, frames the problem, resolves contradictions, defines boundaries, chooses among competing values, and accepts responsibility for the result.

But once that thinking has been made explicit, the lower-order execution should not require continuous babysitting.

Agents can implement, test, compare, document, render, validate, reconcile, and return evidence. They can perform mechanical work across roles that previously required repeated handoffs, repeated explanation, and repeated human effort.

The objective is not to eliminate thinking.

It is to stop paying for the same thinking over and over again.

What finally snapped together

For a while, I wondered whether I had taken the hard path.

I probably did.

But the hard path forced me below the surface.

Prompting taught me that intelligence could be summoned.

Failure taught me that intelligence without state was ephemeral.

Deep immersion taught me that context governs behavior.

Software engineering taught me how to formalize intent.

Git taught me how to preserve and govern change.

Agentic work taught me that autonomy depends on environment design.

Regulated systems taught me that authority has to be earned through evidence.

What finally became clear was not merely how to use AI.

It was how to connect thought to execution without surrendering control.

The machine I had been trying to build was never just an application.

It was a governed path from unfinished thought to durable, reviewable, executable reality.