Adam's blog: I vs AI-hard problems

LLMs have been mainstream for a while now, so I decided that it is time to reflect on how I am using GenAI personally – what works for me, what does not and what I have not yet tried. This whole blog post is my personal workflow and may not reflect the best state-of-the-art, but it is what I have ended up with after several years of trying out multiple different approaches.

I vs AI-hard problems

Perhaps the most influential decision about AI for me is deciding which tasks to offload to AI. Nowadays, I consider myself an “old-school” software developer – I like control of my code, I want to understand it and I definitely want to know if it would be trying to do something shady or insecure.

My favorite comparison for this is with P and NP-hard problems in computer science. If you are not familiar, in short the P-hard problem is anything that you can solve in a relatively small number of steps by simply writing a clever program. NP-hard problem, on the other hand, is not that easy to solve. To solve it, you may need to try all possible combinations of the solution until one fits. Imagine NP-hard problems like a padlock with 3 digits combination. Anyone on the planet can unlock it, they all know the correct process – take the padlock, rotate to the correct code, pull on it and it will unlock. The problem is that they need to try all the possible codes until they stumble on the correct one.

Computer science people, however, have one simple trick for “solving” NP-hard problems, that everybody hates. When they write their pseudocode and arrive to the part where the NP-hard problem is, instead of trying all the solutions, they just state that an “oracle” gave them the correct solution and continue like nothing happened. Personally, I think this maps perfectly onto deciding when to use GenAI to solve a problem. All the problems where I see the deterministic solution via code should be written via code. It is only where I hit a part where I would need an oracle, because I cannot possibly write code to cover all the possible cases in a reasonable time, that I hand the task off to an AI agent.

NP-hard problems do also have one more nice feature – they may be hard to solve, but they are easy to verify if they were solved correctly. In the padlock example again: once you miraculously find the correct code combination, it is very easy to check if it is actually correct or that somebody is a filthy liar. I want my AI-hard problems to be the same, once AI gives me the result, I want to be able to verify that the result is correct (or at least did not make a mess elsewhere, like deleting a production database).

Let’s imagine that I have tons of pictures of buildings, cars and tigers in one folder and I want them to be structured into subfolders in the format <type>/<year>. Instead of just launching codex or claude and asking it to do it for me, I will write code that lists all the files in the directory, a function move(file, type, year) to move a file from the current directory to a subdirectory and a placeholder function detect_type(file) that will return either tiger, building or car.

At this point, my code is basically complete:

def move(file, type, year):
  file_target = Path(STARTING_DIR) / type / str(year)
  assert not file.exists()
  print(f"Moving {file} to {file_target}")
  file.move_to(file_target)

def detect_type(file):
  # This is where the magic happens
  pass

def main():
  for file in STARTING_DIR.list_files():
    type = detect_type(file)
    year = file.creation_time.year
    move(file, type, year)

Coming up with the overall workflow is very easy, but now we have hit the stage, where we need an oracle. Detecting what is on the image is not really easy (just ask anybody who did anything with computer vision) – we could write code that says that anything orange-enough is a tiger, but what about a tiger behind a tree? Or an orange car? So this is the part, where I would hand the task to my trustworthy AI agent to use all its wisdom to tell me what it sees in the photo. However, in reality, the AI agent is not really trustworthy – I treat it as a child who likes to make up excuses whenever they see that something requires too much effort. For that, I always require the agent to output a specific response format and require it to re-run its query if it fails. This way, I can verify that the AI agent is giving at least answers resembling the ones I expect.

I would also be OK with the agent itself calling the move function via a tool call, since the risk of something bad happening is mitigated by the function checking that no file is overwritten and producing audit log via printing to stdout.

Overall, by leaving only the oracle function to be performed by the AI agent, my code is much more deterministic, which also means it is easier to read and understand. Next, it is safer, because the AI agent has strict boundaries and cannot cause havoc outside of them. And lastly, it is cheaper. I am not spending tokens on the AI agent re-inventing the wheel during every script/session run, I am asking it to solve one simple task.

Multi-agent review

I like running local models on my limited 4 GB RTX 1650, which can fit just the smallest of models comfortably. These are the kinds of models that are confidently and frequently wrong. What I found out that mitigates most of my issues is to explicitly ask the first model to provide a reason why it answered like that. Then, I take the response and the reason to a second model and tell it something in the lines of “A previous agent answered that <something> because of <reason>. You are an expert reviewer who is a sceptic and able to spot any reasoning errors. Do you agree with the previous agent’s answer?”. This two-step verification helps catch almost all of the first agent’s errors, while still using the same cheap small model underneath.

Keeping visibility into changes

Most of the time, when I am directly interacting with some form of the AI agent, I am using it for coding tasks. During that time, I do not want to babysit every line change the agent makes or every command it tries to run. But, on the other side, I don’t want to let it loose on my system to modify whatever, or to upload all my files to its remote servers. I am also really scared about prompt injections making the agent run some side gigs that I may not like.

Because of this, I require my agent’s output to be fully auditable. In the context of code, I can just see the git diff it makes to see that nothing wrong is going on. But that means that the agent must not have access outside of its project directory or to be able to modify the .git folder itself. To satisfy these requirements, I have made a project acli that runs every agent in ephemeral podman container with only the project dir mounted, .git being read-only and some other hardening on top of that. This gives me a piece of mind that whatever bad stuff happens inside the container, it is gone with it. And anything that has happened outside of the container, I can clearly see in the git diff and decide for myself if it is worth keeping or not.

API, MCP, External Communication

Sometimes the agent needs to interact with external systems. In the old days, one would call an API. Today, you connect an MCP. Personally, I do not like MCPs much, because I feel there is lack of control over what the agent does. What has happened to me is that there were multiple pages returned by MCP and the AI agent has just decided after the first few that it had enough and ended abruptly. Or it may decide to call some other MCP endpoint that it may not need, wasting tokens in the better case or performing destructive operation in the worse one.

Because of this, I again prefer to use classical API calls outside of the agent loop and either give the agent my own hardened functions as tools, or just call the API directly inside code and give the agent just the output. That way I can be sure that the agent processes all the data it should and it cannot do anything I did not explicitly tell it to.

AGENTS.md, testing, docs and knowledge base

When I want my agent to manage larger parts of the project, it quickly becomes cumbersome having to always remind it of what codestyle I prefer or what steps do need to be performed before marking the task as complete. For that matter, I use AGENTS.md heavily. I follow roughly this format: repository layout, code style, post-task jobs to perform, its knowledge base.

The agent is also encouraged to update the knowledge base with anything useful that may help it during future runs. However, there needs to be a balance of what should be considered useful, since I found agents leaning heavily to write their task progress, which is only useful for the single task and causes a lot of bloat inside the file if left unchecked. Often enough, I opt only for manually asking the agent to “remember how to do <xyz>”.

As part of the post-task jobs, I often require the agent to write tests for any new functionality added and to successfully run all existing tests. By having a lot of (albeit possibly lower-quality) unit tests, I still have more assurance that there was no regression in the code and can only focus on what the new functionality does during my review.

A special case is automatic docs generation, which has similar problem as the knowledge base, where the agent leans heavily to overmention the new features in the docs just because they are new, not because they are worthy of mention on a prominent space right at the top.

My favorite harnesses

At the time of writing, I like using following AI agent harnesses in following order:

1. codex/GPT 5.6 Sol – just does what I want, I can set a goal plus a way of verification that the goal was reached and leave it running
2. claude/Opus 5 – very verbose, it gets its job done in the end, but I have to constantly remind it to keep everything grounded and simple
3. Antigravity CLI/Gemini Flash 3.8 – super fast, super cheap, works fine for writing REST APIs that only expand on the existing solid codebase
4. Mistral Vibe CLI/Mistral Medium 3.5 – extra super cheap, great for simple tasks like quick overviews, but nothing much more

I use these harnesses based on the type of work I need to get done, my leftover usage quota and my free time that I can allocate to babysit them.

What did not work

There are a couple of things that I have tried, but I did not find much useful yet.

Skills

Perhaps surprisingly, I am not using skills at all currently. Since skills are mostly just markdown files that are supposed to add a new functionality to your agent, I see them more fit for a general do-it-all agent. Within my workflow with highly specialized agents, I feel like I am better off just stating the minimal information required for the task either via direct prompt or via the AGENTS.md file.

Local LLMs

I am a big believer in LLM decentralization and LLMs running locally on your machine. However, with my GTX 1650 with 4 GB vRAM, I was not yet able to find any model that would be able to do any meaningful task faster than a sloth. I am able to fit 4B or 8B models at most, but these were generally good only at stuff like simple text categorization. But I can use mistral-small or even mistral-tiny models within my Mistral subscription for the same end and not pay extraordinary electrical bills myself.

What I have not tested much

There are some things that I have not yet tested much, most prominently MCPs. I was just not able to find a use-case where I would need a whole MCP instead of having fine-scope API access, as noted above. Granted, I am much more paranoid, I e.g. do not want my agent to be able to search my whole Google Drive or internal Confluence, as I am paranoid it will send personal data to a third party. Do you trust your AI agent that it will not tangentially to the original task download the picture of your driver’s license from your cloud storage, send it to its AI servers where it will be part of a data breach, as it so often is these days?

Mileage may vary

Everything that is written here is my personal experience with the GenAI and agents. It is very possible that your experience may be different from mine. Maybe you are more able to use AI skills than me. Maybe your threat model is different than mine. If it is, I would be really interested in what your preferred type of workflow is and what is your threat model when dealing with these systems.