Comet, AI Browser Operator
Design Engineer · 2026
“Two tasks that finish every time beat three where one breaks during the demo.”
- 01
Plain English in, a real Chrome window working through the task, live status per step over a WebSocket.
- 02
No hardcoded selectors and no per site scripts. One generic agent reads the page structure and figures it out.
- 03
Honest results. Every run ends as success, partial, or failed with a readable reason and a screenshot trail.
Comet is a natural language browser operator I built for a hiring take home. You type 'look up the weekend weather for Berlin' or 'find one way flights from New York to London under $600' and a real Chrome window opens and does it. A browser-use agent powered by Claude drives the page, a FastAPI backend streams one plain sentence per step to the UI over a WebSocket, and a reporter turns the raw run into a clean result card with a summary, the extracted data, and a screenshot trail of what the agent actually did.
Context
The brief asked for an agent that takes a natural language command and completes it in a real browser, with messy real websites called out explicitly. Cookie banners, slow pages, layouts that change, bot detection. The interesting work was not the agent loop itself. It was the judgment around it. What to build, what to skip, and how to be honest about what a live demo can promise.
Approach
I built on browser-use as the engine and spent my effort on everything around it. Configuration, live status, reporting, error handling, and scope. browser-use reads the page structure into something the model can reason about instead of guessing from screenshots, which is faster, cheaper, and more reliable for web tasks. After each step I read the agent's own state and send one plain sentence to the UI, so the live log is the agent thinking in real time. The final answer comes back in a fixed shape (status, summary, extracted data), so the agent has to say clearly whether it succeeded, partly succeeded, or failed instead of leaving us to guess from raw text.
Key decisions
browser-use over screenshot based computer use
Rebuilding the look, decide, act loop would take weeks and end up worse. browser-use already solves element finding, retries, and timeouts. Reading page structure beats guessing from pixels for web tasks, and web tasks were the whole scope. I treated it as the engine and owned the parts that make it a product.Sonnet by default, tested against Haiku
browser-use enables coordinate clicking for Claude 4 models, which makes the agent much better at hitting the right element. I tried Haiku for speed. It was faster per step but kept producing broken output and could not finish the harder tasks. Speed is not worth it if the agent does not finish. Model swaps live in .env, no code changes.Two guaranteed scenarios, not three flaky ones
Weather and flights cover the full range. Navigation, typing, dropdowns, date pickers, dynamic results, cookie banners. They run on sites that allow automation, so they finish reliably. Restaurant booking and Amazon are login gated and bot protected, so I refused to promise them. The agent still tries them and stops gracefully with a partial result. Knowing what not to promise is part of the job.One generic agent, zero hardcoded selectors
No per site scripts. The same code path handles every scenario, wrapped with safety rules. A moved button is just another page for the agent to read, not a broken script. Adding a third scenario is one line of prompt, not new code.What I chose not to build
CAPTCHA solving and login flows. Those are not problems a take home should pretend to solve. A stealth cloud browser sits behind a single flag for sites that block local automation. I also skipped mocking the agent loop in tests. Unit tests cover the deterministic logic (config, reporter mapping), and the live loop is verified by running it and watching. Mocking a non deterministic agent on the live web would test the mock, not the agent.
Result
Both guaranteed scenarios finish reliably in a live run. Blocked tasks end as a clear partial or failed message with screenshots of how far the agent got, never a raw crash. The whole thing is two commands to run locally.