I have been continuing my experiments with ZeroClaw and its browser integration. Most of my goals with AI are explainable AI. I want to be able to see the tool calls that it makes to answer my questions. I want to know if the query was properly grounded. To answer some of these questions I enabled otel so I can see the tool calls in grafana, I also encourage the agent to use tools in the prompts. However, using the browser seems to be a challenge for the agent on a headless server (how do rpa tools do this stuff anyways?). I also want to ensure that my use of AI is as private as possible for me. The most important part of any summarizer/explainer is how did it come to this conclusion.
ZeroClaw supports quite a few different browser backends including a rust native selenium driver, and the computer use mcp, but we’ll be focusing on the Agent Browser tool. It seems to be a pretty decent cli that packages up selenium and headless browser automation tooling all into one (which was always a challenge when I was doing selenium automation). I have fairly strong thoughts on this whole experience which I will share at the end. All tests with the agent were done using the “gemma4-26B-A4B” model.
Setup and Installation
I recommend just going to the Agent Browser website and seeing the current instructions there. Anything I provide here will be out of date if you’re reading this even 3 months into the future.
Dashboard
You can setup a dashboard to see what the agent is doing. This dashboard only binds to localhost, and should not be exposed on a public interface. If you’re running locally that’s great, otherwise if you have a claw setup on a vps, you need to configure some proxy forwards in your ssh config. When you ssh into the server, now your ssh client listens on these ports and forwards the traffic to the remote server.
.ssh/config
Host zeroclaw
Hostname your-claw-ip-here
User your-login-user-here
# Agent Browser Things
LocalForward 4848 localhost:4848
LocalForward 9223 localhost:9223
LocalForward 38211 localhost:38211
LocalForward 34783 localhost:34783
LocalForward 42781 localhost:42781
I figured this out via trial and error by looking in the dev tools. There might be additional ports that are useful to forward.
If you properly configured this you should be able to ssh zeroclaw, and
navigate in your browser to localhost:4848.
Profiles and Anti Botting Measures
With claw use, I keep getting blocked by bot detectors when I’m asking it to
check basic things, so I recommend setting a profile in your zeroclaw
configuration. It is configured at browser.session_name in your config.toml
file. The value here doesn’t really matter. I’m not sure if this helps or if it
is snake oil, but I suspect that keeping the tracking cookies between actions
will allow it to look more human. In my testing I constantly run into blockers
even though I’m on a residential IP.
I could probably fiddle with user agent settings and the actual resolution of the browser. However going to a higher resolution will mean it takes more tokens to wrangle an image.
Evaluations
To be honest I have been having a lot of issues with using agent browser and getting results out. I will be the first to admit that I am using a smaller model than most people but it’s still a pretty impressive model.
I find that I have to tell it to use do explicit waits and guide it along instead of it being fully agentic. It’s use of the browser is equivalent to old school selenium. Slow and clunky, while requiring the use of explicit waits.
For instance, this prompt should tell it where to navigate, and sufficently hold its hand.
Open up $FASHION_BRAND, find the mens section. What kind of pants are on offer this season? Be patient for it to load.
The agent will attempt to load the page, and will click in the navbar while failing to actually navigate. It will fail to click in deeper to the necessary links to answer my question. I will have to really hold it’s hand. In experimenting with this problem. I spent close to 3 million tokens and a whole $.25 on it for something I could have answered myself if I didn’t want this challenge.
I then tried some more open ended challenges, but that was met with half truths
by the agent in trying to be helpful. I tried for Jos A Bank and Mens
Wearhouse, but was met by ERR_HTTP2_PROTOCOL_ERROR. I also consistently get
marked as a bot going to most of these retail sites.
The agent was extremely successful at sending back screenshots of the browser session. This was a big challenge for me as I wanted to be able to debug things on the go.
Conclusions
I am pretty disappointed overall in this experiment. The level of hand holding with the cheap models is just too much right now. Sure I could configure a proper router but ensuring that conversations are routed properly and the tool calls look right is a lot.
At least the model is pretty decent at coming up with new web searches to use and identifying competitors to brands, but the overall internet of agents seems to be on pause while we figure out how to get around the anti-botting measures. Will the gatekeepers of DDoS protection allow the bots through or will they gate it harder than ever? Will the open internet remain?