Curriculum by critic uncertainty: training a web agent on one GPU

The constraint is the problem statement

Training a web agent with RL is expensive: the environment is slow, episodes are long, reward is sparse. The usual answer is a cluster. The condition for this work was the opposite: a single consumer GPU with 16 GB. Everything else in the architecture follows from that.

Hierarchy: frozen planner, trainable executor

Qwen3-8B (frozen, 4-bit NF4)         ← planner, sets a subgoal
        ↓ subgoal (JSON)
DeBERTa-v3-small (frozen)
        → Transformer (2 layers, 4 heads) → GRU(256)
        → action type + pointer attention (which page element)
        ↓ CLICK / TYPE / SCROLL / SELECT
BrowserGym / MiniWoB++

Only the executor is trained - a few million parameters instead of eight billion. The planner and the encoder stay frozen in 4-bit quantization. Element selection uses pointer attention rather than classification, because the number of elements on a page is not known in advance.

Idea 1: a curriculum driven by value variance

A classic curriculum moves from easy to hard in an order fixed in advance. The problem is that "hard" is defined by the author, not by the agent.

Here the order comes from the critic. For each task the variance of its value estimates is tracked, and tasks are sampled more often the higher that variance is. High variance means the agent does not understand what its actions are worth on that task - which is exactly where training pays off most.

The spread between tasks turned out to be 2740×: from zero on click-link, long since solved, to 0.00274 on social-media. So the mechanism really did redistribute the training budget rather than spreading it evenly.

Idea 2: Reward Fusion instead of binary success

A binary reward on long episodes is almost always zero, leaving the gradient nothing to hold on to. The reward is assembled from four terms:

R = 0.5·ORM + 0.3·R_DOM + 0.2·R_waypoint − penalties

• ORM - a judge model's score of the final state;
• R_DOM - a programmatic DOM check: is the field actually filled, the dialog closed, the checkbox ticked;
• R_waypoint - intermediate milestones inside an episode;
• penalties - for redundant actions and invalid transitions.

The run logs contain 100 distinct reward values ranging from −8.4 to +0.92. That is the check that the scheme works: the reward really is graded, not a binary signal in disguise.

Results

Metric	Value
Mean SR (phase 3)	42%
Final SR (last 10 iterations)	61.7%
Tasks with non-zero SR	20 of 25
Environment steps	50,000

Best tasks: click-dialog and focus-text (100%), click-button (60%).

What did not work

An honest list matters more than the table.

• choose-date / book-flight: the jQuery datepicker accumulates element ids as it redraws the DOM, which makes long navigation unreliable.
• enter-date / enter-time: three inputs share a single identifier, and the built-in fill() does not work in the current environment version.
• email-inbox / search-engine: these tasks demand reasoning rather than element selection - the ceiling of a pointer policy itself.
• PopArt: value-normalization statistics are not stored in the checkpoint and get recomputed when training resumes.

The first two are environment version drift, not a property of the method. The last two are genuine architectural limits - and they set the direction for the next round of work.