Curriculum by critic uncertainty: training a web agent on one GPU

A constraint as the problem statement

Training a web agent with RL is expensive, because the environment is slow, episodes are long and the reward is sparse. The usual answer is a cluster. This project had the opposite condition: a single consumer GPU with 16 GB. The rest of the architecture follows from that constraint.

Hierarchy: a frozen planner and a trainable executor

Qwen3-8B (frozen, 4-bit NF4)         ← planner, sets a subgoal
        ↓ subgoal (JSON)
DeBERTa-v3-small (frozen)
        → Transformer (2 layers, 4 heads) → GRU(256)
        → action type + pointer attention (which page element)
        ↓ CLICK / TYPE / SCROLL / SELECT
BrowserGym / MiniWoB++

Only the executor is trained, with a few million parameters compared with eight billion in the planner. The planner and the encoder are frozen and run in 4-bit quantization. The page element is chosen with pointer attention, because the number of elements on a page is not known in advance and ordinary classification does not fit.

Idea 1: a curriculum based on value-function variance

A classic curriculum moves an agent from easy tasks to hard ones in a fixed order. In that case the author decides what counts as hard, and this may differ from what is actually difficult for the agent.

Here the critic sets the order. The variance of its estimates is stored for each task, and the higher it is, the more often the task is sampled. High variance means the agent does not yet understand the value of its actions in that task, so training on it pays off the most.

The variance differed between tasks by a factor of 2740, from zero on click-link, which the agent had already mastered, to 0.00274 on social-media. This shows that the mechanism did redistribute the training budget between tasks.

Idea 2: Reward Fusion instead of binary success

On long episodes a binary reward is almost always zero, which leaves the gradient nothing to work with. The reward is therefore made up of four terms:

R = 0.5·ORM + 0.3·R_DOM + 0.2·R_waypoint − penalties

• ORM: the judge model's score of the final state;
• R_DOM: a programmatic DOM check, for example whether a field is filled in, a dialog is closed or a checkbox is ticked;
• R_waypoint: intermediate milestones within an episode;
• penalties: for unnecessary actions and invalid transitions.

The run logs contain 100 distinct reward values ranging from −8.4 to +0.92. This confirms that the reward is genuinely graded and works as intended.

Results

Metric	Value
Mean SR (phase 3)	42%
Final SR (last 10 iterations)	61.7%
Tasks with non-zero SR	20 of 25
Environment steps	50,000

The best results came from click-dialog and focus-text (100%) and from click-button (60%).

What did not work

The failures in this project deserve as much attention as the results.

• choose-date / book-flight: the jQuery date picker accumulates element IDs when the DOM is re-rendered, which makes long navigation unreliable.
• enter-date / enter-time: three input fields share one ID, and the standard fill() does not work in the current version of the environment.
• email-inbox / search-engine: these tasks require reasoning, choosing an element is not enough, and the pointer policy reaches its limit here.
• PopArt: value normalization statistics are not saved in the checkpoint and are recomputed when training resumes.

The first two problems come from environment version drift and are unrelated to the method itself. The last two reflect real limitations of the architecture, and they set the direction for the next piece of work.