Hive Browser Automation

GCU Browser Automation

All GCU browser tools drive a real Chrome instance through the Beeline extension and Chrome DevTools Protocol (CDP). That means clicks, keystrokes, and screenshots are processed by the actual browser's native hit testing, focus, and layout engines — not a synthetic event layer. Understanding this unlocks strategies that make hard sites easy.

Coordinates

Every browser tool that takes or returns coordinates operates in fractions of the viewport (0..1 for both axes). Read a target's proportional position off browser_screenshot — "this button is about 35% from the left and 20% from the top" → pass (0.35, 0.20). Rect-returning tools (browser_get_rect, browser_shadow_query, and the rect inside focused_element) also return fractions. The tools convert to CSS pixels internally before dispatching to Chrome.

browser_screenshot()                  → image + cssWidth/cssHeight in meta
browser_click_coordinate(x, y)        → x, y are fractions 0..1
browser_hover_coordinate(x, y)        → fractions
browser_press_at(x, y, key)           → fractions
browser_get_rect(selector) → rect     → rect.cx / rect.cy are fractions
browser_shadow_query(...)  → rect     → same

Why fractions: every vision model (Claude ~1.15 MP target, GPT-4o 512-px tiles, Gemini, local VLMs) resizes or tiles images differently before the model sees the pixels. Proportions survive every such transform; pixel coordinates only "work" per-model and silently break when you swap backends. Four-decimal precision (0.0001 ≈ 0.17 CSS px on a 1717-wide viewport) is more than enough for the tightest targets.

Exception for zoomed elements: pages that use zoom or transform: scale() on a container (LinkedIn's #interop-outlet, some embedded iframes) render in a scaled local coordinate space. getBoundingClientRect there may not match CDP's hit space. Prefer browser_shadow_query (which handles the math and returns fractions) or visually pick coordinates from a screenshot. Avoid raw browser_evaluate + getBoundingClientRect() for coord lookup — that returns CSS px and will be wrong when fed to click tools.

Screenshot + coordinates is shadow-agnostic — prefer it on shadow-heavy sites

Start with browser_snapshot when you need to inspect the page structure or find ordinary controls. If the snapshot does not show the thing you need, shows stale or misleading refs, or cannot prove where a visible target is, take browser_screenshot and use the screenshot + coordinate path. This is especially useful on sites that use Shadow DOM heavily

Why:

• CDP hit testing walks shadow roots natively. browser_click_coordinate(x, y) routes through Chrome's native hit tester, which traverses open shadow roots automatically. You don't need to know the shadow structure.
• Keyboard dispatch follows focus into shadow roots. After a click focuses an input (even one three shadow levels deep), browser_press(...) with no selector dispatches keys to document.activeElement's computed focus target.
• Screenshots render the real layout regardless of DOM implementation.

Whereas wait_for_selector, browser_click(selector=...), browser_type(selector=...) all use document.querySelector under the hood, which stops at shadow boundaries. They cannot see elements inside shadow roots. For shadow-DOM inputs, use browser_type_focused after focusing via click-coordinate.

Recommended workflow on shadow-heavy sites

1. browser_screenshot() → JPEG; meta includes cssWidth/cssHeight for reference.
2. Identify the target visually → estimate its proportional position (fx, fy) where each is in 0..1.
3. browser_click_coordinate(fx, fy) → tool converts to CSS px and dispatches; CDP native hit testing focuses the element. The response includes focused_element: {tag, id, role, contenteditable, rect, inFrame?, ...} — use it to verify you actually focused what you intended. rect is in fractions (same space as your input). When focus is inside a same-origin iframe, the descriptor reports the inner element and adds inFrame: [...] breadcrumbs.
4. browser_type_focused(text="...") → inserts text into document.activeElement (traverses into same-origin iframes automatically). Shadow roots, iframes, Lexical, Draft.js, ProseMirror all just work. Use browser_type(selector, text) instead when you have a reliable CSS selector for a light-DOM element.
5. Verify via browser_screenshot OR browser_get_attribute on a known-reachable marker (e.g. check that the Send button's aria-disabled flipped to false).

The click→type loop (canonical pattern)

1. Call browser_click_coordinate(x, y) to click the target element.
2. Check the focused_element field in the response — it tells you what actually received focus (tag, id, role, contenteditable, rect).
3. If the focused element is editable, call browser_type_focused(text="...") to insert text. Use tools to verify the text took effect — prefer checking the underlying .value / innerText via browser_evaluate or confirming the submit button enabled. A screenshot alone can mislead: narrow input boxes visually clip long text, so only a portion may appear on screen even though the full string was accepted.
4. If it is NOT editable, your click landed on the wrong thing — refine coordinates and retry. Do NOT reach for browser_evaluate + execCommand('insertText') or shadow-root traversals. The problem is the click target, not the typing method.

browser_click (selector-based) also returns focused_element, so the same check works whether you clicked by selector or coordinate.

Empirically verified (2026-04-11)

Tested against https://www.reddit.com/r/programming/ whose search input lives at:

document > reddit-search-large [shadow]
         > faceplate-search-input#search-input [shadow]
         > input[name="q"]

Shadow-piercing selectors

When you DO want a selector-based approach and know the shadow structure, browser_shadow_query and browser_get_rect support >>> shadow-piercing syntax:

browser_shadow_query("reddit-search-large >>> #search-input")
browser_get_rect("#interop-outlet >>> #ember37 >>> p")

Returns the element's rect as fractions of the viewport (feed rect.cx / rect.cy directly to click tools). Remember: browser_type and wait_for_selector do not support >>> — only shadowquery and getrect do.

Navigation and waiting

The basics

browser_navigate(url, wait_until="load")   # "load" | "domcontentloaded" | "networkidle"
browser_wait_for_selector("h1", timeout_ms=2000)
browser_wait_for_text("Some text", timeout_ms=2000)
browser_go_back()
browser_go_forward()
browser_reload()

All return real URLs and titles. On a fast page navigate(wait_until="load") returns in sub-second. wait_for_selector and wait_for_text typically resolve in single-digit milliseconds on elements already in the DOM.

Timing expectations (measured against real sites)

| Site | Navigate load time | | ------------------------ | ------------------ | | example.com | 100–400 ms | | wikipedia.org | 200–500 ms | | reddit.com | 1.5–2 s | | x.com/twitter | 1.2–1.6 s | | linkedin.com (logged in) | 4–5 s |

For LinkedIn and other heavy SPAs, rely on sleep() after navigation to let the page hydrate.

After navigate, always let SPA hydrate

Even after wait_until="load", React/Vue SPAs often render their real chrome in a second pass. Add await sleep(2) to await sleep(3) before querying for site-specific elements. Otherwise wait_for_selector will fail on elements that do exist moments later.

Reading pages efficiently

• Prefer browser_snapshot over browser_get_text("body") — returns a compact ~1–5 KB accessibility tree vs 100+ KB of raw HTML.
• Interaction tools browser_click, browser_type, browser_type_focused, and browser_scroll wait 0.5 s for the page to settle after a successful action, then attach a fresh accessibility snapshot under the snapshot key of their result. Use it to decide your next action — do NOT call browser_snapshot separately after every action. Tune the capture via auto_snapshot_mode: "default" (full tree, the default), "simple" (trims unnamed structural nodes), "interactive" (only controls — tightest token footprint), or "off" to skip the capture entirely (useful when batching several interactions and you don't need the intermediate trees). Call browser_snapshot explicitly only when you need a newer view or a different mode than what was auto-captured.
• Complex pages (LinkedIn, Twitter/X, SPAs with virtual scrolling) can have DOMs that don't match what's visually rendered — snapshot refs may be stale, missing, or misaligned with visible layout. Try the available snapshot first; when the target is not present in that snapshot or visual position matters, switch to browser_screenshot to orient yourself.
• Only fall back to browser_get_text for extracting specific small elements by CSS selector.

Typing and keyboard input

ALWAYS click before typing into rich-text editors

The single most common "looks like it worked but send button stays disabled" failure. If you're typing into a modern editor (X/Twitter's Draft.js compose, LinkedIn's post composer, Reddit's comment box, Gmail compose, Slack, Discord, Notion, Monaco, any contenteditable), click the input area first with browser_click_coordinate or browser_click(selector) before you type.

Why this is necessary:

• React / Vue controlled components don't trust JS-sourced .focus(). React uses event delegation and watches for native pointer/focus events — a click dispatched via CDP fires the real pointerdown/pointerup/click/focus sequence that React listens to, and updates its internal state. A JS-only .focus() sets document.activeElement but the framework's controlled state doesn't see it.
• Draft.js (X/Twitter compose) and Lexical (Gmail, LinkedIn DMs) use contenteditable divs with immutable editor state. They only enter "edit mode" after a real click on the editor surface. Typing at them without clicking routes keys to document.body or gets silently discarded.
• Send/submit buttons are bound to framework state, not DOM state. They're typically disabled={!hasRealContent} where hasRealContent is computed from React/Vue/Svelte state. The input field can have characters in the DOM but the button stays disabled because the framework never saw a real input event.

The symptom is always the same: you type, the characters appear visually, and the send button doesn't enable. The agent then clicks send anyway, nothing happens, and it thinks the post failed.

Safe "click-then-type-then-verify" pattern

1. Focus the real element via a real click (not JS .focus()). Use browser_get_rect(selector) (or browser_shadow_query for shadow sites) to get coordinates, then browser_click_coordinate(cx, cy). Wait ~0.5 s for the editor to open and focus to settle.

1. Type the text. Use browser_type(selector, text) for light-DOM inputs, or browser_type_focused(text=...) for shadow-DOM / already-focused inputs. Both use CDP Input.insertText by default, which is the most reliable method for rich editors (Lexical, Draft.js, ProseMirror). Wait ~500 ms for framework state to commit.

1. Verify the submit button is enabled before clicking it. Use browser_evaluate to check the button's disabled or aria-disabled attribute. Do NOT trust that typing worked — always check state.

Partial visibility is fine. Small single-line inputs, chat boxes with fixed width, and search fields commonly clip or truncate long text visually — only the tail or head may be shown on screen. Don't treat that as failure. What matters is that the framework accepted the input: the submit button enabled, or element.value / innerText read via browser_evaluate contains the full string. If the visible pixels don't match what you typed but the button is enabled and the underlying value is correct, typing succeeded — proceed.

1. Only click send if the button is enabled. If the button is still disabled, try the recovery dance: click the textarea again, press End, press a space, press Backspace — this forces React to recompute hasRealContent. Then re-check the button state.

Why browser_type uses Input.insertText by default

CDP has a dedicated method — Input.insertText — for committing text into the focused element as if IME just committed it. It bypasses the keyboard event pipeline entirely and works cleanly on every rich-text editor tested to date: Lexical (LinkedIn DMs, Gmail), Draft.js (X compose), ProseMirror (Reddit), Monaco, and plain contenteditable. Playwright uses this under the hood for keyboard.type() on rich editors.

Per-character Input.dispatchKeyEvent looks equivalent on paper, but some rich editors listen for beforeinput events with a specific shape and route insertion through their own state machine — the raw keys arrive but never get turned into text. That was the exact failure mode that left LinkedIn's message composer empty (and its Send button disabled) during the 2026-04-11 empirical run.

If you need per-keystroke dispatch (autocomplete testing, code editors, animated typing with delay_ms), pass use_insert_text=False to fall back to the old keyDown/keyUp path.

Neutralizing beforeunload draft dialogs

When a composer has unsent text and you try to navigate away or close the tab, sites like LinkedIn pop a native "You have an unsent message, leave?" confirm dialog via window.onbeforeunload. Your automation hangs waiting on the dialog — browser_close_tab and browser_navigate both time out.

Strip the handler via browser_evaluate before navigating:

browser_evaluate("""
    (function(){
      window.onbeforeunload = null;
      window.addEventListener('beforeunload', function(e){
        e.stopImmediatePropagation();
      }, true);
      return true;
    })()
""")
# Now browser_navigate / close_tab work without hitting a confirm

Always include an equivalent cleanup block in any script that types into a compose UI — without it, a script crash mid-type leaves the tab in an unusable state with the draft modal blocking every subsequent automation call.

Verified site-specific quirks

| Site | Editor | Workaround | | ---------------------------------------------------- | ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | X / Twitter compose | Draft.js | Click [data-testid='tweetTextarea_0'] first, then type with delay_ms=20. First 1-2 chars may be eaten — accept truncation or prepend a throwaway char. Verify [data-testid='tweetButton'] has disabled: false before clicking. | | LinkedIn messaging | contenteditable (inside #interop-outlet shadow root) | Use browser_shadow_query to find the rect, click-coordinate to focus, then browser_type_focused(text=...) (selector-based browser_type can't reach shadow). Send button is .msg-form__send-button. | | LinkedIn feed post composer | Qu

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: aden-hive
• Source: aden-hive/hive
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.