Skip to content

keyboard:type action does not trigger Vue component's filter/search in filterable Select components #154

@wangjia2016

Description

@wangjia2016

Title: keyboard:type action does not trigger Vue component's filter/search in filterable Select components

Body:


Bug Description

When using agent.act() with keyboard:type to type into a filterable <select>-like component (e.g., a custom Vue 2 component with a searchable dropdown list), the list does not filter as expected. The text appears visually in the input field, but the underlying component's filter logic is never triggered.

Environment

  • magnitude-core version: 0.3.1
  • Browser: Chromium (via Playwright/Patchright)
  • Frontend framework of target app: Vue 2 (production build, __vue__ not exposed)
  • Component type: Custom filterable select component (lls-select) — similar to Element UI's <el-select filterable>

Steps to Reproduce

  1. Start a browser agent targeting a page with a filterable select component
  2. Use agent.act() to instruct the AI to type a keyword into the select's search input
  3. The AI executes keyboard:type action with the keyword
  4. Observe: the keyword appears in the input visually, but the dropdown list remains unfiltered (shows all options)

Root Cause Analysis

The keyboard:type action in magnitude-core (via Playwright's page.keyboard.type()) simulates keydown/keyup events but does not dispatch the native input event or compositionend event that Vue/custom component frameworks rely on to trigger their internal filter logic.

We also tried:

  • page.keyboard.insertText(keyword) — same result, list not filtered
  • element.value = keyword + dispatchEvent(new Event('input')) via page.evaluate — works in browser console manually, but when called after agent.act() opens the dropdown, the dropdown state gets reset because the AI may re-click the trigger, causing a race condition
  • Direct Playwright locator.fill() — the target input is readonly, fill is rejected

The core issue is there is no way to hook into the agent.act() lifecycle between "click opens dropdown" and "type into search box" — by the time page.evaluate runs after act() completes, the dropdown may have already closed or reset.

Expected Behavior

Either:

  1. keyboard:type should dispatch a native input event (and optionally compositionstart/compositionend for CJK input) so that framework components respond correctly
  2. Or, provide a way to execute custom Playwright code between AI actions within a single act() call (e.g., a hook or callback)

Workaround Attempted

None that fully works. The closest approach was using page.evaluate to directly set the value and dispatch input event, but it's unreliable due to the dropdown lifecycle race condition.

Additional Context

This is a very common scenario in enterprise web apps built with Vue/React that use virtual-scroll filterable select components (Element UI, Ant Design, custom components). The keyboard:type limitation makes it practically impossible to automate these components reliably.


Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions