🧪 Google Hackathon - Game Testing Agent

⚙️ Prerequisites

Before you can set up and run your agent, ensure you have the following installed:

• 🔄 AskUI Shell - The command line tool for AskUI Agents.
• 🖊️ A code editor of your choice (e.g., VSCode, PyCharm).
• 🔑 An Anthropic API key - Required for the VisionAgent to function properly.

🔧 Setup

1. Install AskUI Agent OS

Agent OS is a device controller that allows agents to take screenshots, move the mouse, click, and type on the keyboard across any operating system.

Windows

AMD64

AskUI Installer for AMD64

ARM64

AskUI Installer for ARM64

Linux

⚠️ Warning: Agent OS currently does not work on Wayland. Switch to XOrg to use it.

AMD64

curl -L -o /tmp/AskUI-Suite-Latest-User-Installer-Linux-AMD64-Web.run https://files.askui.com/releases/Installer/Latest/AskUI-Suite-Latest-User-Installer-Linux-AMD64-Web.run

bash /tmp/AskUI-Suite-Latest-User-Installer-Linux-AMD64-Web.run

ARM64

curl -L -o /tmp/AskUI-Suite-Latest-User-Installer-Linux-ARM64-Web.run https://files.askui.com/releases/Installer/Latest/AskUI-Suite-Latest-User-Installer-Linux-ARM64-Web.run

bash /tmp/AskUI-Suite-Latest-User-Installer-Linux-ARM64-Web.run

MacOS

curl -L -o /tmp/AskUI-Suite-Latest-User-Installer-MacOS-ARM64-Web.run https://files.askui.com/releases/Installer/Latest/AskUI-Suite-Latest-User-Installer-MacOS-ARM64-Web.run

bash /tmp/AskUI-Suite-Latest-User-Installer-MacOS-ARM64-Web.run

2. Install vision-agent in your Python environment

pip install askui

Note: Requires Python version >=3.10.

3a. Authenticate with an AI Model Provider

	AskUI INFO	Anthropic INFO
ENV Variables	ASKUI_WORKSPACE_ID, ASKUI_TOKEN	ANTHROPIC_API_KEY
Supported Commands	click(), get(), locate(), mouse_move()	act(), click(), get(), locate(), mouse_move()
Description	Faster Inference, European Server, Enterprise Ready	Supports complex actions

To get started, set the environment variables required to authenticate with your chosen model provider.

How to set an environment variable?

1. Create .env file similar to .env.template and update required credentials.

3b. Test with 🤗 Hugging Face AI Models (Spaces API)

You can test the Vision Agent with Huggingface models via their Spaces API. Please note that the API is rate-limited so for production use cases, it is recommended to choose step 3a.

Note: Hugging Face Spaces host model demos provided by individuals not associated with Hugging Face or AskUI. Don't use these models on screens with sensible information.

Supported Models:

• AskUI/PTA-1
• OS-Copilot/OS-Atlas-Base-7B
• showlab/ShowUI-2B
• Qwen/Qwen2-VL-2B-Instruct
• Qwen/Qwen2-VL-7B-Instruct

Example Code:

agent.click("search field", model="OS-Copilot/OS-Atlas-Base-7B")

3c. Host your own AI Models

UI-TARS

You can use Vision Agent with UI-TARS if you provide your own UI-TARS API endpoint.

1. Step: Host the model locally or in the cloud. More information about hosting UI-TARS can be found here.

2. Step: Provide the TARS_URL and TARS_API_KEY environment variables to Vision Agent.

3. Step: Use the model="tars" parameter in your click(), get() and act() etc. commands or when initializing the VisionAgent.

▶️ Run Your Agent

This agent runs automated UI tests using Pytest and AskUI's VisionAgent. When run, it:

1. Initializes the VisionAgent
2. Executes test cases defined in the tests/ directory
3. Performs UI interactions like clicks and assertions
4. Reports test results

To run your agent locally:

rm -rf venv && python3 -m venv venv && source venv/bin/activate && pip install -r requirements.txt

pytest --verbose

Happy Coding! 🚀