Merge OSWorld Agents into synthetic data gen

examples/SETUP_CHANGES.md

@@ -0,0 +1,233 @@
+# Setup.py Modification Summary
+
+## Changes Made
+
+### 1. Removed Proxy Functionality
+- **Removed imports:**
+  - `from desktop_env.providers.aws.proxy_pool import get_global_proxy_pool, init_proxy_pool, ProxyInfo`
+
+- **Removed code:**
+  - `PROXY_CONFIG_FILE` constant
+  - `init_proxy_pool(PROXY_CONFIG_FILE)` initialization
+  - `_proxy_setup()` method (entire method removed)
+  - `self.use_proxy` attribute from `__init__`
+  - `use_proxy` parameter from `setup()` method
+  - Proxy server logic in `_launch_setup()` method
+
+### 2. Replaced PythonController with Runtime and OSWorldInteractiveAction
+
+- **Modified `__init__` method:**
+  - Added `runtime=None` parameter
+  - Added `self.runtime = runtime` to store runtime object for environment interaction
+  - Removed `self.use_proxy` attribute
+
+- **Modified `setup()` method:**
+  - Replaced HTTP call to `/terminal` endpoint with `OSWorldInteractiveAction`:
+    ```python
+    action = OSWorldInteractiveAction(
+        method='get_terminal_output',
+        params={}
+    )
+    observation = self.runtime.run_action(action)
+    ```
+  - **Returns False** if runtime is not available (runtime is now **required**)
+
+- **Modified `_execute_setup()` method:**
+  - Replaced HTTP call to `/setup/execute` endpoint with `OSWorldInteractiveAction`:
+    ```python
+    action = OSWorldInteractiveAction(
+        method='run_bash_script',
+        params={'script': script, 'timeout': 120}
+    )
+    observation = self.runtime.run_action(action)
+    ```
+  - **Raises Exception** if runtime is not available (runtime is now **required**)
+
+- **Modified `_update_browse_history_setup()` method:**
+  - Replaced `controller = PythonController(self.vm_ip, self.server_port)` with `OSWorldInteractiveAction` and runtime
+  - Changed `controller.get_vm_platform()` to:
+    ```python
+    action = OSWorldInteractiveAction(
+        method='run_python_script',
+        params={'script': 'import platform; print(platform.system())'}
+    )
+    observation = self.runtime.run_action(action)
+    ```
+  - Changed all `controller.execute_python_command()` calls to use `OSWorldInteractiveAction` with `run_python_script` method
+  - Added proper error handling checking `observation.exit_code` and `observation.content`
+  - Added runtime validation check at the beginning of the method
+
+- **Added new import:**
+  - `from openhands.events.action.os import OSWorldInteractiveAction`
+
+- **HTTP endpoints kept as-is (not standard OSWorld methods):**
+  - `/setup/upload` - File upload endpoint
+  - `/setup/change_wallpaper` - Change desktop wallpaper
+  - `/setup/open_file` - Open file with default application
+  - `/setup/launch` - Launch applications
+  - `/setup/execute_with_verification` - Execute command with verification
+  - `/setup/activate_window` - Activate window by name
+  - `/setup/close_window` - Close window by name
+
+  These endpoints are setup-specific and have no corresponding methods in OSWorldInteractiveAction, so they remain as HTTP requests. However, all methods now check for runtime availability at the beginning and raise an Exception if runtime is None.
+
+- **All setup methods now validate runtime:**
+  - `_download_setup()` - Checks runtime at start
+  - `_upload_file_setup()` - Checks runtime at start
+  - `_change_wallpaper_setup()` - Checks runtime at start
+  - `_open_setup()` - Checks runtime at start
+  - `_launch_setup()` - Checks runtime at start
+  - `_execute_setup()` - Checks runtime at start (already had this)
+  - `_execute_with_verification_setup()` - Checks runtime at start
+  - `_activate_window_setup()` - Checks runtime at start
+  - `_close_window_setup()` - Checks runtime at start
+  - `_chrome_open_tabs_setup()` - Checks runtime at start
+  - `_chrome_close_tabs_setup()` - Checks runtime at start
+  - `_googledrive_setup()` - Checks runtime at start
+  - `_login_setup()` - Checks runtime at start
+  - `_update_browse_history_setup()` - Checks runtime at start (already had this)
+
+### 3. Copied Necessary Functions from desktop_env
+
+- **Added `compare_urls()` function:**
+  - Copied from `desktop_env.evaluators.metrics.utils`
+  - Includes helper functions: `parse_with_default_scheme()` and `normalize_url()`
+  - Added necessary imports: `re`, `urllib.parse` components, and `tldextract`
+  - Properly documented as copied from desktop_env
+
+- **Removed imports:**
+  - `from desktop_env.controllers.python import PythonController`
+  - `from desktop_env.evaluators.metrics.utils import compare_urls` (function now defined locally)
+
+## New Dependencies
+- Added `re` import for URL parsing
+- Added `from urllib.parse import urlparse, urlunparse, ParseResult` for URL manipulation
+- Added `tldextract` import for domain extraction
+- Added `from openhands.events.action.os import OSWorldInteractiveAction` for runtime actions
+
+## Usage Changes
+
+### Before:
+```python
+# Initialize without runtime
+setup_controller = SetupController(
+    vm_ip="192.168.1.1",
+    server_port=5000,
+    cache_dir="cache"
+)
+
+# Setup with proxy support
+setup_controller.setup(config, use_proxy=True)
+```
+
+### After:
+```python
+# Initialize with runtime
+setup_controller = SetupController(
+    vm_ip="192.168.1.1",
+    server_port=5000,
+    cache_dir="cache",
+    runtime=runtime_object  # Pass your runtime object here
+)
+
+# Setup without proxy parameter
+setup_controller.setup(config)
+```
+
+## Backward Compatibility Notes
+
+1. **Breaking changes:**
+   - `setup()` method no longer accepts `use_proxy` parameter
+   - `SetupController.__init__()` now **REQUIRES** `runtime` parameter - will fail without it
+   - `_proxy_setup` setup type is no longer supported in config
+   - **ALL setup methods now require runtime:** Every `_*_setup()` method validates that runtime is not None at the beginning and raises an Exception if it is
+   - **NO HTTP FALLBACK:** All methods that previously had HTTP fallbacks now require runtime
+   - **Setup methods that still use HTTP endpoints** (like `/setup/upload`, `/setup/launch`, etc.) now also require runtime to be set, even though they don't directly use runtime.run_action()
+
+2. **Runtime requirement:**
+   - The runtime object must implement `run_action(action: OSWorldInteractiveAction)` method
+   - The method should return an observation object with:
+     - `content`: The output/result as a string
+     - `exit_code`: The exit code (0 for success, non-zero for errors)
+   - Example usage:
+     ```python
+     from openhands.events.action.os import OSWorldInteractiveAction
+
+     action = OSWorldInteractiveAction(
+         method='run_python_script',
+         params={'script': 'import platform; print(platform.system())'}
+     )
+     observation = runtime.run_action(action)
+     if observation.exit_code == 0:
+         print(f"Result: {observation.content}")
+     else:
+         print(f"Error: {observation.content}")
+     ```
+
+## Complete Example
+
+Here's a complete example showing how to use the updated SetupController:
+
+```python
+from openhands.events.action.os import OSWorldInteractiveAction
+from setup import SetupController
+
+# Assume you have a runtime object that implements run_action()
+# For example: runtime = OSWorldSingularityRuntime(...)
+
+# Initialize SetupController with runtime
+setup_controller = SetupController(
+    vm_ip="192.168.1.1",
+    server_port=5000,
+    cache_dir="cache",
+    runtime=runtime  # Pass your runtime object
+)
+
+# Setup configuration (no proxy support)
+config = [
+    {
+        "type": "download",
+        "parameters": {
+            "files": [
+                {
+                    "url": "https://example.com/file.zip",
+                    "path": "/home/user/Downloads/file.zip"
+                }
+            ]
+        }
+    },
+    {
+        "type": "launch",
+        "parameters": {
+            "command": ["google-chrome"]
+        }
+    }
+]
+
+# Run setup (no use_proxy parameter)
+success = setup_controller.setup(config)
+
+if success:
+    print("Setup completed successfully!")
+else:
+    print("Setup failed!")
+```
+
+## Testing Recommendations
+
+1. Test all setup types without proxy functionality
+2. Verify `_update_browse_history_setup` works with runtime object
+3. Ensure `compare_urls` function works correctly for Chrome setup operations
+4. Test runtime integration with various environment types
+5. Verify that `OSWorldInteractiveAction` is correctly created and handled by runtime
+6. **Test that SetupController fails gracefully when runtime is None**
+7. **Verify error messages are clear when runtime is missing**
+
+## Files Modified
+
+- `/home/jayliu/ProRL-Agent-Server/examples/setup.py`
+
+## Files Not Modified
+
+- Original file at `/home/jayliu/OSWorld/desktop_env/controllers/setup.py` remains unchanged
+