# Screen Utilities

> **Relevant source files**
> * [src/io/github/samera2022/mouse_macros/Localizer.java](https://github.com/Samera2022/MouseMacros/blob/1eb6620b/src/io/github/samera2022/mouse_macros/Localizer.java)
> * [src/io/github/samera2022/mouse_macros/MouseMacro.java](https://github.com/Samera2022/MouseMacros/blob/1eb6620b/src/io/github/samera2022/mouse_macros/MouseMacro.java)
> * [src/io/github/samera2022/mouse_macros/constant/OtherConsts.java](https://github.com/Samera2022/MouseMacros/blob/1eb6620b/src/io/github/samera2022/mouse_macros/constant/OtherConsts.java)
> * [src/io/github/samera2022/mouse_macros/listener/GlobalMouseListener.java](https://github.com/Samera2022/MouseMacros/blob/1eb6620b/src/io/github/samera2022/mouse_macros/listener/GlobalMouseListener.java)
> * [src/io/github/samera2022/mouse_macros/util/FileUtil.java](https://github.com/Samera2022/MouseMacros/blob/1eb6620b/src/io/github/samera2022/mouse_macros/util/FileUtil.java)

## Purpose and Scope

The `ScreenUtil` class provides coordinate transformation utilities for multi-monitor setups and DPI scaling handling. This utility enables MouseMacros to record and play back mouse actions consistently across different monitor configurations by normalizing coordinates to a virtual origin that spans all connected displays.

For information about other utility classes, see [Utility Classes](System-Integration-Utilities). For details on how these utilities integrate with macro playback, see [MouseAction](MouseAction-Data-Structure).

**Sources:** [src/io/github/samera2022/mouse_macros/util/ScreenUtil.java L1-L42](https://github.com/Samera2022/MouseMacros/blob/1eb6620b/src/io/github/samera2022/mouse_macros/util/ScreenUtil.java#L1-L42)

## Multi-Monitor Coordinate Systems

In Java AWT, each monitor has its own coordinate space. The primary monitor typically has its origin at `(0, 0)`, while secondary monitors may have negative or positive offsets depending on their physical arrangement. The virtual screen space encompasses all monitors as a single continuous coordinate system.

### Virtual Origin Concept

The virtual origin is the top-left corner of the bounding rectangle that contains all connected monitors. This point has the minimum X and Y coordinates across all screens and serves as the reference point for coordinate normalization.

```mermaid
flowchart TD

M1Origin["Origin: (-1920, 0)<br>Size: 1920x1080"]
M2Origin["Origin: (0, 0)<br>Size: 2560x1440"]
M3Origin["Origin: (2560, -200)<br>Size: 1920x1080"]
VirtualOrigin["Virtual Origin<br>Point(-1920, -200)<br>minX across all screens<br>minY across all screens"]
Note1["Normalized coordinates<br>use virtual origin as (0,0)"]
Note2["Global coordinates<br>use primary screen as (0,0)"]

VirtualOrigin --> M1Origin
VirtualOrigin --> M2Origin
VirtualOrigin --> M3Origin
VirtualOrigin --> Note1
M2Origin --> Note2

subgraph subGraph3 ["Multi-Monitor Layout Example"]

subgraph subGraph2 ["Monitor3[Monitor 3 - Secondary]"]
    M3Origin
end

subgraph subGraph1 ["Monitor2[Monitor 2 - Primary]"]
    M2Origin
end

subgraph subGraph0 ["Monitor1[Monitor 1 - Secondary]"]
    M1Origin
end
end
```

**Sources:** [src/io/github/samera2022/mouse_macros/util/ScreenUtil.java L24-L35](https://github.com/Samera2022/MouseMacros/blob/1eb6620b/src/io/github/samera2022/mouse_macros/util/ScreenUtil.java#L24-L35)

## Coordinate Transformation Methods

The `ScreenUtil` class provides three coordinate transformation methods to enable consistent macro recording and playback across different monitor configurations.

### Method Overview

| Method | Input | Output | Purpose |
| --- | --- | --- | --- |
| `normalizeToVirtualOrigin` | Global coordinates | Normalized coordinates | Record phase: Store coordinates relative to virtual origin |
| `denormalizeFromVirtualOrigin` | Normalized coordinates | Robot coordinates | Playback phase: Convert to primary screen coordinates for Robot |
| `getVirtualOrigin` | None | Virtual origin point | Calculate the minimum X/Y across all screens |

### Coordinate Transformation Flow

```mermaid
sequenceDiagram
  participant User
  participant Operating System
  participant JNativeHook
  participant GlobalMouseListener
  participant ScreenUtil
  participant MouseAction
  participant java.awt.Robot

  note over User,java.awt.Robot: Recording Phase
  User->>Operating System: Mouse click at screen position
  Operating System->>JNativeHook: Global coordinates (x, y)
  JNativeHook->>GlobalMouseListener: nativeMousePressed(x, y)
  GlobalMouseListener->>ScreenUtil: normalizeToVirtualOrigin(x, y)
  ScreenUtil->>ScreenUtil: getVirtualOrigin()
  ScreenUtil->>ScreenUtil: normalized = (x - minX, y - minY)
  ScreenUtil-->>GlobalMouseListener: Normalized Point
  GlobalMouseListener->>MouseAction: Create MouseAction with normalized coords
  MouseAction->>MouseAction: Store normalized coordinates
  note over User,java.awt.Robot: Playback Phase
  User->>GlobalMouseListener: Trigger playback hotkey
  GlobalMouseListener->>MouseAction: action.perform()
  MouseAction->>ScreenUtil: denormalizeFromVirtualOrigin(x, y)
  ScreenUtil->>ScreenUtil: getVirtualOrigin()
  ScreenUtil->>ScreenUtil: globalX = x + virtualOrigin.x
  ScreenUtil->>ScreenUtil: globalY = y + virtualOrigin.y
  ScreenUtil->>ScreenUtil: Get primary screen bounds
  ScreenUtil->>ScreenUtil: Get DPI scale factors
  ScreenUtil->>ScreenUtil: robotX = (globalX - primaryX) / scaleX
  ScreenUtil->>ScreenUtil: robotY = (globalY - primaryY) / scaleY
  ScreenUtil-->>MouseAction: Robot-compatible Point
  MouseAction->>java.awt.Robot: mouseMove(robotX, robotY)
  java.awt.Robot->>Operating System: Inject mouse event
```

**Sources:** [src/io/github/samera2022/mouse_macros/util/ScreenUtil.java L7-L41](https://github.com/Samera2022/MouseMacros/blob/1eb6620b/src/io/github/samera2022/mouse_macros/util/ScreenUtil.java#L7-L41)

### Normalization: Global to Virtual Coordinates

The `normalizeToVirtualOrigin` method converts global screen coordinates to coordinates relative to the virtual origin. This is used during macro recording to store coordinates in a monitor-configuration-independent format.

```mermaid
flowchart TD

Input["Input<br>Global Coordinates<br>(x, y)"]
GetOrigin["getVirtualOrigin()<br>Returns (minX, minY)"]
Calculate["Calculate<br>normalizedX = x - minX<br>normalizedY = y - minY"]
Output["Output<br>Normalized Point<br>(x - minX, y - minY)"]

Input --> GetOrigin
GetOrigin --> Calculate
Calculate --> Output
```

**Implementation:** [src/io/github/samera2022/mouse_macros/util/ScreenUtil.java L38-L41](https://github.com/Samera2022/MouseMacros/blob/1eb6620b/src/io/github/samera2022/mouse_macros/util/ScreenUtil.java#L38-L41)

**Sources:** [src/io/github/samera2022/mouse_macros/util/ScreenUtil.java L37-L41](https://github.com/Samera2022/MouseMacros/blob/1eb6620b/src/io/github/samera2022/mouse_macros/util/ScreenUtil.java#L37-L41)

### Denormalization: Virtual to Robot Coordinates

The `denormalizeFromVirtualOrigin` method converts normalized coordinates back to the coordinate system expected by `java.awt.Robot`. This is more complex than normalization because it must:

1. Convert from virtual origin coordinates to global coordinates
2. Convert from global coordinates to primary screen coordinates
3. Apply DPI scaling corrections

```mermaid
flowchart TD

Input["Input: Normalized Coordinates<br>(x, y) relative to virtual origin"]
Step1["Step 1: Get Virtual Origin<br>virtualOrigin = getVirtualOrigin()"]
Step2["Step 2: Convert to Global Coordinates<br>globalX = x + virtualOrigin.x<br>globalY = y + virtualOrigin.y"]
Step3["Step 3: Get Primary Screen Info<br>gc = defaultScreenDevice.defaultConfiguration<br>primaryBounds = gc.getBounds()<br>scaleX = gc.getDefaultTransform().getScaleX()<br>scaleY = gc.getDefaultTransform().getScaleY()"]
Step4["Step 4: Convert to Primary Screen Coords<br>relativeX = globalX - primaryBounds.x<br>relativeY = globalY - primaryBounds.y"]
Step5["Step 5: Apply DPI Scaling<br>robotX = round(relativeX / scaleX)<br>robotY = round(relativeY / scaleY)"]
Output["Output: Robot Coordinates<br>(robotX, robotY)<br>Ready for java.awt.Robot"]

Input --> Step1
Step1 --> Step2
Step2 --> Step3
Step3 --> Step4
Step4 --> Step5
Step5 --> Output
```

**Implementation:** [src/io/github/samera2022/mouse_macros/util/ScreenUtil.java L7-L22](https://github.com/Samera2022/MouseMacros/blob/1eb6620b/src/io/github/samera2022/mouse_macros/util/ScreenUtil.java#L7-L22)

**Sources:** [src/io/github/samera2022/mouse_macros/util/ScreenUtil.java L6-L22](https://github.com/Samera2022/MouseMacros/blob/1eb6620b/src/io/github/samera2022/mouse_macros/util/ScreenUtil.java#L6-L22)

## Virtual Origin Calculation

The `getVirtualOrigin` method calculates the top-left corner of the virtual screen space by finding the minimum X and Y coordinates across all connected displays.

### Algorithm

```mermaid
flowchart TD

Start["Start: getVirtualOrigin()"]
Init["Initialize<br>minX = Integer.MAX_VALUE<br>minY = Integer.MAX_VALUE"]
GetEnv["Get GraphicsEnvironment<br>ge = getLocalGraphicsEnvironment()"]
GetDevices["Get all GraphicsDevice[]<br>devices = ge.getScreenDevices()"]
LoopStart["For each<br>device"]
GetBounds["Get device bounds<br>bounds = device.getDefaultConfiguration().getBounds()"]
UpdateMinX["bounds.x < minX?"]
SetMinX["minX = bounds.x"]
UpdateMinY["bounds.y < minY?"]
SetMinY["minY = bounds.y"]
LoopEnd["Next device"]
Return["Return Point(minX, minY)"]

Start --> Init
Init --> GetEnv
GetEnv --> GetDevices
GetDevices --> LoopStart
LoopStart --> GetBounds
GetBounds --> UpdateMinX
UpdateMinX --> SetMinX
UpdateMinX --> UpdateMinY
SetMinX --> UpdateMinY
UpdateMinY --> SetMinY
UpdateMinY --> LoopEnd
SetMinY --> LoopEnd
LoopEnd --> LoopStart
LoopStart --> Return
```

**Implementation:** [src/io/github/samera2022/mouse_macros/util/ScreenUtil.java L25-L35](https://github.com/Samera2022/MouseMacros/blob/1eb6620b/src/io/github/samera2022/mouse_macros/util/ScreenUtil.java#L25-L35)

The method iterates through all connected displays and tracks the minimum X and Y values. This ensures that all screen coordinates can be represented as positive offsets from the virtual origin.

**Sources:** [src/io/github/samera2022/mouse_macros/util/ScreenUtil.java L24-L35](https://github.com/Samera2022/MouseMacros/blob/1eb6620b/src/io/github/samera2022/mouse_macros/util/ScreenUtil.java#L24-L35)

## DPI Scaling Handling

Modern operating systems support high-DPI displays with scaling factors (e.g., 150%, 200%). Java's `java.awt.Robot` class expects coordinates in the physical pixel space, not the logical coordinate space used by most applications.

### DPI Scaling Transformation

The `denormalizeFromVirtualOrigin` method handles DPI scaling by:

1. Obtaining the default transform from the primary screen's `GraphicsConfiguration`
2. Extracting X and Y scale factors using `getScaleX()` and `getScaleY()`
3. Dividing the relative coordinates by the scale factors
4. Rounding to the nearest integer pixel

```mermaid
flowchart TD

LogicalCoords["Logical Coordinates<br>(e.g., 1000, 500)<br>at 200% scaling"]
ScaleFactor["Scale Factors<br>scaleX = 2.0<br>scaleY = 2.0"]
Division["Division<br>physicalX = 1000 / 2.0 = 500<br>physicalY = 500 / 2.0 = 250"]
Rounding["Rounding<br>robotX = round(500) = 500<br>robotY = round(250) = 250"]
PhysicalCoords["Physical Coordinates<br>(500, 250)<br>for Robot API"]

LogicalCoords --> ScaleFactor
ScaleFactor --> Division
Division --> Rounding
Rounding --> PhysicalCoords
```

**Implementation:** [src/io/github/samera2022/mouse_macros/util/ScreenUtil.java L16-L20](https://github.com/Samera2022/MouseMacros/blob/1eb6620b/src/io/github/samera2022/mouse_macros/util/ScreenUtil.java#L16-L20)

### Why DPI Scaling Matters

| Without DPI Handling | With DPI Handling |
| --- | --- |
| Robot clicks at wrong location | Robot clicks at correct location |
| Coordinates off by scale factor | Coordinates match recorded position |
| Macros fail on high-DPI displays | Macros work consistently |
| User must manually adjust | Automatic correction applied |

**Sources:** [src/io/github/samera2022/mouse_macros/util/ScreenUtil.java L13-L21](https://github.com/Samera2022/MouseMacros/blob/1eb6620b/src/io/github/samera2022/mouse_macros/util/ScreenUtil.java#L13-L21)

## Integration with Macro System

The `ScreenUtil` class integrates with the macro recording and playback pipeline through the `MouseAction` class.

### Recording Integration

During recording, `GlobalMouseListener` receives mouse events with global screen coordinates from `JNativeHook`. Before creating a `MouseAction` object, the coordinates are normalized using `normalizeToVirtualOrigin`. This ensures that recorded coordinates are independent of the user's monitor configuration at recording time.

### Playback Integration

During playback, the `MouseAction.perform()` method calls `denormalizeFromVirtualOrigin` to convert the stored normalized coordinates back to coordinates that `java.awt.Robot` can use. This transformation accounts for:

* Changes in monitor configuration since recording
* DPI scaling differences between recording and playback systems
* Primary screen position differences

```mermaid
flowchart TD

RecEvent["Mouse Event<br>Global Coords"]
Normalize["ScreenUtil.normalizeToVirtualOrigin"]
StoreAction["Store MouseAction<br>with normalized coords"]
MMCFile[".mmc File<br>CSV format<br>x,y,type,..."]
LoadAction["Load MouseAction<br>from file"]
Denormalize["ScreenUtil.denormalizeFromVirtualOrigin"]
RobotExec["Robot.mouseMove<br>Robot.mousePress"]

StoreAction --> MMCFile
MMCFile --> LoadAction

subgraph Playback ["Playback"]
    LoadAction
    Denormalize
    RobotExec
    LoadAction --> Denormalize
    Denormalize --> RobotExec
end

subgraph Persistence ["Persistence"]
    MMCFile
end

subgraph Recording ["Recording"]
    RecEvent
    Normalize
    StoreAction
    RecEvent --> Normalize
    Normalize --> StoreAction
end
```

**Sources:** [src/io/github/samera2022/mouse_macros/util/ScreenUtil.java L1-L42](https://github.com/Samera2022/MouseMacros/blob/1eb6620b/src/io/github/samera2022/mouse_macros/util/ScreenUtil.java#L1-L42)

## Class Structure

The `ScreenUtil` class is a utility class with static methods only. It has no instance variables or constructors.

### Public Methods

| Method Signature | Return Type | Description |
| --- | --- | --- |
| `denormalizeFromVirtualOrigin(int x, int y)` | `Point` | Converts normalized coordinates to Robot-compatible coordinates with DPI scaling |
| `normalizeToVirtualOrigin(int x, int y)` | `Point` | Converts global screen coordinates to normalized coordinates |

### Private Methods

| Method Signature | Return Type | Description |
| --- | --- | --- |
| `getVirtualOrigin()` | `Point` | Calculates the minimum X and Y coordinates across all screens |

**Sources:** [src/io/github/samera2022/mouse_macros/util/ScreenUtil.java L5-L42](https://github.com/Samera2022/MouseMacros/blob/1eb6620b/src/io/github/samera2022/mouse_macros/util/ScreenUtil.java#L5-L42)

## Usage Examples

### Recording Phase Usage

When `GlobalMouseListener` receives a mouse event during recording:

```yaml
Input: nativeMousePressed event at global coordinates (2500, 800)
Call: ScreenUtil.normalizeToVirtualOrigin(2500, 800)
Virtual Origin: (-1920, -200)
Output: Point(4420, 1000) stored in MouseAction
```

### Playback Phase Usage

When `MouseAction.perform()` is called during playback:

```python
Input: Normalized coordinates (4420, 1000) from stored action
Call: ScreenUtil.denormalizeFromVirtualOrigin(4420, 1000)
Virtual Origin: (-1920, -200)
Global Coords: (2500, 800)
Primary Screen: origin (0, 0), scale (2.0, 2.0)
Robot Coords: (1250, 400)
Output: Robot.mouseMove(1250, 400)
```

**Sources:** [src/io/github/samera2022/mouse_macros/util/ScreenUtil.java L7-L41](https://github.com/Samera2022/MouseMacros/blob/1eb6620b/src/io/github/samera2022/mouse_macros/util/ScreenUtil.java#L7-L41)

## Technical Considerations

### Thread Safety

The `ScreenUtil` class methods are stateless and thread-safe. Each method invocation queries the current graphics environment state and performs calculations without modifying shared state.

### Performance

The `getVirtualOrigin` method is called during each coordinate transformation. This involves querying all connected displays through the Java AWT API. For typical macro playback with delays between actions, this overhead is negligible.

### Coordinate Precision

Coordinate calculations use integer arithmetic for X and Y values. The DPI scaling division uses floating-point arithmetic with `Math.round()` to ensure accurate conversion to physical pixel coordinates.

### Monitor Configuration Changes

If the user changes their monitor configuration (adds/removes displays, changes arrangement) between recording and playback, the normalized coordinates ensure macros still execute at the correct relative positions. However, if the recording was made on a screen that no longer exists, the action may execute outside the visible screen area.

**Sources:** [src/io/github/samera2022/mouse_macros/util/ScreenUtil.java L1-L42](https://github.com/Samera2022/MouseMacros/blob/1eb6620b/src/io/github/samera2022/mouse_macros/util/ScreenUtil.java#L1-L42)