Skip to content

experiments.md

Latest commit

 

History

History
288 lines (220 loc) · 6.05 KB

experiments.md

File metadata and controls

288 lines (220 loc) · 6.05 KB

Experiment Guide

This guide provides templates for common power measurement experiments.

Experiment 1: IDE Comparison

Goal: Compare power consumption of different IDEs/editors during typical coding tasks.

Setup

  1. Close all applications except the editor being tested
  2. Open the same project in each editor
  3. Prepare a script of actions (e.g., open 5 files, search, build)

Procedure

# Test VS Code
./scripts/start_session.sh 300  # 5 minutes
# Perform scripted actions
# Wait for session to complete

# Rename output
mv data data_vscode

# Test Vim
./scripts/start_session.sh 300
# Perform same actions
mv data data_vim

# Analyze
python -m src.correlate analyze --input-dir data_vscode --plot
python -m src.correlate analyze --input-dir data_vim --plot

Metrics to Compare

  • Average power (avg_w)
  • Baseline power (idle state)
  • Energy per key press
  • Total energy for the task

Experiment 2: Browser Tab Impact

Goal: Measure how open browser tabs affect power consumption.

Procedure

# Baseline: browser closed
./scripts/start_session.sh 120
mv data data_baseline

# 1 tab (blank page)
./scripts/start_session.sh 120
mv data data_1tab

# 10 tabs (various sites)
./scripts/start_session.sh 120
mv data data_10tabs

# 50 tabs
./scripts/start_session.sh 120
mv data data_50tabs

Analysis

import json

results = {}
for name in ['baseline', '1tab', '10tabs', '50tabs']:
    with open(f'data_{name}/session_summary.json') as f:
        results[name] = json.load(f)

for name, data in results.items():
    print(f"{name}: {data['power_stats']['avg_w']:.1f} W")

Experiment 3: Video Codec Comparison

Goal: Compare power consumption of different video codecs.

Setup

  • Prepare same video in H.264, H.265, VP9, AV1 formats
  • Use same player for all tests
  • Disable hardware acceleration to measure CPU impact (or enable to measure GPU)

Procedure

for codec in h264 h265 vp9 av1; do
    ./scripts/start_session.sh 180 &
    sleep 2
    mpv --no-hwdec video_${codec}.mkv
    wait
    mv data data_${codec}
done

Experiment 4: Compilation Energy

Goal: Measure energy cost of compiling a project.

Procedure

# Clean build
make clean

# Start measurement
./scripts/start_session.sh 600 &
SESSION_PID=$!

# Build project
time make -j$(nproc)

# Stop measurement
kill -TERM $SESSION_PID
wait

# Analyze
python -m src.correlate analyze --plot

Key Metrics

  • Total energy (Wh) for complete build
  • Peak power during compilation
  • Comparison: parallel (-j8) vs sequential (-j1)

Experiment 5: Sleep States

Goal: Measure power in different system states.

Procedure

# Active use
./scripts/start_session.sh 60
# Type continuously
mv data data_active

# Idle (screen on)
./scripts/start_session.sh 60
# Don't touch anything
mv data data_idle_screen_on

# Idle (screen off)
xset dpms force off
./scripts/start_session.sh 60
mv data data_idle_screen_off
xset dpms force on

Experiment 6: Application Startup

Goal: Measure energy cost of launching applications.

Procedure

# Prepare: close all apps
./scripts/start_session.sh 30 &
sleep 5
# Launch application
firefox &
sleep 20
# Measurement continues until Firefox fully loaded
wait

python -m src.correlate analyze --plot

Metrics

  • Energy in first 10 seconds after launch
  • Peak power during startup
  • Time to reach baseline power

Experiment 7: Network Activity

Goal: Compare power during different network activities.

Tests

  1. Idle (no network)
  2. Downloading large file
  3. Video streaming (720p, 1080p, 4K)
  4. Video call
# Download test
./scripts/start_session.sh 120 &
sleep 5
wget -O /dev/null http://speedtest.example.com/100MB.bin
wait
mv data data_download

# Streaming test
./scripts/start_session.sh 120 &
sleep 5
mpv https://example.com/video_1080p.m3u8
wait
mv data data_streaming

Data Analysis Tips

Comparing Sessions

import json
from pathlib import Path

def load_summary(path):
    with open(Path(path) / 'session_summary.json') as f:
        return json.load(f)

# Load multiple sessions
sessions = {
    'vscode': load_summary('data_vscode'),
    'vim': load_summary('data_vim'),
}

# Compare average power
for name, data in sessions.items():
    stats = data['power_stats']
    print(f"{name}:")
    print(f"  Avg: {stats['avg_w']:.1f} W")
    print(f"  Max: {stats['max_w']:.1f} W")
    print()

Statistical Significance

For reliable comparisons:

  1. Run each experiment 3-5 times
  2. Calculate mean and standard deviation
  3. Use t-test for significance (if samples are normal)
import statistics

runs = [45.2, 44.8, 46.1, 45.5, 44.9]  # avg power from 5 runs
print(f"Mean: {statistics.mean(runs):.1f} W")
print(f"StdDev: {statistics.stdev(runs):.2f} W")

Controlling Variables

To get reproducible results:

  • Same room temperature
  • Same battery level (or plugged in)
  • Same background services
  • Same display brightness
  • Wait for system to "settle" after boot (5-10 minutes)

Reporting Results

When sharing experiment results, include:

  1. Hardware: CPU model, RAM, GPU, SSD/HDD
  2. OS: Distribution, kernel version
  3. Shelly firmware version
  4. Methodology: What actions were performed, timing
  5. Raw data: Link to CSV files
  6. Analysis: Summary statistics, plots
  7. Conclusions: What the data shows (carefully!)

Example Report Template

# Experiment: VS Code vs Neovim Power Comparison

## Setup
- Hardware: AMD Ryzen 7 5800X, 32GB RAM, RTX 3070
- OS: Ubuntu 22.04, kernel 6.5.0
- Shelly Plug S Gen3, firmware 1.2.0

## Methodology
1. Opened same TypeScript project (100 files)
2. Performed scripted actions: open 5 files, search, format, build
3. 5-minute session each, 3 runs per editor

## Results

| Editor | Avg Power (W) | StdDev | Energy/Build (J) |
|--------|---------------|--------|------------------|
| VS Code | 48.2 | 2.1 | 125 |
| Neovim | 32.5 | 1.8 | 89 |

## Conclusion
Neovim consumed 33% less power on average for equivalent tasks.
Build operations showed 29% energy reduction.

## Data
[Link to CSV files]