Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
ff95ab0
Add tests with mocked server responses
tobixen Nov 23, 2025
4b04278
Fix test_calendar_auto_creation_detected mock
tobixen Nov 23, 2025
2a1b907
New checks save-load.task.recurrences
tobixen Nov 23, 2025
26531d8
Expand search tests and remove fragile mocked tests
tobixen Nov 27, 2025
1ee4f55
Expand search tests and remove fragile mocked tests
tobixen Nov 27, 2025
76657ec
Add sync token support detection
tobixen Nov 29, 2025
483863b
Fix CheckSyncToken API usage and add comprehensive tests
tobixen Nov 29, 2025
4d40571
Add proper cleanup for sync token test event
tobixen Nov 29, 2025
bcdf933
Add alarm time-range search support detection
tobixen Nov 30, 2025
8a6e890
Add principal search support detection
tobixen Nov 30, 2025
39d46a5
Add duplicate UID cross-calendar support detection
tobixen Nov 30, 2025
16c40f4
Rename duplicate-uid.cross-calendar to save.duplicate-uid.cross-calendar
tobixen Nov 30, 2025
e434760
Add search.time-range.accurate compatibility check
tobixen Dec 2, 2025
4fe9c7e
Add freebusy-query.rfc4791 feature check and fix checker bugs
tobixen Dec 2, 2025
37f8109
Simplify CheckDuplicateUID to reuse event from PrepareCalendar
tobixen Dec 2, 2025
9868286
Working with Bedework
tobixen Dec 3, 2025
728f5ab
Add search.text.by-uid feature detection in caldav-server-tester
tobixen Dec 3, 2025
2fa38c3
Add timezone test for GitHub issue #372 and fix syntax error
tobixen Dec 3, 2025
8ace88d
Detect and handle Zimbra moving events instead of copying
tobixen Dec 3, 2025
2b0c9cb
Use disable_fallback parameter in CheckSyncToken
tobixen Dec 3, 2025
77d083b
some refacotrings
tobixen Dec 14, 2025
3e48ce5
don't catch every exception
tobixen Dec 14, 2025
ba1ecbf
Fix alarm search test to reuse persistent events
tobixen Dec 14, 2025
b297cbe
Improve alarm test to handle duplicate UID errors gracefully
tobixen Dec 14, 2025
f688958
Add workaround for broken caldav search parameters
tobixen Dec 14, 2025
69c16e7
Fix CheckAlarmSearch to use proper UID search
tobixen Dec 15, 2025
58da704
Move alarm test event creation to PrepareCalendar
tobixen Dec 15, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
146 changes: 146 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
.python-version

# pipenv
Pipfile.lock

# poetry
poetry.lock

# pdm
.pdm.toml

# PEP 582
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# Editor backup files
*~
.*.swp
.*.swo

# IDE
.vscode/
.idea/
*.iml
36 changes: 36 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,42 @@ This file should adhere to [Keep a Changelog](https://keepachangelog.com/en/1.1.

This project should adhere to [Semantic Versioning](https://semver.org/spec/v2.0.0.html), though some earlier releases may be incompatible with the SemVer standard.

## [Unreleased]

### Added
- Expanded search feature coverage with new feature flags:
- `search.text` - Basic text/summary search
- `search.text.case-sensitive` - Case-sensitive text matching (default behavior)
- `search.text.case-insensitive` - Case-insensitive text matching via CalDAVSearcher
- `search.text.substring` - Substring matching for text searches
- `search.is-not-defined` - Property filter with is-not-defined operator
- `search.text.category` - Category search support
- `search.text.category.substring` - Substring matching for category searches
- `post_filter=False` parameter to all server behavior tests to ensure testing actual server responses
- New `CheckSyncToken` check class for RFC6578 sync-collection reports:
- Tests for sync token support (full/fragile/unsupported)
- Detects time-based sync tokens (second-precision, requires sleep(1) between operations)
- Detects fragile sync tokens (occasionally returns extra content due to race conditions)
- Tests sync-collection reports after object deletion
- New `CheckAlarmSearch` check class for alarm time-range searches (RFC4791 section 9.9):
- Tests if server supports searching for events based on when their alarms trigger
- Verifies correct filtering of alarm times vs event times
- New `CheckPrincipalSearch` check class for principal search operations:
- Tests basic principal access
- Tests searching for own principal by display name (`principal-search.by-name.self`)
- Tests listing all principals (`principal-search.list-all`)
- Note: Full `principal-search.by-name` testing requires multiple users and is not yet implemented
- New `CheckDuplicateUID` check class for duplicate UID handling:
- Tests if server allows events with same UID in different calendars (`save.duplicate-uid.cross-calendar`)
- Detects if duplicates are silently ignored or rejected with errors
- Verifies events are treated as separate entities when allowed

### Changed
- Improved `search.comp-type-optional` test with additional text search validation

### Fixed
- `create-calendar` feature detection to not incorrectly mark mkcol method as standard calendar creation

## [0.1] - [2025-11-08]

This release corresponds with the caldav version 2.1.2
Expand Down
205 changes: 205 additions & 0 deletions TEST_PERFORMANCE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,205 @@
# Test Performance Statistics

This document provides detailed performance statistics for the caldav-server-tester test suite.

## Quick Summary

| Test Category | Count | Total Time | Avg Time/Test | Memory Usage |
|--------------|-------|------------|---------------|--------------|
| **Fast Tests** (unit) | 54 | ~1.2s | <5ms | ~76 MB |
| **Slow Tests** (mocked server) | 14 | ~70s | ~5s | ~77 MB |
| **Total** | 68 | ~71s | ~1s | ~77 MB |

## Running Tests

### Fast Tests Only (Recommended for Development)
```bash
# Run only fast unit tests
pytest -m "not slow"

# With verbose output
pytest -m "not slow" -v

# With duration statistics
pytest -m "not slow" --durations=10
```

**Performance:** 54 tests in ~1.2 seconds

### All Tests (Including Slow Mocked Server Tests)
```bash
# Run all tests
pytest

# With detailed timing
pytest --durations=20
```

**Performance:** 68 tests in ~71 seconds

### Slow Tests Only
```bash
pytest -m "slow"
```

**Performance:** 14 tests in ~70 seconds

## Detailed Test Timing

### Fast Unit Tests (< 5ms each)

All 54 fast unit tests complete in under 5 milliseconds each:

- **test_ai_check_base.py**: 18 tests for Check base class
- set_feature method: 8 tests
- feature_checked method: 3 tests
- run_check dependency resolution: 7 tests

- **test_ai_checker.py**: 24 tests for ServerQuirkChecker
- Initialization: 7 tests
- Properties: 2 tests
- Methods (check_one, report, cleanup): 15 tests

- **test_ai_filters.py**: 12 tests for _filter_2000 function
- Date range filtering
- Edge cases and boundary conditions

### Slow Mocked Server Tests

These tests run actual check logic with mocked server responses:

| Test | Duration | Category |
|------|----------|----------|
| `test_calendar_auto_creation_detected` | ~60s | CheckMakeDeleteCalendar |
| `test_calendar_creation_with_displayname` | ~10s | CheckMakeDeleteCalendar |
| Other mocked tests | <0.5s each | Various |

**Why these are slow:**
- They execute the full `_run_check()` logic
- Complex retry/fallback mechanisms
- Multiple calendar creation/deletion cycles
- Extensive feature detection logic

## Resource Usage

### CPU Usage
```
CPU: 99% (single-threaded)
Context switches: ~75 involuntary
Page faults: ~22,000 minor
```

### Memory Usage
```
Maximum resident set size: ~77 MB
Average memory footprint: Stable throughout execution
No memory leaks detected
```

### I/O
```
File system inputs: 0
File system outputs: 32 (test result files)
No network I/O (all tests are offline)
```

## Performance Tips

### For Development (Fast Feedback)
```bash
# Run only fast tests - get results in ~1 second
pytest -m "not slow" -x

# Run specific test file
pytest tests/test_ai_filters.py

# Run specific test
pytest tests/test_ai_filters.py::TestFilter2000::test_filter_includes_dtstart_at_start_boundary
```

### For CI/CD
```bash
# Run all tests with coverage
pytest --cov=caldav_server_tester --cov-report=html

# Run with JUnit XML output for CI
pytest --junitxml=test-results.xml

# Parallel execution (if pytest-xdist installed)
pytest -n auto
```

### Profiling Individual Tests
```bash
# Show detailed timing for all tests
pytest --durations=0 -vv

# Profile specific test with cProfile
python -m cProfile -o profile.stats -m pytest tests/test_ai_filters.py
python -c "import pstats; p=pstats.Stats('profile.stats'); p.sort_stats('cumulative'); p.print_stats(20)"
```

## Performance Optimization

The test suite is optimized for:

1. **Fast Iteration**: Unit tests run in ~1 second for rapid development
2. **Comprehensive Coverage**: 71 total tests (54 fast + 14 slow + 3 skipped)
3. **Selective Execution**: Use markers to run appropriate test subset
4. **Low Memory**: < 80 MB memory footprint
5. **No Dependencies**: All tests run offline without external services

## Monitoring Performance Over Time

To track test performance over time:

```bash
# Generate timing report
pytest --durations=0 --tb=no > timing_report.txt

# Compare with previous run
diff timing_report_old.txt timing_report.txt
```

For continuous monitoring, consider integrating with CI to track:
- Total test execution time
- Individual slow test trends
- Memory usage patterns
- Test failure rates

## Troubleshooting Slow Tests

If tests are slower than expected:

1. **Check for slow markers**: Some tests are intentionally slow
```bash
pytest --co -m slow # List slow tests
```

2. **Profile specific test**:
```bash
pytest tests/test_checks_with_mocks.py::TestCheckMakeDeleteCalendar::test_calendar_auto_creation_detected --durations=0 -vv
```

3. **Check system load**: Ensure system isn't under heavy load
```bash
top # Check CPU/memory availability
```

4. **Reduce test scope**: Run subset of tests
```bash
pytest tests/test_ai_filters.py # Just one file
```

## Generated Reports

- **Duration Report**: Use `pytest --durations=N` to see N slowest tests
- **Coverage Report**: Use `pytest --cov` for coverage analysis
- **JUnit XML**: Use `pytest --junitxml` for CI integration
- **HTML Report**: Use `pytest-html` plugin for visual reports

---

**Last Updated**: Generated from test run statistics
**Test Framework**: pytest 8.4.2
**Python Version**: 3.13.7
Loading