Skip to content

Commit c819c80

Browse files
karlspacekarlspace
authored
Merge pull request #1 from bauer-group/feature-20250901
feat: Extensive Implementation and Test Coverage
2 parents 3ab12ac + 9cf5d8b commit c819c80

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+43117
-332
lines changed

‎.github/workflows/README.md‎

Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,185 @@
1+
# GitHub Actions Workflows
2+
3+
This directory contains the CI/CD workflows for the NocoDB Simple Client project.
4+
5+
## Workflows Overview
6+
7+
### 1. `python-automatic-release.yml`
8+
**Purpose**: Automated releases on main branch
9+
**Trigger**: Push to `main` branch
10+
**Tests**: Unit tests only (fast, no external dependencies)
11+
**Dependencies**: None (uses mocks)
12+
13+
### 2. `feature-test.yml`
14+
**Purpose**: Comprehensive testing on feature branches
15+
**Trigger**: Push to `feature-*` branches
16+
**Tests**: Unit tests + Integration tests with live NocoDB
17+
**Dependencies**: Automatic NocoDB setup with SQLite
18+
19+
## Workflow Details
20+
21+
### Release Workflow (`python-automatic-release.yml`)
22+
- **Fast execution** (~2-3 minutes)
23+
- **No external dependencies**
24+
- **All Python versions** (3.8-3.12)
25+
- **Unit tests only** via `--ci` flag
26+
- **Automatic versioning** and PyPI publishing
27+
28+
### Feature Testing Workflow (`feature-test.yml`)
29+
30+
#### Job 1: Unit Tests
31+
- **Matrix**: Python 3.9, 3.11, 3.12
32+
- **Duration**: ~2-4 minutes
33+
- **Dependencies**: None (mocked)
34+
- **Purpose**: Fast feedback on basic functionality
35+
36+
#### Job 2: Integration Tests
37+
- **Setup**: Automatic NocoDB instance with SQLite
38+
- **User Creation**: Automated admin user setup
39+
- **API Token**: Dynamic token generation
40+
- **Tests**: Full integration test suite
41+
- **Duration**: ~8-12 minutes
42+
- **Environment Variables**: Automatically configured
43+
44+
#### Job 3: Performance Tests (Optional)
45+
- **Trigger**: PR label `test-performance`
46+
- **Setup**: Optimized NocoDB instance
47+
- **Tests**: Performance benchmarks
48+
- **Reduced Dataset**: CI-appropriate test sizes
49+
50+
## NocoDB Setup Process
51+
52+
The feature workflow automatically:
53+
54+
1. **Starts NocoDB Container**:
55+
```bash
56+
docker run -d --name nocodb-test \
57+
-p 8080:8080 \
58+
-e NC_DB="sqlite3://data/nc.db" \
59+
-e NC_AUTH_JWT_SECRET="test-jwt-secret-$(date +%s)" \
60+
-e NC_DISABLE_TELE=true \
61+
nocodb/nocodb:latest
62+
```
63+
64+
2. **Creates Admin User**:
65+
```bash
66+
curl -X POST /api/v1/auth/user/signup \
67+
-d '{"email":"test@example.com","password":"TestPassword123!"}'
68+
```
69+
70+
3. **Gets API Token**:
71+
```bash
72+
curl -X POST /api/v1/auth/user/signin \
73+
-d '{"email":"test@example.com","password":"TestPassword123!"}'
74+
```
75+
76+
4. **Configures Environment**:
77+
```bash
78+
NOCODB_BASE_URL=http://localhost:8080
79+
NOCODB_TOKEN=$TOKEN
80+
TEST_TABLE_PREFIX=gh_test_
81+
MAX_FILE_SIZE_MB=1
82+
```
83+
84+
## Environment Configuration
85+
86+
### Automatic Environment Variables
87+
The workflow automatically configures:
88+
89+
| Variable | Value | Description |
90+
|----------|-------|-------------|
91+
| `NOCODB_BASE_URL` | `http://localhost:8080` | NocoDB instance URL |
92+
| `NOCODB_TOKEN` | `${{ steps.setup-nocodb.outputs.token }}` | Dynamic API token |
93+
| `TEST_TABLE_PREFIX` | `gh_test_` | Prefix for test tables |
94+
| `CLEANUP_TEST_DATA` | `true` | Auto-cleanup enabled |
95+
| `RUN_INTEGRATION_TESTS` | `true` | Enable integration tests |
96+
| `TEST_TIMEOUT` | `60` | Extended timeout for CI |
97+
| `MAX_FILE_SIZE_MB` | `1` | File upload limit |
98+
| `PERFORMANCE_TEST_RECORDS` | `50` | Reduced for CI speed |
99+
| `BULK_TEST_BATCH_SIZE` | `10` | Small batches for CI |
100+
101+
### Error Handling & Debugging
102+
103+
#### Automatic Debugging on Failure:
104+
```bash
105+
# Show NocoDB logs
106+
docker logs nocodb-test
107+
108+
# Show container status
109+
docker ps -a
110+
111+
# Test API connectivity
112+
curl -v http://localhost:8080/api/v1/health
113+
```
114+
115+
#### Cleanup on Success/Failure:
116+
```bash
117+
docker stop nocodb-test || true
118+
docker rm nocodb-test || true
119+
rm -rf ./nocodb-data || true
120+
```
121+
122+
## Usage Examples
123+
124+
### Triggering Feature Tests
125+
```bash
126+
# Push to feature branch triggers automatic testing
127+
git checkout -b feature/new-functionality
128+
git push origin feature/new-functionality
129+
```
130+
131+
### Adding Performance Tests
132+
```bash
133+
# Add label to PR to trigger performance tests
134+
gh pr edit --add-label "test-performance"
135+
```
136+
137+
### Local Testing Equivalent
138+
```bash
139+
# Same tests locally
140+
python scripts/run-all.py --integration # Integration tests
141+
python scripts/run-all.py --performance # Performance tests
142+
python scripts/run-all.py --all-tests # Everything
143+
```
144+
145+
## Troubleshooting
146+
147+
### Common Issues
148+
149+
1. **NocoDB startup timeout**:
150+
- Increased timeout to 120s
151+
- Multiple health check methods
152+
- Fallback token generation
153+
154+
2. **API token extraction failure**:
155+
- Multiple extraction methods
156+
- Fallback token generation
157+
- Graceful error handling
158+
159+
3. **Test data conflicts**:
160+
- Unique table prefixes (`gh_test_`, `perf_test_`)
161+
- Automatic cleanup
162+
- Isolated containers per job
163+
164+
### Debug Steps
165+
166+
1. **Check workflow logs** in GitHub Actions
167+
2. **Review NocoDB container logs** (shown on failure)
168+
3. **Test API endpoints manually** using curl commands
169+
4. **Run locally** with same environment variables
170+
171+
## Performance Considerations
172+
173+
### Optimizations Applied:
174+
- **Reduced Python matrix** for feature tests (3 versions vs 5)
175+
- **SQLite database** (faster than PostgreSQL/MySQL)
176+
- **Disabled telemetry** (`NC_DISABLE_TELE=true`)
177+
- **Reduced test datasets** for CI environment
178+
- **Parallel job execution** where possible
179+
- **Efficient cleanup** to minimize resource usage
180+
181+
### Expected Durations:
182+
- **Unit tests**: 2-4 minutes per Python version
183+
- **Integration tests**: 8-12 minutes total
184+
- **Performance tests**: 10-15 minutes (when enabled)
185+
- **Total feature workflow**: ~15-20 minutes

‎.github/workflows/feature-test.yml‎

Lines changed: 182 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,182 @@
1+
name: 🧪 Feature Integration Tests
2+
3+
on:
4+
push:
5+
branches: [ feature-* ]
6+
pull_request:
7+
branches: [ feature-* ]
8+
9+
workflow_dispatch:
10+
11+
workflow_call:
12+
13+
jobs:
14+
# 🔬 Unit tests on multiple Python versions (fast)
15+
unit-tests:
16+
name: 🧪 Unit Tests
17+
runs-on: ubuntu-latest
18+
strategy:
19+
matrix:
20+
python-version: ["3.12"] # Use Python 3.12 for tests
21+
22+
steps:
23+
- name: 📥 Checkout code
24+
uses: actions/checkout@v4
25+
26+
- name: 🐍 Set up Python ${{ matrix.python-version }}
27+
uses: actions/setup-python@v4
28+
with:
29+
python-version: ${{ matrix.python-version }}
30+
31+
- name: 📦 Cache pip dependencies
32+
uses: actions/cache@v3
33+
with:
34+
path: ~/.cache/pip
35+
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements*.txt', '**/pyproject.toml') }}
36+
restore-keys: |
37+
${{ runner.os }}-pip-
38+
39+
- name: ⚙️ Install dependencies
40+
run: |
41+
python -m pip install --upgrade pip
42+
pip install -e .
43+
pip install -e ".[dev]"
44+
45+
- name: 🧪 Run unit tests
46+
run: |
47+
python scripts/run-all.py --ci
48+
env:
49+
PYTHONPATH: ${{ github.workspace }}/src
50+
51+
# 🔗 Integration tests with Python-managed NocoDB instance
52+
integration-test:
53+
name: 🔗 Integration Tests (Python-Managed)
54+
runs-on: ubuntu-latest
55+
needs: unit-tests
56+
57+
steps:
58+
- name: 📥 Checkout code
59+
uses: actions/checkout@v4
60+
61+
- name: 🐍 Set up Python 3.12
62+
uses: actions/setup-python@v4
63+
with:
64+
python-version: "3.12"
65+
66+
- name: ⚙️ Install dependencies
67+
run: |
68+
python -m pip install --upgrade pip
69+
pip install -e .
70+
pip install -e ".[dev]"
71+
72+
- name: 🐳 Setup NocoDB Container
73+
run: |
74+
# Make script executable
75+
chmod +x scripts/ci-setup.sh
76+
77+
# Run setup script
78+
CONTAINER_NAME=nocodb-integration-test \
79+
NOCODB_PORT=8080 \
80+
NC_ADMIN_EMAIL=test@integration.local \
81+
NC_ADMIN_PASSWORD=IntegrationTest123 \
82+
./scripts/ci-setup.sh setup
83+
84+
# Verify that config files were created
85+
echo "=== Checking generated config files ==="
86+
ls -la nocodb-config.json .env.test 2>/dev/null || echo "Config files not found!"
87+
88+
# Show config content (without sensitive data in logs)
89+
if [ -f nocodb-config.json ]; then
90+
echo "✅ nocodb-config.json created"
91+
cat nocodb-config.json | jq 'del(.NOCODB_TOKEN)' || cat nocodb-config.json
92+
fi
93+
94+
- name: 🔗 Run integration tests
95+
run: |
96+
# Verify config files are available
97+
if [ ! -f nocodb-config.json ]; then
98+
echo "❌ ERROR: nocodb-config.json not found!"
99+
exit 1
100+
fi
101+
102+
# Run tests with config file
103+
python -m pytest tests/test_integration.py -v --tb=short
104+
env:
105+
PYTHONPATH: ${{ github.workspace }}/src
106+
SKIP_INTEGRATION: 0
107+
# Test configuration
108+
TEST_TIMEOUT: 300
109+
MAX_FILE_SIZE_MB: 1
110+
CLEANUP_TEST_DATA: true
111+
112+
- name: 🔍 Show Docker logs on failure
113+
if: failure()
114+
run: |
115+
echo "=== Available Docker containers ==="
116+
docker ps -a
117+
echo "=== Docker system info ==="
118+
docker system df
119+
echo "=== Check for NocoDB container logs ==="
120+
docker logs nocodb-integration-test 2>/dev/null || echo "Container not found or no logs"
121+
122+
- name: 🧹 Cleanup Docker containers
123+
if: always()
124+
run: |
125+
# Use cleanup script
126+
chmod +x scripts/ci-setup.sh
127+
CONTAINER_NAME=nocodb-integration-test ./scripts/ci-setup.sh cleanup
128+
129+
# ⚡ Optional performance tests (when PR has performance label)
130+
performance-test:
131+
name: ⚡ Performance Tests (Python-managed)
132+
runs-on: ubuntu-latest
133+
needs: unit-tests
134+
if: contains(github.event.pull_request.labels.*.name, 'test-performance')
135+
136+
steps:
137+
- name: 📥 Checkout code
138+
uses: actions/checkout@v4
139+
140+
- name: 🐍 Set up Python 3.12
141+
uses: actions/setup-python@v4
142+
with:
143+
python-version: "3.12"
144+
145+
- name: ⚙️ Install dependencies
146+
run: |
147+
python -m pip install --upgrade pip
148+
pip install -e .
149+
pip install -e ".[dev]"
150+
151+
- name: 🐳 Setup NocoDB Container
152+
run: |
153+
# Make script executable
154+
chmod +x scripts/ci-setup.sh
155+
156+
# Run setup script with performance test configuration
157+
CONTAINER_NAME=nocodb-integration-test \
158+
NOCODB_PORT=8080 \
159+
NC_ADMIN_EMAIL=test@integration.local \
160+
NC_ADMIN_PASSWORD=IntegrationTest123 \
161+
./scripts/ci-setup.sh setup
162+
env:
163+
PYTHONPATH: ${{ github.workspace }}/src
164+
165+
- name: ⚡ Run Python-managed performance tests
166+
run: |
167+
python -m pytest tests/test_integration.py::TestIntegration::test_bulk_operations -v --tb=short
168+
python -m pytest tests/test_performance.py -v --tb=short 2>/dev/null || echo "Performance tests not available"
169+
env:
170+
PYTHONPATH: ${{ github.workspace }}/src
171+
SKIP_INTEGRATION: 0
172+
# Performance test configuration
173+
PERFORMANCE_TEST_RECORDS: 100
174+
BULK_TEST_BATCH_SIZE: 20
175+
TEST_TIMEOUT: 600
176+
177+
- name: 🧹 Cleanup performance test containers
178+
if: always()
179+
run: |
180+
# Use cleanup script
181+
chmod +x scripts/ci-setup.sh
182+
CONTAINER_NAME=nocodb-integration-test ./scripts/ci-setup.sh cleanup

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -206,6 +206,9 @@ marimo/_static/
206206
marimo/_lsp/
207207
__marimo__/
208208

209+
# Tools
210+
.benchmarks/
211+
209212
# IDE
210213
.vscode/
211214
.claude/

0 commit comments

Comments
 (0)