Refactor Docker/Kubernetes deployment to use sidecar pattern for IPFS

[bryanchriswhite]

Summary

Currently, the PinShare deployment runs both the IPFS daemon and the PinShare application in a single container. This violates Kubernetes best practices and causes several operational issues, including permission conflicts, difficult resource management, and poor observability.

This issue proposes refactoring the deployment to use the sidecar pattern, where IPFS runs in a separate container within the same pod.

Current Architecture Problems

1. Single Point of Failure

• If either IPFS or PinShare crashes, the entire container restarts
• No independent process supervision
• Both processes managed via shell backgrounding (&) which is fragile

2. Permission Conflicts

• IPFS daemon needs to run as the ipfs user (UID 1000)
• PinShare application has different permission requirements
• Init container creates files as root, causing permission denied errors
• Current issue: Error: error loading plugins: open /data/ipfs/config: permission denied

3. Resource Management

• Cannot set independent CPU/memory limits for IPFS vs PinShare
• Cannot independently scale IPFS and PinShare
• Resource contention between processes is not visible to Kubernetes scheduler

4. Health Checks & Observability

• Cannot independently monitor IPFS daemon health vs PinShare API health
• Liveness probe only checks IPFS, not PinShare
• Readiness probe only checks PinShare, not IPFS
• Mixed logs from both processes make debugging difficult

5. Update & Maintenance Challenges

• Cannot update IPFS independently from PinShare
• Docker image contains both IPFS (from Kubo) and PinShare binaries
• Larger image size and longer build times
• Difficult to test IPFS and PinShare separately

6. Startup Complexity

The entrypoint.sh script has brittle startup logic:

/usr/local/bin/start_ipfs daemon --migrate=true --agent-version-suffix=docker &
sleep 10  # Hope IPFS starts in 10 seconds
ipfs config ...  # Configure CORS
sleep 50  # Wait more just to be safe
/opt/pinshare/bin/pinshare  # Finally start PinShare

Hard-coded sleep timers are unreliable and waste time.

Proposed Solution: Sidecar Pattern

Architecture

Pod: pinshare-backend
├── Container: ipfs (sidecar)
│   ├── Image: ipfs/kubo:latest
│   ├── Ports: 5001 (API), 8080 (Gateway), 4001 (Swarm)
│   ├── Volume: ipfs-data (PVC)
│   ├── Health: IPFS dag stat check
│   └── Resources: Independent limits
│
├── Container: pinshare (main)
│   ├── Image: pinshare:latest (simplified, no IPFS)
│   ├── Ports: 9090 (API), 50001 (libp2p)
│   ├── Volume: backend-data (PVC)
│   ├── Health: HTTP /health check
│   ├── Resources: Independent limits
│   └── Connects to: localhost:5001 (IPFS)
│
└── InitContainer: ipfs-config
    ├── Configures IPFS CORS before startup
    └── Sets proper permissions

Benefits

✅ Proper Separation of Concerns

• Each container has a single responsibility
• IPFS manages storage and IPFS protocol
• PinShare manages business logic and P2P metadata

✅ Independent Health Monitoring

# IPFS Container
livenessProbe:
  exec:
    command: ["ipfs", "dag", "stat", "/ipfs/QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn"]

# PinShare Container  
livenessProbe:
  httpGet:
    path: /health
    port: 9090

✅ Independent Resource Management

# IPFS Container
resources:
  requests:
    memory: "512Mi"
    cpu: "250m"
  limits:
    memory: "2Gi"
    cpu: "1000m"

# PinShare Container
resources:
  requests:
    memory: "256Mi"
    cpu: "100m"
  limits:
    memory: "1Gi"
    cpu: "500m"

✅ Cleaner Logs

• kubectl logs pod/pinshare-xxx -c ipfs → Only IPFS logs
• kubectl logs pod/pinshare-xxx -c pinshare → Only PinShare logs
• Better for log aggregation systems (ELK, Loki, etc.)

✅ Faster Iteration

• Rebuild only PinShare without touching IPFS
• Use standard ipfs/kubo image directly (no custom build)
• Smaller PinShare image (no IPFS binaries)
• Faster Tilt hot reload

✅ Easier Testing

• Test IPFS independently
• Test PinShare with mock IPFS
• Integration tests use the same sidecar pattern

Implementation Plan

Phase 1: Refactor Dockerfile

Current: Multi-stage build with Go builder + IPFS layer + Debian
Proposed: Simple Go builder → minimal runtime (no IPFS)

FROM golang:latest AS builder
WORKDIR /build
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go build -o pinshare .

FROM debian:stable-slim
RUN apt-get update && apt-get install -y \
    chromium \
    ca-certificates \
    clamav clamav-daemon \
    && rm -rf /var/lib/apt/lists/*

COPY --from=builder /build/pinshare /usr/local/bin/pinshare
EXPOSE 9090 50001
CMD ["pinshare"]

Phase 2: Update Kubernetes Manifests

Deployment Changes:

• Add IPFS sidecar container
• Remove IPFS from main container
• Add init container for IPFS CORS configuration
• Update volume mounts (shared data, separate IPFS repo)
• Independent health checks

Service Changes:

• Ensure all necessary ports are exposed
• IPFS API (5001) accessible from PinShare container via localhost

Phase 3: Update Tiltfile

# Simplified live_update for PinShare (no IPFS layers)
docker_build(
    'pinshare-backend',
    '.',
    dockerfile='Dockerfile',
    live_update=[
        sync('.', '/app'),
        run('go build -o /usr/local/bin/pinshare .',
            trigger=['./cmd', './internal', './pkg', 'go.mod', 'go.sum']),
    ],
)

# IPFS uses standard image (no rebuild needed)

Phase 4: Update Documentation

• Update README with new architecture
• Update k8s/README.md with sidecar explanation
• Add architecture diagrams
• Document troubleshooting for each container

Phase 5: Docker Compose (Optional)

Consider updating docker-compose.yml to also use separate services for consistency:

services:
  ipfs:
    image: ipfs/kubo:latest
    volumes:
      - ipfs-data:/data/ipfs
    ports:
      - "5001:5001"
      - "8080:8080"
      - "4001:4001"
  
  pinshare:
    build: .
    depends_on:
      - ipfs
    environment:
      - IPFS_API=http://ipfs:5001
    ports:
      - "9090:9090"
      - "50001:50001"

Migration Strategy

Backwards Compatibility

1. Keep existing Dockerfile with a -legacy suffix for rollback
2. Create new Dockerfile with sidecar-compatible build
3. Both can coexist during transition

Testing

1. Test locally with new Dockerfile
2. Test with Tilt in local Kubernetes
3. Verify all functionality works (file upload, IPFS pinning, P2P sync)
4. Load testing to verify resource limits are appropriate

Rollout

1. Update documentation first
2. Merge Dockerfile changes
3. Update Kubernetes manifests
4. Update Tiltfile
5. Notify users of new architecture

Acceptance Criteria

• PinShare container no longer contains IPFS binaries
• IPFS runs in a separate sidecar container
• Both containers have independent health checks
• Both containers have independent resource limits
• Permission errors are resolved
• Tilt hot reload works for PinShare code changes
• All existing functionality works (file upload, IPFS pinning, metadata sync)
• Logs are cleanly separated
• Documentation updated with new architecture
• k8s/README.md includes sidecar pattern explanation

Related Issues

• Kubernetes deployment support (depends on #TBD)
• Permission issues with IPFS persistent volumes

References

• Kubernetes Sidecar Pattern
• Best Practices for Multi-Container Pods
• IPFS Kubo Docker Documentation

[bryanchriswhite]

Updated Recommendation: Separate Deployments (Not Sidecar)

After further consideration, separate deployments is a better architecture than the sidecar pattern for PinShare's use case.

Why Separate Deployments?

1. IPFS Reusability

• Multiple PinShare instances can share a single IPFS node
• Other dApps/frontends can use the same IPFS backend
• Better resource utilization - one IPFS node instead of N sidecars

2. Independent Scaling

• Scale PinShare replicas independently from IPFS
• Add more PinShare instances without adding IPFS overhead
• Horizontal pod autoscaling based on API load

3. Operational Flexibility

• Upgrade IPFS without touching PinShare
• Upgrade PinShare without touching IPFS
• Different update cadences for each component

4. Cost Efficiency

• IPFS is resource-heavy (storage, CPU for pinning/GC)
• Running one IPFS node vs N sidecars saves significant resources
• Better for multi-tenant scenarios

Proposed Architecture

┌─────────────────────────────────────────┐
│         IPFS StatefulSet                │
│  ┌────────────────────────────────────┐ │
│  │  ipfs-0 (Persistent Storage)       │ │
│  │  - API: 5001                       │ │
│  │  - Gateway: 8080                   │ │
│  │  - Swarm: 4001                     │ │
│  └────────────────────────────────────┘ │
└─────────────────────────────────────────┘
                    ↑
                    │ Service: ipfs.pinshare.svc.cluster.local:5001
                    │
    ┌───────────────┼───────────────┐
    │               │               │
┌───▼───────┐  ┌───▼───────┐  ┌───▼───────┐
│ PinShare  │  │ PinShare  │  │ Other     │
│ Pod 1     │  │ Pod 2     │  │ dApp      │
│ (API)     │  │ (API)     │  │           │
└───────────┘  └───────────┘  └───────────┘

Implementation

IPFS StatefulSet

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ipfs
  namespace: pinshare
spec:
  serviceName: ipfs
  replicas: 1  # Start with 1, can add more for HA later
  selector:
    matchLabels:
      app: ipfs
  template:
    metadata:
      labels:
        app: ipfs
    spec:
      containers:
      - name: ipfs
        image: ipfs/kubo:latest
        ports:
        - name: api
          containerPort: 5001
        - name: gateway
          containerPort: 8080
        - name: swarm-tcp
          containerPort: 4001
        - name: swarm-udp
          containerPort: 4001
          protocol: UDP
        volumeMounts:
        - name: ipfs-data
          mountPath: /data/ipfs
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "2Gi"
            cpu: "1000m"
  volumeClaimTemplates:
  - metadata:
      name: ipfs-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 50Gi
---
apiVersion: v1
kind: Service
metadata:
  name: ipfs
  namespace: pinshare
spec:
  clusterIP: None  # Headless service for StatefulSet
  selector:
    app: ipfs
  ports:
  - name: api
    port: 5001
  - name: gateway
    port: 8080
  - name: swarm-tcp
    port: 4001
  - name: swarm-udp
    port: 4001
    protocol: UDP

PinShare Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pinshare
  namespace: pinshare
spec:
  replicas: 2  # Can scale up!
  selector:
    matchLabels:
      app: pinshare
  template:
    metadata:
      labels:
        app: pinshare
    spec:
      containers:
      - name: pinshare
        image: pinshare:latest
        env:
        - name: IPFS_API_URL
          value: "http://ipfs-0.ipfs.pinshare.svc.cluster.local:5001"
        ports:
        - name: api
          containerPort: 9090
        - name: libp2p
          containerPort: 50001
        volumeMounts:
        - name: backend-data
          mountPath: /opt/pinshare/data
        resources:
          requests:
            memory: "256Mi"
            cpu: "100m"
          limits:
            memory: "1Gi"
            cpu: "500m"
      volumes:
      - name: backend-data
        persistentVolumeClaim:
          claimName: pinshare-backend-data

Benefits Over Sidecar

Aspect	Sidecar Pattern	Separate Deployments
IPFS Reuse	❌ One per PinShare pod	✅ Shared by all
Scaling	⚠️ Coupled (1:1)	✅ Independent
Resource Usage	❌ N×IPFS overhead	✅ Single IPFS node
Multi-tenant	❌ Not suitable	✅ Perfect
Complexity	✅ Simpler networking	⚠️ Service discovery
HA/Failover	⚠️ Limited	✅ StatefulSet features

Migration Path

1. Phase 1: Deploy IPFS StatefulSet
2. Phase 2: Update PinShare to use IPFS service via env var
3. Phase 3: Update Dockerfile to remove IPFS binaries
4. Phase 4: Update Tiltfile for separate services
5. Phase 5: Optional - Add IPFS cluster for HA

Future Enhancements

With separate deployments, you can:

• Add IPFS Cluster for high availability
• Run multiple IPFS nodes behind a load balancer
• Share IPFS with other applications (UI, other dApps)
• Use external IPFS services (Infura, Pinata) by just changing the URL
• Implement IPFS node pools for different workloads

Configuration

PinShare needs one environment variable:

IPFS_API_URL=http://ipfs-0.ipfs.pinshare.svc.cluster.local:5001

This makes PinShare portable - can work with:

• Local IPFS in Kubernetes
• External IPFS services
• IPFS Cluster
• Multiple IPFS nodes (load balanced)

---

Updated Recommendation: Let's implement separate deployments instead of the sidecar pattern. This aligns better with the goal of reusing IPFS across multiple frontends/dapps.