From 5bbc3f7e5d868c8d5788f4360d7b4a4f6115a745 Mon Sep 17 00:00:00 2001 From: Steven Lindsay Date: Wed, 7 Jan 2026 14:12:14 +0000 Subject: [PATCH 01/16] add health tech guide --- .../guides/pub-sub/health-tech-dashboard.mdx | 1092 +++++++++++++++++ 1 file changed, 1092 insertions(+) create mode 100644 src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx diff --git a/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx b/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx new file mode 100644 index 0000000000..f90fd49d41 --- /dev/null +++ b/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx @@ -0,0 +1,1092 @@ +--- +title: "Guide: Building real-time patient monitoring dashboards with Ably" +meta_description: "Build HIPAA-compliant patient monitoring dashboards at scale with Ably: continuous telemetry streaming, delta compression, multi-device sync, and exactly-once delivery for life-critical applications." +meta_keywords: "patient monitoring, healthcare dashboards, medical telemetry, vital signs streaming, HIPAA compliance, delta compression, healthcare IoT, real-time monitoring, ICU dashboards, medical device integration" +--- + +In an ICU, a patient's heart rate spikes from 72 to 140 bpm. The bedside monitor, nurse station display, and doctor's mobile app must all show this change instantly and identically. Any delay, data loss, or desynchronization could have life-threatening consequences. + +Traditional approaches fall short: polling-based dashboards deliver stale data, basic WebSocket implementations lack ordering guarantees and fail during network disruptions, and proprietary hospital systems create vendor lock-in with poor integration capabilities. Healthcare demands better. + +Ably is purpose-built for life-critical realtime applications. It provides continuous telemetry streaming without delay, HIPAA-compliant infrastructure with encryption and audit trails, exactly-once delivery to prevent duplicate vital readings, guaranteed message ordering for temporal accuracy, 99.999% uptime with automatic failover, and a global edge network ensuring low latency regardless of hospital location. + +## Common health tech dashboard applications + +While this guide focuses on patient monitoring, Ably's capabilities apply across healthcare realtime applications: + +- **ICU/Critical Care Monitoring:** Continuous multi-parameter vital signs with high-frequency waveforms streamed to multiple displays simultaneously. +- **Remote Patient Monitoring (RPM):** Home health devices streaming to care team dashboards, enabling virtual care at scale. +- **Operating Room Displays:** Multi-device synchronization ensures surgical teams see identical patient state across monitors. +- **Emergency Department Triage:** Realtime bed status and patient acuity tracking across emergency departments. +- **Neonatal Monitoring:** Ultra-sensitive infant vital signs with immediate alerting for critical changes. +- **Telemetry Units:** Wireless cardiac monitoring streamed across hospital floors to centralized monitoring stations. + +## Why Ably for healthcare dashboards? + +Ably is engineered around the four pillars of dependability, proven at scale by delivering over 500 billion messages per month: + +* **[Performance](/docs/platform/architecture/performance):** Sub-50ms latency globally ensures time-sensitive medical data arrives instantly. +* **[Integrity](/docs/platform/architecture/message-ordering):** Guaranteed message ordering and exactly-once delivery prevent duplicate vital readings and ensure temporal accuracy. +* **[Reliability](/docs/platform/architecture/fault-tolerance):** 99.999% uptime SLA (maximum 5.26 minutes downtime per year) with automatic failover undetectable to clients. +* **[Availability](/docs/platform/architecture/edge-network):** Global edge infrastructure provides low latency from any hospital location worldwide. + +### Healthcare-specific capabilities + +**HIPAA Compliance:** +- Transport encryption via TLS 1.2+ automatically enforced on all connections +- Message history encrypted at rest with AES-256 and regular key rotation +- Audit logging captures all channel access for compliance reporting +- Business Associate Agreement (BAA) available for enterprise customers +- No Protected Health Information (PHI) stored in message routing metadata + +**Data Integrity for Medical Applications:** +- Exactly-once delivery prevents duplicate vital sign readings that could mislead clinical decisions +- Guaranteed message ordering ensures temporal accuracy critical for medical data interpretation +- Automatic connection recovery maintains message continuity during network disruptions +- [Delta compression](/docs/channels/options/deltas) reduces bandwidth while preserving complete state history + +**Life-Critical Reliability:** +- Automatic multi-region failover occurs transparently without client awareness +- No single point of failure across Ably's distributed infrastructure +- Graceful degradation under load maintains service during traffic spikes +- Proven scalability supporting millions of concurrent connections per channel + +Ably's [serverless architecture](/docs/platform/architecture) eliminates infrastructure management, automatically scaling to handle any load without provisioning or maintenance. + +## Architecting your patient monitoring dashboard + +Effective dashboard architecture begins with channel design, data modeling, and optimization strategies tailored to healthcare requirements. + +### Channel architecture patterns + +**Recommended: Patient-centric channels** + +Structure channels around individual patients for natural access control and scalability: + +``` +Channel naming: patient:{patient_id}:vitals + +Benefits: +- Natural access control (subscribe only to authorized patients) +- Isolated streams (one patient's data spike doesn't affect others) +- Horizontal scalability via consistent hashing distributes load +- HIPAA-friendly (patient ID can be de-identified/hashed) +``` + +**Alternative: Device-centric channels** + +Use when one device monitors multiple parameters across patients: + +``` +Channel naming: device:{device_id}:telemetry + +Trade-offs: +- Requires client-side filtering for specific patients +- More complex access control implementation +- Useful for device management and diagnostics +``` + +**Multi-device synchronization** + +Multiple displays monitoring the same patient (bedside monitor, nurse station, mobile app, family display) all subscribe to the same channel. Ably automatically handles message fanout, guaranteeing identical state across all devices. Connection recovery ensures no device falls behind during network disruptions. + +### Publisher implementation + +Medical device gateways typically run on hospital infrastructure, collecting data from monitoring equipment and publishing to Ably: + + +```javascript +// Medical device gateway (Node.js) +const Ably = require('ably'); +const realtime = new Ably.Realtime({ key: process.env.ABLY_API_KEY }); + +const patientChannel = realtime.channels.get('patient:7f3a9b2e:vitals'); + +// Publish vital signs update +function publishVitals(vitals) { + patientChannel.publish('vitals-update', { + timestamp: Date.now(), + patientId: '7f3a9b2e', // De-identified hash + heartRate: vitals.heartRate, + bloodPressure: { + systolic: vitals.bloodPressure.systolic, + diastolic: vitals.bloodPressure.diastolic + }, + spO2: vitals.spO2, + temperature: vitals.temperature, + respiratoryRate: vitals.respiratoryRate + }); +} + +// Collect vitals from medical device at 1Hz +setInterval(() => { + const vitals = collectFromDevice(); + publishVitals(vitals); +}, 1000); +``` + +```go +// Medical device gateway (Go) +package main + +import ( + "context" + "time" + "github.com/ably/ably-go/ably" +) + +func main() { + client, _ := ably.NewRealtime(ably.WithKey(apiKey)) + channel := client.Channels.Get("patient:7f3a9b2e:vitals") + + ticker := time.NewTicker(1 * time.Second) + defer ticker.Stop() + + for range ticker.C { + vitals := collectFromDevice() + + vitalSigns := map[string]interface{}{ + "timestamp": time.Now().UnixMilli(), + "patientId": "7f3a9b2e", + "heartRate": vitals.HeartRate, + "bloodPressure": map[string]int{ + "systolic": vitals.BPSystolic, + "diastolic": vitals.BPDiastolic, + }, + "spO2": vitals.SpO2, + "temperature": vitals.Temperature, + "respiratoryRate": vitals.RespiratoryRate, + } + + err := channel.Publish(ctx, "vitals-update", vitalSigns) + if err != nil { + // Handle critical error - vitals not transmitted + log.Printf("Failed to publish vital signs: %v", err) + } + } +} +``` + + +### Data modeling for medical telemetry + +Structure vital signs payloads to support delta compression, temporal ordering, and quality monitoring: + +```javascript +{ + "timestamp": 1704563280000, // Unix epoch ms (critical for temporal accuracy) + "patientId": "7f3a9b2e", // De-identified hash + "deviceId": "monitor-icu-3", // Source device tracking + "vitals": { + "heartRate": { + "value": 78, + "unit": "bpm", + "quality": "good" // Signal quality indicator + }, + "bloodPressure": { + "systolic": 120, + "diastolic": 80, + "unit": "mmHg", + "quality": "good" + }, + "spO2": { + "value": 98, + "unit": "%", + "quality": "good" + }, + "temperature": { + "value": 37.2, + "unit": "celsius", + "quality": "good" + }, + "respiratoryRate": { + "value": 16, + "unit": "breaths/min", + "quality": "good" + } + }, + "alarms": [ + { + "type": "high_heart_rate", + "severity": "warning", + "threshold": 100, + "triggered_at": 1704563280000 + } + ], + "metadata": { + "version": "1.0", + "checksum": "abc123" // Data integrity verification + } +} +``` + +**Design principles:** +- Always include timestamps for temporal ordering +- Include signal quality indicators to detect sensor detachment +- Use structured nested objects (delta compression works best with similar structure between messages) +- Include metadata for versioning and integrity verification +- For high-frequency waveforms, use sequence numbers to detect dropped messages + +### High-frequency data and channel limits + +Ably limits inbound messages to 50 msg/s per channel to ensure platform stability. For higher-frequency data sources like ECG waveforms at 250Hz, batch multiple samples into single publishes: + + +```javascript +// Batch high-frequency samples into single publishes +const sampleBuffer = []; +const BATCH_SIZE = 25; // 25 samples per publish = 10 publishes/sec at 250Hz + +function collectSample(sample) { + sampleBuffer.push(sample); + + if (sampleBuffer.length >= BATCH_SIZE) { + channel.publish('waveform-batch', { + timestamp: Date.now(), + patientId: '7f3a9b2e', + waveformType: 'ecg_lead_ii', + sampleRate: 250, + samples: sampleBuffer.splice(0, BATCH_SIZE), + sequenceNumber: currentSequence++ + }); + } +} + +// Collect ECG samples at 250Hz +setInterval(() => { + collectSample(readECGSample()); +}, 4); // 4ms interval = 250Hz +``` + + +**Important:** When publishing batched messages via REST API from ephemeral connections, consider [message ordering](/docs/pub-sub/advanced#message-ordering) and [idempotency](/docs/pub-sub/advanced#idempotency) requirements. Realtime connections provide serial ordering guarantees; REST does not unless you explicitly set message IDs. + +### Optimization: Delta compression for vital signs + +Vital signs messages often grow over time as cumulative patient data accumulates. Repeatedly sending 3-5KB payloads every second wastes bandwidth, especially for mobile and tablet displays. + +[Delta compression](/docs/channels/options/deltas) solves this by transmitting only changes between successive messages. Publishers continue sending complete state (maintaining simplicity), while Ably computes deltas server-side and transmits only changes. Subscriber SDKs automatically reconstruct full state from deltas. + +**When to use delta compression:** +- ✅ Vital signs with incremental changes (most parameters remain stable between updates) +- ✅ Multi-parameter monitoring where only 1-2 vitals change per update +- ✅ Mobile or bandwidth-constrained displays +- ✅ Payloads > 1KB with high similarity between successive messages + +**When NOT to use delta compression:** +- ❌ High-frequency waveforms where most data changes between messages +- ❌ Small alert messages (< 1KB where overhead exceeds savings) +- ❌ Scenarios requiring channel-level encryption (deltas are incompatible; use TLS transport encryption instead) + +**Implementation:** + + +```javascript +// Dashboard: Subscribe with delta compression enabled +const vcdiffPlugin = require('@ably/vcdiff-decoder'); + +const realtime = new Ably.Realtime({ + key: 'your-api-key', + plugins: { vcdiff: vcdiffPlugin } +}); + +const channel = realtime.channels.get('patient:7f3a9b2e:vitals', { + params: { delta: 'vcdiff' } +}); + +channel.subscribe('vitals-update', (msg) => { + // SDK automatically reconstructs full state from delta + // New joiners receive full state on first message + updateVitalsDisplay(msg.data); +}); +``` + + +See the [delta compression documentation](/docs/channels/options/deltas#subscribe) for plugin installation details and browser compatibility. + +**Bandwidth savings example:** +- Full payload: ~3KB (comprehensive vital signs with history) +- Delta payload: ~300-500 bytes (typical 80-85% compression) +- 1Hz updates × 10 subscribed devices = 30KB/s → 3-5KB/s +- **Result: 80-85% bandwidth reduction** + +**Key considerations:** +- First message after connection is always full state +- Connection recovery sends full state, then resumes delta mode +- Monitor compression ratios via [application statistics](/docs/metadata-stats/stats) +- For cost optimization, use the [persist last message](/docs/storage-history/storage#persist-last-message) rule to store only the final complete state rather than every delta + +## Ensuring data freshness with message conflation + +Medical devices often publish vital signs faster than dashboards can render them. A device publishing at 10Hz (10 updates/second) overwhelms a dashboard UI that refreshes at 1-2 FPS due to complex chart rendering. Without optimization, 8-9 stale updates accumulate per second in the browser queue. + +### The staleness problem + +This creates three issues: +- Clinicians see delayed vital signs (dangerous in critical care) +- Dashboard becomes unresponsive (unusable during emergencies) +- Bandwidth wasted on data that's never displayed + +### Solution: Message conflation + +[Message conflation](/docs/messages#conflation) aggregates messages on the server over a configured time window (e.g., 100ms), discards intermediate values, and delivers only the most current state to subscribers. + +Configure conflation through [channel rules](/docs/channels#rules) in your Ably dashboard: + +1. Navigate to app settings → Channel Rules +2. Create rule for channel pattern: `patient:*:vitals` +3. Enable conflation with interval (e.g., 100ms) +4. Set conflation key pattern: `#{message.name}` + +**When to use conflation:** +- ✅ Display refresh rate < data publish rate +- ✅ Only current value matters (e.g., current heart rate, not historical sequence) +- ✅ Preventing UI performance degradation +- ✅ Cost-sensitive deployments where intermediate values aren't critical + +**When NOT to use conflation:** +- ❌ Every update must be processed (e.g., waveform displays) +- ❌ Audit trail or compliance requirements (conflation permanently discards intermediate messages) +- ❌ Alert messages (small, infrequent, every alert matters) + +**Critical warning:** + + + +**Performance example:** +- Device publishes at 10Hz = 10 msg/sec +- With 100ms conflation = 10 msg/sec → ~10 msg/sec delivered (one per 100ms window) +- For 10 subscribed dashboards: 100 msg/sec → 100 msg/sec outbound +- **Result: Prevents dashboard overload while maintaining current state visibility** + +## Multi-device synchronization and connection recovery + +A single ICU patient may be monitored simultaneously by a bedside display, nurse station dashboard showing 10 patients, doctor's mobile app, and family viewing screen. All must show identical, current vital signs. Any desynchronization could lead to clinical errors. + +### Ably's synchronization guarantee + +All devices subscribe to the same channel (`patient:7f3a9b2e:vitals`). Ably guarantees: +- **Message ordering** ensures all devices see updates in the same sequence +- **Exactly-once delivery** prevents any device from receiving duplicate readings +- **Automatic fanout** to all subscribers without application logic +- **Connection-independent state** allows devices to join/leave independently + + +```javascript +// Bedside monitor +const channel = realtime.channels.get('patient:7f3a9b2e:vitals'); +channel.subscribe('vitals-update', (msg) => { + updateBedsideMonitor(msg.data); +}); + +// Nurse station (same subscription, different UI) +channel.subscribe('vitals-update', (msg) => { + updatePatientTile('7f3a9b2e', msg.data); +}); + +// Mobile app (same subscription, optimized for mobile) +channel.subscribe('vitals-update', (msg) => { + updateMobileVitals(msg.data); +}); +``` + + +### Handling network disruptions + +Network disruptions occur frequently in hospitals: Wi-Fi dead zones in elevators and stairwells, cellular connectivity drops for mobile clinicians, network maintenance windows, and infrastructure failures. Dashboards must automatically reconnect and recover missed data without clinician intervention. + +**Ably's automatic recovery:** + + +```javascript +realtime.connection.on('disconnected', () => { + // Ably SDK automatically attempts reconnection + showReconnectingIndicator(); +}); + +realtime.connection.on('connected', () => { + // Connection restored, channel state recovered automatically + hideReconnectingIndicator(); + // Ably SDK has already backfilled missed messages +}); +``` + + +The SDK tracks the last received message ID. On reconnection, it requests missed messages from Ably and seamlessly backfills to current state. No application logic required. + +If the primary region becomes unreachable, the SDK automatically tries fallback regions until connection is restored. This multi-region failover is transparent to your application. + +**History backfill for extended disconnections:** + +Ably stores message history for 2 minutes by default (sufficient for connection recovery and providing context when joining mid-stream). This can be extended up to 1 year if needed. For disconnections exceeding the history retention period, explicitly request history: + + +```javascript +channel.subscribe('vitals-update', (msg) => { + updateDisplay(msg.data); +}); + +// After extended disconnection, backfill missed data +channel.history({ limit: 100 }, (err, resultPage) => { + if (err) { + console.error('Failed to retrieve history:', err); + return; + } + + // Display gap-fill indicator to user + resultPage.items.forEach((msg) => { + updateDisplayWithHistoricalData(msg.data); + }); +}); +``` + + +**Best practices:** +- Display connection state indicator in UI (connected/reconnecting/disconnected) +- Use history API for disconnections exceeding retention period +- Test recovery scenarios in realistic hospital network conditions +- Configure [connection timeout](/docs/connect/states#connection-state-recovery) based on application criticality + +See the [fault tolerance documentation](/docs/platform/architecture/fault-tolerance) for details on Ably's infrastructure resilience. + +## HIPAA compliance and security + +Healthcare applications handling Protected Health Information (PHI) must comply with HIPAA's Security Rule. Ably provides the infrastructure and capabilities needed to build HIPAA-compliant applications. + +### HIPAA requirements overview + +- **Encryption:** Data encrypted at rest and in transit (§164.312(a)(2)(iv)) +- **Access Control:** Only authorized users access patient data (§164.312(a)(1)) +- **Audit Trails:** All data access must be logged (§164.312(b)) +- **Data Integrity:** Prevent unauthorized modification (§164.312(c)(1)) +- **Business Associate Agreement (BAA):** Required for vendors handling PHI + +### How Ably supports HIPAA compliance + +**Encryption:** +- Transport encryption via TLS 1.2+ automatically enforced on all connections +- Perfect Forward Secrecy (PFS) enabled +- Message history encrypted at rest with AES-256 +- Encryption keys rotated regularly +- Certificate pinning available for mobile apps + +**Access Control:** + +Implement token-based authentication with capability constraints: + + +```javascript +// Generate token on secure backend (never in client code) +const Ably = require('ably'); +const rest = new Ably.Rest({ key: process.env.ABLY_API_KEY }); + +async function generatePatientToken(nurseId, assignedPatientIds) { + const capability = {}; + + // Grant access only to assigned patients + assignedPatientIds.forEach(patientId => { + capability[`patient:${patientId}:vitals`] = ['subscribe']; + capability[`patient:${patientId}:alerts`] = ['subscribe', 'publish']; // Can acknowledge alerts + }); + + const tokenRequest = await rest.auth.createTokenRequest({ + clientId: nurseId, + capability: capability, + ttl: 3600000 // 1 hour expiration (short-lived tokens) + }); + + return tokenRequest; +} + +// Client uses token +const realtime = new Ably.Realtime({ + authCallback: async (tokenParams, callback) => { + // Fetch token from your secure backend + const tokenRequest = await fetch('/api/ably-token').then(r => r.json()); + callback(null, tokenRequest); + } +}); +``` + + +**Audit Logging:** + +Ably provides audit-ready logs via integrations. Export logs to your SIEM: +- Who accessed which channel (client ID) +- When access occurred (timestamp) +- What actions were performed (publish/subscribe) +- Connection source (IP address, user agent) + +Common integration patterns: +- Kafka integration → Elasticsearch → Kibana +- AWS Lambda → CloudWatch Logs +- Azure Functions → Azure Monitor + +See [Ably integrations](/docs/platform/integrations) for configuration details. + +**Data Integrity:** + +Ably guarantees: +- Message ordering (temporal accuracy for medical data) +- Exactly-once delivery (no duplicate vital readings) +- Checksum verification (detect data corruption) +- Immutable message history (tamper-proof audit trail) + +**Business Associate Agreement (BAA):** + +Ably provides HIPAA BAA to enterprise customers, covering PHI transmission through Ably's infrastructure and committing to HIPAA security and privacy requirements. Contact [Ably sales](https://ably.com/contact) for BAA execution. + +### De-identification best practices + +Use de-identified patient IDs in channel names and message payloads to reduce PHI exposure: + + +```javascript +// ❌ Don't use PHI directly +const channelName = 'patient:John-Doe-DOB-1985-03-15:vitals'; + +// ✅ Use de-identified hash +const crypto = require('crypto'); +const patientHash = crypto + .createHash('sha256') + .update('internal-patient-id-12345') + .digest('hex') + .substring(0, 12); + +const channelName = `patient:${patientHash}:vitals`; + +// Store mapping in your secure database: +// internal-patient-id-12345 → 7f3a9b2e8c1d (hash) +``` + + +**Benefits:** +- Reduces PHI exposure in Ably infrastructure +- Simplifies compliance audit (less PHI to track) +- Channel names can be safely logged and monitored + +**Important:** De-identification does NOT eliminate HIPAA requirements. You still need BAA, encryption, access controls, and audit trails. + +## External storage and long-term data retention + +Ably's message history (2 minutes by default, extendable to 1 year) enables connection recovery and context on join but is not designed for long-term medical records. Healthcare organizations require years of retention for compliance, complex querying for trend analysis, and data warehouse integration for business intelligence. + +### Architecture pattern: Real-time + archival + +Stream data through Ably for real-time delivery to dashboards while simultaneously archiving to your own storage infrastructure: + +``` +Medical Device → Ably Pub/Sub → [1] Real-time Dashboard (immediate display) + → [2] Ably Integration → Kafka → Time-series DB + → Data Lake (S3) + → Analytics Pipeline +``` + +**Why this pattern:** +- Real-time path: Ably handles immediate delivery to dashboards +- Archival path: Ably integration forwards all messages to your storage +- Decoupled: Storage issues don't affect real-time monitoring +- Compliant: Meet your specific retention requirements without impacting live systems + +**Cost optimization:** If using delta compression, configure the [persist last message](/docs/storage-history/storage#persist-last-message) rule to store only the final complete state rather than every delta. This provides the latest state for reconnections while reducing storage costs, with full historical data captured in your external archive. + +### Implementation: Ably integration to Kafka + +Configure Ably integration via your dashboard: + +1. Navigate to app settings → Integrations +2. Create Kafka integration +3. Configure rule: Trigger on all messages to `patient:*:vitals` +4. Kafka broker: `your-kafka-cluster.hospital.internal:9092` +5. Topic: `patient-vitals-archive` +6. Format: JSON (or Confluent Avro for schema registry) + +**Kafka consumer to time-series database:** + + +```go +// Kafka consumer to persist vitals to time-series database +package main + +import ( + "context" + "encoding/json" + "log" + "time" + "github.com/segmentio/kafka-go" +) + +type VitalsMessage struct { + Timestamp int64 `json:"timestamp"` + PatientID string `json:"patientId"` + DeviceID string `json:"deviceId"` + Vitals struct { + HeartRate struct { + Value int `json:"value"` + Quality string `json:"quality"` + } `json:"heartRate"` + BloodPressure struct { + Systolic int `json:"systolic"` + Diastolic int `json:"diastolic"` + Quality string `json:"quality"` + } `json:"bloodPressure"` + SpO2 struct { + Value int `json:"value"` + Quality string `json:"quality"` + } `json:"spO2"` + Temperature struct { + Value float64 `json:"value"` + Quality string `json:"quality"` + } `json:"temperature"` + RespiratoryRate struct { + Value int `json:"value"` + Quality string `json:"quality"` + } `json:"respiratoryRate"` + } `json:"vitals"` +} + +func consumeVitals(db *sql.DB) { + reader := kafka.NewReader(kafka.ReaderConfig{ + Brokers: []string{"kafka.hospital.internal:9092"}, + Topic: "patient-vitals-archive", + GroupID: "vitals-archiver", + }) + defer reader.Close() + + for { + msg, err := reader.ReadMessage(context.Background()) + if err != nil { + log.Println("Error reading from Kafka:", err) + continue + } + + var vitals VitalsMessage + if err := json.Unmarshal(msg.Value, &vitals); err != nil { + log.Println("Error parsing vitals:", err) + continue + } + + storeVitals(db, vitals) + } +} + +func storeVitals(db *sql.DB, vitals VitalsMessage) { + // Insert into TimescaleDB (PostgreSQL-based time-series DB) + query := ` + INSERT INTO patient_vitals + (patient_id, timestamp, heart_rate, blood_pressure_systolic, + blood_pressure_diastolic, spo2, temperature, respiratory_rate, + device_id, heart_rate_quality) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) + ` + + _, err := db.Exec( + query, + vitals.PatientID, + time.UnixMilli(vitals.Timestamp), + vitals.Vitals.HeartRate.Value, + vitals.Vitals.BloodPressure.Systolic, + vitals.Vitals.BloodPressure.Diastolic, + vitals.Vitals.SpO2.Value, + vitals.Vitals.Temperature.Value, + vitals.Vitals.RespiratoryRate.Value, + vitals.DeviceID, + vitals.Vitals.HeartRate.Quality, + ) + + if err != nil { + log.Println("Error storing vitals:", err) + // Consider dead letter queue for failed writes + } +} +``` + + +**Key decisions:** +- Storage technology: Time-series DB (TimescaleDB, InfluxDB) vs Data Lake (S3 + Parquet) +- Retention policy: Automated deletion based on your legal/business requirements +- Schema management: Avro or Protobuf for forward compatibility +- Failure handling: Dead letter queue for failed writes + +Your retention period should be determined by your specific legal and business requirements, which vary by jurisdiction and use case. + +## Alerting and threshold monitoring + +Real-time clinical alerts ensure care teams respond immediately to critical changes in patient condition. + +### Alert architecture patterns + +**Option 1: Client-side alerting (simple, low latency)** + +Each dashboard implements threshold checks locally: + + +```javascript +channel.subscribe('vitals-update', (msg) => { + const vitals = msg.data; + + if (vitals.heartRate.value > 120) { + triggerAlert('HIGH_HEART_RATE', vitals); + } + + if (vitals.spO2.value < 90) { + triggerAlert('LOW_SPO2', vitals); + } + + updateDisplay(vitals); +}); +``` + + +**Option 2: Server-side alerting (centralized, auditable)** + +A dedicated alert service monitors thresholds and publishes alerts to a separate channel: + +``` +Medical Device → Ably → Alert Service (monitors thresholds) + ↓ + Alerts Channel → All Dashboards +``` + +**Recommendation:** Use server-side alerting for: +- Centralized alert logic (single source of truth) +- Alert deduplication (prevent alert fatigue) +- Audit trail (regulatory requirement) +- Multi-user coordination (alert acknowledgment tracking) + +### Implementation: Alert service with acknowledgment + +Medical alerts often require acknowledgment to ensure clinical staff have seen critical notifications. Ably's [message annotations](/docs/messages/annotations) feature provides a reliable mechanism for alert acknowledgment. + +**How it works:** +1. Alert service publishes alert messages to `patient:{id}:alerts` channel +2. Clinicians receive alerts on their dashboards +3. When acknowledged, dashboard publishes a `flag.v1` annotation with `name: 'acknowledged'` +4. Alert service monitors annotation summaries to track acknowledgment status +5. If not acknowledged within threshold, escalate to supervisor + +**Dashboard implementation:** + + +```javascript +// Dashboard: Subscribe to alerts and handle acknowledgment +const alertChannel = realtime.channels.get('patient:7f3a9b2e:alerts', { + modes: ['MESSAGE_SUBSCRIBE', 'ANNOTATION_PUBLISH'] +}); + +alertChannel.subscribe((message) => { + if (message.action === 'message.create') { + const alert = message.data; + + // Display alert banner with audio notification + displayAlertBanner(alert); + playAlertSound(alert.severity); + + // Clinician clicks "Acknowledge" button + document.getElementById('ack-btn').onclick = async () => { + await alertChannel.annotations.publish(message, { + type: 'acknowledgment:flag.v1', + name: 'acknowledged' + }); + hideAlertBanner(); + }; + } + + // Track annotation summaries to show who acknowledged + if (message.action === 'message.summary') { + const ackSummary = message.annotations?.summary?.['acknowledgment:flag.v1']; + if (ackSummary) { + console.log(`Alert acknowledged by: ${ackSummary.clientIds.join(', ')}`); + } + } +}); +``` + + +**Alert service implementation:** + + +```go +// Alert service monitoring thresholds and acknowledgments +package main + +import ( + "context" + "log" + "time" + "github.com/ably/ably-go/ably" +) + +type AlertService struct { + realtime *ably.Realtime + thresholds map[string]VitalThresholds +} + +type VitalThresholds struct { + HeartRateMax int + HeartRateMin int + SpO2Min int +} + +type Alert struct { + Type string `json:"type"` + Severity string `json:"severity"` + PatientID string `json:"patientId"` + Value int `json:"value"` + Threshold int `json:"threshold"` + Timestamp int64 `json:"timestamp"` +} + +func (s *AlertService) MonitorPatient(ctx context.Context, patientID string) { + vitalsChannel := s.realtime.Channels.Get("patient:" + patientID + ":vitals") + alertChannel := s.realtime.Channels.Get("patient:" + patientID + ":alerts") + + // Subscribe to vitals and check thresholds + _, err := vitalsChannel.Subscribe(ctx, "vitals-update", func(msg *ably.Message) { + vitals := msg.Data.(map[string]interface{}) + + // Check heart rate threshold + if hr, ok := getHeartRate(vitals); ok { + if hr > s.thresholds[patientID].HeartRateMax { + s.publishAlert(ctx, alertChannel, Alert{ + Type: "HIGH_HEART_RATE", + Severity: "WARNING", + PatientID: patientID, + Value: hr, + Threshold: s.thresholds[patientID].HeartRateMax, + Timestamp: time.Now().UnixMilli(), + }) + } + } + + // Check SpO2 threshold + if spO2, ok := getSpO2(vitals); ok { + if spO2 < s.thresholds[patientID].SpO2Min { + s.publishAlert(ctx, alertChannel, Alert{ + Type: "LOW_SPO2", + Severity: "CRITICAL", // SpO2 < 90% is critical + PatientID: patientID, + Value: spO2, + Threshold: s.thresholds[patientID].SpO2Min, + Timestamp: time.Now().UnixMilli(), + }) + } + } + }) + + if err != nil { + log.Fatal("Failed to subscribe to vitals:", err) + } + + // Monitor alert acknowledgments via annotation summaries + _, err = alertChannel.Subscribe(ctx, func(msg *ably.Message) { + if msg.Action == ably.MessageActionSummary { + ackSummary := msg.Annotations.Summary["acknowledgment:flag.v1"] + if ackSummary != nil { + log.Printf("Alert %s acknowledged by %v", msg.Serial, ackSummary.ClientIDs) + // Cancel escalation timer for this alert + cancelEscalation(msg.Serial) + } + } + }) + + if err != nil { + log.Fatal("Failed to subscribe to alert acknowledgments:", err) + } +} + +func (s *AlertService) publishAlert(ctx context.Context, channel *ably.RealtimeChannel, alert Alert) { + err := channel.Publish(ctx, "clinical-alert", alert) + if err != nil { + log.Println("Failed to publish alert:", err) + // Critical: alerts must be delivered. Consider fallback mechanism + } +} +``` + + +**Best practices:** +- Use separate alert channel (don't mix with vitals) +- Include severity levels (INFO, WARNING, CRITICAL) +- Implement alert acknowledgment via annotations (`flag.v1` type) +- Debounce alerts (prevent flooding during sustained abnormal readings) +- Escalation logic (if not acknowledged within threshold, page supervisor) +- Monitor annotation summaries to track who acknowledged which alerts + +See the [message annotations documentation](/docs/messages/annotations) for complete details. + +## Cost optimization strategies + +Understanding costs and implementing optimization techniques ensures sustainable economics as you scale. + +### Healthcare dashboard cost profile + +**Ably pricing components:** +- Messages: Inbound (from devices) + Outbound (to dashboards) +- Connections: Each device and dashboard connection +- Channels: Active channels with subscribers +- Storage: Message history retention + +**Single ICU patient scenario:** +- 1 medical device publishing vitals at 1Hz = 86,400 inbound messages/day +- 4 subscribed dashboards = 345,600 outbound messages/day +- 5 concurrent connections = negligible cost +- 1 active channel = negligible cost + +**Daily cost (standard rates):** ~$0.86/patient/day or ~$26/patient/month + +**Cost scales with:** +- Number of monitored patients +- Number of subscribed displays per patient +- Publishing frequency (1Hz vs 10Hz) +- Payload size (basic vitals vs waveforms) + +### Optimization techniques + +**1. Delta compression (80-85% bandwidth reduction)** + +``` +Before: 3KB payload × 1Hz × 4 displays = 43.2KB/s = ~3.7GB/day +After: 500B payload × 1Hz × 4 displays = 7.2KB/s = ~0.6GB/day +Savings: ~85% reduction in data transfer +``` + +**2. Message conflation (alternative cost optimization)** + +For cost-sensitive deployments where intermediate values aren't critical: + +``` +10 patients, 1Hz updates, 10 displays each: +Without conflation: 10 × 86,400 inbound + 10 × 864,000 outbound = ~$260/month +With 500ms conflation (2Hz effective): ~$105/month (80% outbound reduction) +``` + +**Trade-off:** Conflation permanently discards intermediate messages. Even external storage integrations receive only conflated data. Use conflation only if losing intermediate vital readings is acceptable. For audit compliance, use delta compression instead. + +**Cost comparison (10 patients, 30 days):** +- No optimization: ~$260/month +- With deltas (85% bandwidth savings): ~$120/month (preserves all data) +- With conflation (80% outbound reduction): ~$105/month (discards intermediate data) + +**3. Connection management** + + +```javascript +// Close connections when dashboard not in use +window.addEventListener('beforeunload', () => { + realtime.close(); // Graceful disconnect +}); + +// For mobile apps, disconnect in background +document.addEventListener('visibilitychange', () => { + if (document.hidden) { + realtime.connection.close(); + } else { + realtime.connection.connect(); + } +}); +``` + + +**4. Selective subscriptions** + +Nurse station dashboards showing multiple patients should subscribe only to visible patients: + + +```javascript +// Subscribe only to currently visible patients +function updateVisiblePatients(visiblePatientIds) { + // Unsubscribe from off-screen patients + currentSubscriptions.forEach((sub, patientId) => { + if (!visiblePatientIds.includes(patientId)) { + sub.channel.detach(); + } + }); + + // Subscribe to newly visible patients + visiblePatientIds.forEach((patientId) => { + if (!currentSubscriptions.has(patientId)) { + subscribeToPatient(patientId); + } + }); +} +``` + + +**5. REST API for infrequent publishing** + +If a device publishes less than once per minute, use REST API instead of maintaining persistent connection: + + +```javascript +// One-off publish (no persistent connection) +const rest = new Ably.Rest({ key: process.env.ABLY_API_KEY }); +rest.channels.get('patient:7f3a9b2e:vitals').publish('vitals-update', data); +``` + + +**Cost optimization checklist:** +- [ ] Enable delta compression on vital signs channels (preserves all data) +- [ ] Consider conflation only if intermediate values aren't needed (discards data) +- [ ] Close connections when displays inactive +- [ ] Subscribe only to visible patients (nurse stations) +- [ ] Use REST for infrequent device updates (less than 1/min) +- [ ] Batch high-frequency samples to respect 50 msg/s channel limit +- [ ] Monitor costs via Ably dashboard and validate compression ratios + +## Production readiness checklist + +Before deploying to production: + +**Security & Compliance:** +- [ ] BAA executed with Ably +- [ ] Token-based authentication implemented (no API keys in clients) +- [ ] Capability-based access control configured +- [ ] Audit logging enabled and exporting to SIEM +- [ ] De-identified patient IDs used in channel names +- [ ] Encryption verified (TLS 1.2+) + +**Reliability & Performance:** +- [ ] Connection recovery tested (simulate network failures) +- [ ] Multi-device synchronization verified +- [ ] Load tested at expected scale (update frequency × patients × displays) +- [ ] Latency measured (p50, p95, p99) +- [ ] Alerting service deployed with failover + +**Data Management:** +- [ ] External storage integration configured and tested +- [ ] Retention policies defined and enforced +- [ ] History API tested for backfill scenarios +- [ ] Data integrity checks implemented (checksums) + +**Cost Management:** +- [ ] Delta compression enabled and compression ratio validated +- [ ] Connection management logic implemented +- [ ] Costs monitored in Ably dashboard +- [ ] Budget alerts configured + +**Operational:** +- [ ] Connection state indicators in UI +- [ ] Error handling and fallback mechanisms +- [ ] Monitoring and alerting for Ably service health +- [ ] Runbook for common failure scenarios +- [ ] Load testing at 2x expected peak + +## Next steps + +* Read the [Delta Compression documentation](/docs/channels/options/deltas) for complete implementation details +* Read the [Message Conflation documentation](/docs/messages#conflation) for configuration options +* Explore [Ably Integrations](/docs/platform/integrations) for external storage patterns +* Review [Message Annotations](/docs/messages/annotations) for alert acknowledgment implementation +* Learn about [Token Authentication](/docs/auth/token) for production security +* Understand [Channel Options](/docs/channels/options) for optimization features +* Review [Connection State Recovery](/docs/connect/states#connection-state-recovery) for resilience patterns + +**Related Guides:** +- [Data Streaming and Distribution](/docs/guides/pub-sub/data-streaming) + +**Get Help:** +- [Contact Ably Sales](https://ably.com/contact) for BAA and enterprise support +- [Ably Community](https://ably.com/community) for technical questions \ No newline at end of file From f80d483eb570e23d36dba5351c9e61b78b401e23 Mon Sep 17 00:00:00 2001 From: Steven Lindsay Date: Wed, 7 Jan 2026 14:19:10 +0000 Subject: [PATCH 02/16] add health tech guide --- .../guides/pub-sub/health-tech-dashboard.mdx | 68 +++++++++---------- 1 file changed, 34 insertions(+), 34 deletions(-) diff --git a/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx b/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx index f90fd49d41..94a70b2961 100644 --- a/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx +++ b/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx @@ -1,57 +1,57 @@ --- -title: "Guide: Building real-time patient monitoring dashboards with Ably" -meta_description: "Build HIPAA-compliant patient monitoring dashboards at scale with Ably: continuous telemetry streaming, delta compression, multi-device sync, and exactly-once delivery for life-critical applications." -meta_keywords: "patient monitoring, healthcare dashboards, medical telemetry, vital signs streaming, HIPAA compliance, delta compression, healthcare IoT, real-time monitoring, ICU dashboards, medical device integration" +title: "Guide: Building realtime monitoring dashboards with Ably" +meta_description: "Build production-ready realtime monitoring dashboards with Ably: optimize bandwidth with Deltas, ensure data integrity with exactly-once delivery, achieve compliance with encryption and audit logging." +meta_keywords: "realtime dashboard, monitoring, telemetry, delta compression, exactly-once delivery, data streaming, HIPAA compliance, medical monitoring, IoT dashboards" --- -In an ICU, a patient's heart rate spikes from 72 to 140 bpm. The bedside monitor, nurse station display, and doctor's mobile app must all show this change instantly and identically. Any delay, data loss, or desynchronization could have life-threatening consequences. +Ably is purpose-built for realtime high-throughput data streaming at scale. Whether you're monitoring patient vitals, tracking industrial equipment, or visualizing IoT sensor networks, Ably handles the complexity of reliable message distribution so you can focus on your application. -Traditional approaches fall short: polling-based dashboards deliver stale data, basic WebSocket implementations lack ordering guarantees and fail during network disruptions, and proprietary hospital systems create vendor lock-in with poor integration capabilities. Healthcare demands better. +Dashboard streaming follows a simple pattern: one or more data sources publish telemetry to channels, and dashboard applications subscribe to receive realtime updates. Messages published to Ably are referred to as inbound messages, while messages delivered to subscribers are outbound messages. -Ably is purpose-built for life-critical realtime applications. It provides continuous telemetry streaming without delay, HIPAA-compliant infrastructure with encryption and audit trails, exactly-once delivery to prevent duplicate vital readings, guaranteed message ordering for temporal accuracy, 99.999% uptime with automatic failover, and a global edge network ensuring low latency regardless of hospital location. +## Common realtime dashboard applications -## Common health tech dashboard applications +- **Patient monitoring systems:** Streaming vital signs (heart rate, blood pressure, SpO2) from ICU patients to nurse stations and mobile devices, with continuous telemetry requiring guaranteed delivery and ordering. +- **Industrial equipment monitoring:** Real-time tracking of machinery health, temperature sensors, vibration analysis, and predictive maintenance alerts across manufacturing facilities. +- **Fleet and logistics dashboards:** Live vehicle location, fuel consumption, driver behavior, and delivery status updates for transportation and supply chain operations. +- **Energy and utilities monitoring:** Smart grid data, power consumption metrics, renewable energy generation, and infrastructure health across distributed networks. +- **Environmental monitoring:** Air quality sensors, weather stations, water quality monitoring, and seismic activity tracking across geographic regions. -While this guide focuses on patient monitoring, Ably's capabilities apply across healthcare realtime applications: +This guide addresses common challenges in building realtime monitoring dashboards and shows how Ably's features provide solutions that ensure data integrity, reduce costs, and maintain compliance requirements. While examples focus on healthcare monitoring—a domain with the strictest requirements—the patterns apply to any mission-critical dashboard application. -- **ICU/Critical Care Monitoring:** Continuous multi-parameter vital signs with high-frequency waveforms streamed to multiple displays simultaneously. -- **Remote Patient Monitoring (RPM):** Home health devices streaming to care team dashboards, enabling virtual care at scale. -- **Operating Room Displays:** Multi-device synchronization ensures surgical teams see identical patient state across monitors. -- **Emergency Department Triage:** Realtime bed status and patient acuity tracking across emergency departments. -- **Neonatal Monitoring:** Ultra-sensitive infant vital signs with immediate alerting for critical changes. -- **Telemetry Units:** Wireless cardiac monitoring streamed across hospital floors to centralized monitoring stations. +## Why Ably for realtime dashboards? -## Why Ably for healthcare dashboards? +Ably is engineered around the four pillars of dependability: -Ably is engineered around the four pillars of dependability, proven at scale by delivering over 500 billion messages per month: +* **[Performance](/docs/platform/architecture/performance):** Ultra-low latency messaging, even at global scale. +* **[Integrity](/docs/platform/architecture/message-ordering):** Guaranteed message ordering and exactly-once delivery, preventing duplicates and ensuring temporal accuracy. +* **[Reliability](/docs/platform/architecture/fault-tolerance):** 99.999% uptime SLA with automatic failover and seamless reconnection. +* **[Availability](/docs/platform/architecture/edge-network):** Global edge infrastructure ensures users connect to the closest point for optimal experience. +* **[Cost efficiency](#cost-optimization-strategies):** Optimization features like delta compression and message conflation provide significant cost savings while maintaining reliability. -* **[Performance](/docs/platform/architecture/performance):** Sub-50ms latency globally ensures time-sensitive medical data arrives instantly. -* **[Integrity](/docs/platform/architecture/message-ordering):** Guaranteed message ordering and exactly-once delivery prevent duplicate vital readings and ensure temporal accuracy. -* **[Reliability](/docs/platform/architecture/fault-tolerance):** 99.999% uptime SLA (maximum 5.26 minutes downtime per year) with automatic failover undetectable to clients. -* **[Availability](/docs/platform/architecture/edge-network):** Global edge infrastructure provides low latency from any hospital location worldwide. +Ably's [serverless architecture](/docs/platform/architecture) eliminates infrastructure management. It automatically scales to handle millions of concurrent connections without provisioning or maintenance. The platform is proven at scale, delivering over 500 billion messages per month for customers, with individual channels supporting millions of concurrent subscribers while maintaining low latency and high throughput. -### Healthcare-specific capabilities +### Mission-critical capabilities -**HIPAA Compliance:** +**Data Integrity:** +- Exactly-once delivery prevents duplicate readings that could mislead decision-making +- Guaranteed message ordering ensures temporal accuracy for time-sensitive data +- Automatic connection recovery maintains message continuity during network disruptions +- [Delta compression](/docs/channels/options/deltas) reduces bandwidth while preserving complete data history + +**Compliance-Ready Infrastructure:** - Transport encryption via TLS 1.2+ automatically enforced on all connections - Message history encrypted at rest with AES-256 and regular key rotation -- Audit logging captures all channel access for compliance reporting -- Business Associate Agreement (BAA) available for enterprise customers -- No Protected Health Information (PHI) stored in message routing metadata - -**Data Integrity for Medical Applications:** -- Exactly-once delivery prevents duplicate vital sign readings that could mislead clinical decisions -- Guaranteed message ordering ensures temporal accuracy critical for medical data interpretation -- Automatic connection recovery maintains message continuity during network disruptions -- [Delta compression](/docs/channels/options/deltas) reduces bandwidth while preserving complete state history +- Comprehensive audit logging for compliance reporting +- Business Associate Agreement (BAA) available for healthcare customers requiring HIPAA compliance +- Flexible data retention policies to meet regulatory requirements -**Life-Critical Reliability:** +**Operational Resilience:** - Automatic multi-region failover occurs transparently without client awareness - No single point of failure across Ably's distributed infrastructure - Graceful degradation under load maintains service during traffic spikes -- Proven scalability supporting millions of concurrent connections per channel +- Proven scalability supporting any number of concurrent connections per channel -Ably's [serverless architecture](/docs/platform/architecture) eliminates infrastructure management, automatically scaling to handle any load without provisioning or maintenance. +The following sections explore how to architect production-ready monitoring dashboards, with healthcare patient monitoring as the primary example due to its demanding requirements around compliance, reliability, and data integrity. ## Architecting your patient monitoring dashboard From 7d490fafcabb3e0ed06a44a50490b79e5f253fa3 Mon Sep 17 00:00:00 2001 From: Steven Lindsay Date: Wed, 7 Jan 2026 15:42:14 +0000 Subject: [PATCH 03/16] wip --- src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx b/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx index 94a70b2961..22b16dd2b3 100644 --- a/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx +++ b/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx @@ -4,7 +4,7 @@ meta_description: "Build production-ready realtime monitoring dashboards with Ab meta_keywords: "realtime dashboard, monitoring, telemetry, delta compression, exactly-once delivery, data streaming, HIPAA compliance, medical monitoring, IoT dashboards" --- -Ably is purpose-built for realtime high-throughput data streaming at scale. Whether you're monitoring patient vitals, tracking industrial equipment, or visualizing IoT sensor networks, Ably handles the complexity of reliable message distribution so you can focus on your application. +Ably is purpose-built for mission-critical realtime data streaming at scale. When monitoring systems demand guaranteed delivery, precise ordering, and zero tolerance for downtime—whether tracking patient vitals in intensive care, controlling industrial processes, or managing fleet operations—Ably provides the dependability that critical infrastructure requires. The platform handles the complexity of reliable message distribution, automatic failover, and multi-region resilience so you can focus on your application. Dashboard streaming follows a simple pattern: one or more data sources publish telemetry to channels, and dashboard applications subscribe to receive realtime updates. Messages published to Ably are referred to as inbound messages, while messages delivered to subscribers are outbound messages. From 1658f9dd406588e1d1fa092bb0a4bd85c83f9382 Mon Sep 17 00:00:00 2001 From: Steven Lindsay Date: Thu, 8 Jan 2026 17:34:06 +0000 Subject: [PATCH 04/16] wip --- .../guides/pub-sub/health-tech-dashboard.mdx | 1360 ++++++----------- 1 file changed, 468 insertions(+), 892 deletions(-) diff --git a/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx b/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx index 22b16dd2b3..0fcaac30d3 100644 --- a/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx +++ b/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx @@ -1,1092 +1,668 @@ --- -title: "Guide: Building realtime monitoring dashboards with Ably" -meta_description: "Build production-ready realtime monitoring dashboards with Ably: optimize bandwidth with Deltas, ensure data integrity with exactly-once delivery, achieve compliance with encryption and audit logging." -meta_keywords: "realtime dashboard, monitoring, telemetry, delta compression, exactly-once delivery, data streaming, HIPAA compliance, medical monitoring, IoT dashboards" +title: "Guide: Building realtime dashboards with Ably" +meta_description: "Architecting realtime dashboards with Ably: from fan engagement at scale to critical monitoring. Key decisions, technical depth, and why Ably is the right choice." +meta_keywords: "realtime dashboard, pub/sub, fan engagement, patient monitoring, IoT dashboards, data streaming, scalability, cost optimization" --- -Ably is purpose-built for mission-critical realtime data streaming at scale. When monitoring systems demand guaranteed delivery, precise ordering, and zero tolerance for downtime—whether tracking patient vitals in intensive care, controlling industrial processes, or managing fleet operations—Ably provides the dependability that critical infrastructure requires. The platform handles the complexity of reliable message distribution, automatic failover, and multi-region resilience so you can focus on your application. +Ably Pub/Sub is purpose-built for realtime data distribution at any scale. Whether you're delivering live sports statistics to millions of fans, streaming critical patient vitals to a nurse's station, or updating stock prices across thousands of trading terminals, Ably handles the complexity of message distribution so you can focus on your application. -Dashboard streaming follows a simple pattern: one or more data sources publish telemetry to channels, and dashboard applications subscribe to receive realtime updates. Messages published to Ably are referred to as inbound messages, while messages delivered to subscribers are outbound messages. +Building with Ably means you no longer need to worry about scaling websocket servers, handling failover, or keeping latency low. Ably handles all of this for you, leaving you free to focus on your end-user experience. -## Common realtime dashboard applications - -- **Patient monitoring systems:** Streaming vital signs (heart rate, blood pressure, SpO2) from ICU patients to nurse stations and mobile devices, with continuous telemetry requiring guaranteed delivery and ordering. -- **Industrial equipment monitoring:** Real-time tracking of machinery health, temperature sensors, vibration analysis, and predictive maintenance alerts across manufacturing facilities. -- **Fleet and logistics dashboards:** Live vehicle location, fuel consumption, driver behavior, and delivery status updates for transportation and supply chain operations. -- **Energy and utilities monitoring:** Smart grid data, power consumption metrics, renewable energy generation, and infrastructure health across distributed networks. -- **Environmental monitoring:** Air quality sensors, weather stations, water quality monitoring, and seismic activity tracking across geographic regions. - -This guide addresses common challenges in building realtime monitoring dashboards and shows how Ably's features provide solutions that ensure data integrity, reduce costs, and maintain compliance requirements. While examples focus on healthcare monitoring—a domain with the strictest requirements—the patterns apply to any mission-critical dashboard application. +This guide explains the architectural decisions, technical challenges, and unique benefits of building realtime dashboards with Ably Pub/Sub. It will help you design for scale, reliability, and cost optimization whilst implementing the key features of successful dashboard applications. ## Why Ably for realtime dashboards? -Ably is engineered around the four pillars of dependability: +Ably is trusted by organizations delivering realtime data to millions of users simultaneously. Its platform is engineered around the four pillars of dependability: * **[Performance](/docs/platform/architecture/performance):** Ultra-low latency messaging, even at global scale. -* **[Integrity](/docs/platform/architecture/message-ordering):** Guaranteed message ordering and exactly-once delivery, preventing duplicates and ensuring temporal accuracy. -* **[Reliability](/docs/platform/architecture/fault-tolerance):** 99.999% uptime SLA with automatic failover and seamless reconnection. +* **[Integrity](/docs/platform/architecture/message-ordering):** Guaranteed message ordering and delivery, with no duplicates or data loss. +* **[Reliability](/docs/platform/architecture/fault-tolerance):** 99.999% uptime SLA, with automatic failover and seamless reconnection. * **[Availability](/docs/platform/architecture/edge-network):** Global edge infrastructure ensures users connect to the closest point for optimal experience. -* **[Cost efficiency](#cost-optimization-strategies):** Optimization features like delta compression and message conflation provide significant cost savings while maintaining reliability. -Ably's [serverless architecture](/docs/platform/architecture) eliminates infrastructure management. It automatically scales to handle millions of concurrent connections without provisioning or maintenance. The platform is proven at scale, delivering over 500 billion messages per month for customers, with individual channels supporting millions of concurrent subscribers while maintaining low latency and high throughput. +![Ably Architecture Overview Diagram](../../../../images/content/diagrams/architecture-overview.png) -### Mission-critical capabilities +Delivering dashboard updates in realtime is key to an engaging and effective user experience. Ably's [serverless architecture](/docs/platform/architecture) eliminates the need for you to manage websocket servers. It automatically scales to handle millions of concurrent connections without provisioning or maintenance. Ably also handles all of the edge-cases around delivery, failover and scaling. -**Data Integrity:** -- Exactly-once delivery prevents duplicate readings that could mislead decision-making -- Guaranteed message ordering ensures temporal accuracy for time-sensitive data -- Automatic connection recovery maintains message continuity during network disruptions -- [Delta compression](/docs/channels/options/deltas) reduces bandwidth while preserving complete data history +Despite the challenges of delivering these guarantees, Ably is designed to keep costs predictable. Using features such as server-side batching, delta compression, and efficient connection management, along with Ably's consumption-based pricing model, ensures costs are kept as low as possible, no matter the scale. -**Compliance-Ready Infrastructure:** -- Transport encryption via TLS 1.2+ automatically enforced on all connections -- Message history encrypted at rest with AES-256 and regular key rotation -- Comprehensive audit logging for compliance reporting -- Business Associate Agreement (BAA) available for healthcare customers requiring HIPAA compliance -- Flexible data retention policies to meet regulatory requirements +## Architecting your dashboard: UX first, scale always -**Operational Resilience:** -- Automatic multi-region failover occurs transparently without client awareness -- No single point of failure across Ably's distributed infrastructure -- Graceful degradation under load maintains service during traffic spikes -- Proven scalability supporting any number of concurrent connections per channel +The most important decision you can make when developing a realtime dashboard is understanding the experience you want users to have and the criticality of the data being delivered. This will determine the architecture, feature set, and ultimately the impression your users leave with. -The following sections explore how to architect production-ready monitoring dashboards, with healthcare patient monitoring as the primary example due to its demanding requirements around compliance, reliability, and data integrity. +The key architectural decision is understanding which category your dashboard falls into. With Ably, you are not limited by technology—only by the user experience you want to deliver. Realtime dashboards typically fall into two categories, each with distinct requirements: -## Architecting your patient monitoring dashboard +### Fan engagement dashboards: Streaming to millions -Effective dashboard architecture begins with channel design, data modeling, and optimization strategies tailored to healthcare requirements. +Fan engagement dashboards prioritize broadcasting data to massive audiences where the experience is shared. These include live sports statistics streaming score updates, player stats, and match events to millions of fans. Stock tickers and market data platforms distribute price updates to trading platforms and financial apps. Live event platforms show concert statistics, voting results, and audience participation metrics. Gaming leaderboards deliver real-time rankings and achievement updates to large player bases. -### Channel architecture patterns +In these scenarios, the relationship is typically one publisher broadcasting to many subscribers. High message throughput with sub-second latency is usually sufficient, and eventual consistency is acceptable—not every intermediate value needs to be delivered. Cost optimization becomes critical due to message fanout, and users expect engaging, fast-updating displays rather than mission-critical accuracy. -**Recommended: Patient-centric channels** +### Critical monitoring dashboards: Every message matters -Structure channels around individual patients for natural access control and scalability: +Critical monitoring dashboards prioritize guaranteed delivery, data integrity, and low latency for operational decisions where lives or significant assets may depend on the data. Patient monitoring systems stream vital signs from ICU equipment to nurse stations and mobile devices. Industrial control systems track equipment telemetry, safety alerts, and process monitoring. Fleet management platforms show vehicle locations, driver alerts, and cargo conditions. Energy grid monitoring displays power generation, consumption, and grid stability metrics. -``` -Channel naming: patient:{patient_id}:vitals +These scenarios often involve 1-to-1 or 1-to-few relationships, where one data source streams to specific authorized viewers. Every message must be delivered with guaranteed ordering, and sub-100ms latency may be required. Connection recovery and message continuity are essential, and audit trails and compliance requirements often apply. For healthcare applications, Ably is HIPAA-compliant and offers Business Associate Agreements (BAAs) for customers handling Protected Health Information (PHI). -Benefits: -- Natural access control (subscribe only to authorized patients) -- Isolated streams (one patient's data spike doesn't affect others) -- Horizontal scalability via consistent hashing distributes load -- HIPAA-friendly (patient ID can be de-identified/hashed) -``` +Understanding which category your dashboard falls into—or whether it combines elements of both—is fundamental to making the right architectural decisions throughout this guide. -**Alternative: Device-centric channels** +## Channel design patterns -Use when one device monitors multiple parameters across patients: +Channels are the foundation of your dashboard architecture. They determine how data flows from publishers to subscribers and significantly impact scalability, access control, and costs. The way you structure your channels should reflect both your data model and your access control requirements. -``` -Channel naming: device:{device_id}:telemetry +### Single channel for broadcast -Trade-offs: -- Requires client-side filtering for specific patients -- More complex access control implementation -- Useful for device management and diagnostics -``` +For scenarios where all subscribers receive the same data stream, a single channel provides the simplest and most cost-effective architecture. This pattern works well for public dashboards like sports scores, stock prices, or weather updates where every viewer sees identical information. -**Multi-device synchronization** + + ```javascript + // Publisher: Broadcast live match statistics to all viewers + const channel = realtime.channels.get('match:12345:stats'); -Multiple displays monitoring the same patient (bedside monitor, nurse station, mobile app, family display) all subscribe to the same channel. Ably automatically handles message fanout, guaranteeing identical state across all devices. Connection recovery ensures no device falls behind during network disruptions. + setInterval(async () => { + await channel.publish('stats-update', { + matchId: '12345', + timestamp: Date.now(), + homeScore: 2, + awayScore: 1, + possession: { home: 58, away: 42 }, + shots: { home: 12, away: 8 } + }); +}, 1000); -### Publisher implementation + // Subscriber: Any fan can subscribe to receive updates + const channel = realtime.channels.get('match:12345:stats'); -Medical device gateways typically run on hospital infrastructure, collecting data from monitoring equipment and publishing to Ably: + channel.subscribe('stats-update', (message) => { + updateDashboard(message.data); +}); + ``` + - -```javascript -// Medical device gateway (Node.js) -const Ably = require('ably'); -const realtime = new Ably.Realtime({ key: process.env.ABLY_API_KEY }); +The single channel pattern maximizes cost efficiency because Ably's fanout delivers each published message to all subscribers with a single outbound message charge per subscriber. There's no duplication of data across channels, and the architecture remains simple to reason about. -const patientChannel = realtime.channels.get('patient:7f3a9b2e:vitals'); +### Per-entity channels for isolation and access control -// Publish vital signs update -function publishVitals(vitals) { - patientChannel.publish('vitals-update', { +When different subscribers need access to different data streams, or when fine-grained access control is required, per-entity channels provide natural isolation. This pattern is common in healthcare, where each patient has their own channel, or in multi-tenant SaaS platforms where each customer's data must remain separate. + + + ```javascript + // Publisher: Medical device publishing patient vitals + const patientId = 'patient-7f3a9b2e'; // De-identified patient ID + const channel = realtime.channels.get(`vitals:${patientId}`); + + setInterval(async () => { + await channel.publish('vitals-update', { timestamp: Date.now(), - patientId: '7f3a9b2e', // De-identified hash - heartRate: vitals.heartRate, - bloodPressure: { - systolic: vitals.bloodPressure.systolic, - diastolic: vitals.bloodPressure.diastolic - }, - spO2: vitals.spO2, - temperature: vitals.temperature, - respiratoryRate: vitals.respiratoryRate + heartRate: getCurrentHeartRate(), + bloodPressure: getCurrentBP(), + spO2: getCurrentSpO2(), + temperature: getCurrentTemp() }); -} - -// Collect vitals from medical device at 1Hz -setInterval(() => { - const vitals = collectFromDevice(); - publishVitals(vitals); }, 1000); -``` - -```go -// Medical device gateway (Go) -package main - -import ( - "context" - "time" - "github.com/ably/ably-go/ably" -) - -func main() { - client, _ := ably.NewRealtime(ably.WithKey(apiKey)) - channel := client.Channels.Get("patient:7f3a9b2e:vitals") - - ticker := time.NewTicker(1 * time.Second) - defer ticker.Stop() - - for range ticker.C { - vitals := collectFromDevice() - - vitalSigns := map[string]interface{}{ - "timestamp": time.Now().UnixMilli(), - "patientId": "7f3a9b2e", - "heartRate": vitals.HeartRate, - "bloodPressure": map[string]int{ - "systolic": vitals.BPSystolic, - "diastolic": vitals.BPDiastolic, - }, - "spO2": vitals.SpO2, - "temperature": vitals.Temperature, - "respiratoryRate": vitals.RespiratoryRate, - } - - err := channel.Publish(ctx, "vitals-update", vitalSigns) - if err != nil { - // Handle critical error - vitals not transmitted - log.Printf("Failed to publish vital signs: %v", err) - } - } -} -``` - -### Data modeling for medical telemetry - -Structure vital signs payloads to support delta compression, temporal ordering, and quality monitoring: - -```javascript -{ - "timestamp": 1704563280000, // Unix epoch ms (critical for temporal accuracy) - "patientId": "7f3a9b2e", // De-identified hash - "deviceId": "monitor-icu-3", // Source device tracking - "vitals": { - "heartRate": { - "value": 78, - "unit": "bpm", - "quality": "good" // Signal quality indicator - }, - "bloodPressure": { - "systolic": 120, - "diastolic": 80, - "unit": "mmHg", - "quality": "good" - }, - "spO2": { - "value": 98, - "unit": "%", - "quality": "good" - }, - "temperature": { - "value": 37.2, - "unit": "celsius", - "quality": "good" - }, - "respiratoryRate": { - "value": 16, - "unit": "breaths/min", - "quality": "good" - } - }, - "alarms": [ - { - "type": "high_heart_rate", - "severity": "warning", - "threshold": 100, - "triggered_at": 1704563280000 - } - ], - "metadata": { - "version": "1.0", - "checksum": "abc123" // Data integrity verification - } -} -``` + // Subscriber: Nurse station subscribes only to assigned patients + const assignedPatients = ['patient-7f3a9b2e', 'patient-3c8d1a4f']; -**Design principles:** -- Always include timestamps for temporal ordering -- Include signal quality indicators to detect sensor detachment -- Use structured nested objects (delta compression works best with similar structure between messages) -- Include metadata for versioning and integrity verification -- For high-frequency waveforms, use sequence numbers to detect dropped messages + assignedPatients.forEach(patientId => { + const channel = realtime.channels.get(`vitals:${patientId}`); + channel.subscribe('vitals-update', (message) => { + updatePatientTile(patientId, message.data); +}); +}); + ``` + + +Per-entity channels enable you to apply different [capabilities](/docs/auth/capabilities) to each channel, ensuring that users can only subscribe to the data they're authorized to see. This pattern also provides natural isolation—a spike in activity on one channel doesn't affect others, and you can apply different optimization rules to different channel namespaces. -### High-frequency data and channel limits +### Hierarchical channels for drill-down dashboards -Ably limits inbound messages to 50 msg/s per channel to ensure platform stability. For higher-frequency data sources like ECG waveforms at 250Hz, batch multiple samples into single publishes: +When your dashboard needs to support both overview and detailed views, hierarchical channels provide a flexible solution. A dispatcher might view aggregated fleet metrics until they need to focus on a specific vehicle, at which point they drill down to detailed telemetry. -```javascript -// Batch high-frequency samples into single publishes -const sampleBuffer = []; -const BATCH_SIZE = 25; // 25 samples per publish = 10 publishes/sec at 250Hz - -function collectSample(sample) { - sampleBuffer.push(sample); - - if (sampleBuffer.length >= BATCH_SIZE) { - channel.publish('waveform-batch', { - timestamp: Date.now(), - patientId: '7f3a9b2e', - waveformType: 'ecg_lead_ii', - sampleRate: 250, - samples: sampleBuffer.splice(0, BATCH_SIZE), - sequenceNumber: currentSequence++ - }); - } -} + ```javascript + // Channel naming convention for hierarchical data: + // fleet:overview - Aggregated metrics for all vehicles + // fleet:region:europe - Regional aggregations + // fleet:vehicle:ABC123 - Individual vehicle telemetry + + // Dispatcher subscribes to regional overview for the map + const overviewChannel = realtime.channels.get('fleet:region:europe'); + overviewChannel.subscribe((message) => { + updateMapOverview(message.data); +}); -// Collect ECG samples at 250Hz -setInterval(() => { - collectSample(readECGSample()); -}, 4); // 4ms interval = 250Hz -``` + // When focusing on a specific vehicle, subscribe to detailed telemetry + function focusOnVehicle(vehicleId) { + const vehicleChannel = realtime.channels.get(`fleet:vehicle:${vehicleId}`); + vehicleChannel.subscribe((message) => { + updateVehicleDetails(message.data); +}); +} + ``` -**Important:** When publishing batched messages via REST API from ephemeral connections, consider [message ordering](/docs/pub-sub/advanced#message-ordering) and [idempotency](/docs/pub-sub/advanced#idempotency) requirements. Realtime connections provide serial ordering guarantees; REST does not unless you explicitly set message IDs. +This pattern enables bandwidth optimization because you don't stream detailed telemetry for every vehicle until the user actually needs it. You can also apply different update frequencies to different levels of the hierarchy—overview data might update every 5 seconds while detailed vehicle data updates every 100ms. -### Optimization: Delta compression for vital signs +## Message throughput and rate management -Vital signs messages often grow over time as cumulative patient data accumulates. Repeatedly sending 3-5KB payloads every second wastes bandwidth, especially for mobile and tablet displays. +Dashboard applications face varying throughput demands, from steady streams of updates to sudden spikes during significant events. When a goal is scored, markets open, or an incident occurs, message rates can spike dramatically. Ably [is engineered](/docs/platform/architecture/platform-scalability) to handle these loads without degradation, delivering over 500 billion messages per month across its customer base. -[Delta compression](/docs/channels/options/deltas) solves this by transmitting only changes between successive messages. Publishers continue sending complete state (maintaining simplicity), while Ably computes deltas server-side and transmits only changes. Subscriber SDKs automatically reconstruct full state from deltas. +### Understanding rate limits -**When to use delta compression:** -- ✅ Vital signs with incremental changes (most parameters remain stable between updates) -- ✅ Multi-parameter monitoring where only 1-2 vitals change per update -- ✅ Mobile or bandwidth-constrained displays -- ✅ Payloads > 1KB with high similarity between successive messages +Ably applies rate limits to ensure platform stability. By default, channels accept up to 50 inbound messages per second. Enterprise plans can request higher limits for specific use cases. When working with high-frequency data sources, you'll need to either batch updates or distribute across multiple channels. -**When NOT to use delta compression:** -- ❌ High-frequency waveforms where most data changes between messages -- ❌ Small alert messages (< 1KB where overhead exceeds savings) -- ❌ Scenarios requiring channel-level encryption (deltas are incompatible; use TLS transport encryption instead) - -**Implementation:** +For data sources generating more than 50 updates per second, batching multiple samples into single publishes is the recommended approach: -```javascript -// Dashboard: Subscribe with delta compression enabled -const vcdiffPlugin = require('@ably/vcdiff-decoder'); + ```javascript + // Batch high-frequency sensor readings into periodic publishes + const sensorBuffer = []; + const BATCH_INTERVAL_MS = 100; // Publish every 100ms + const MAX_BATCH_SIZE = 10; + + function collectSensorReading(reading) { + sensorBuffer.push({ + sensorId: reading.id, + value: reading.value, + timestamp: Date.now() + }); -const realtime = new Ably.Realtime({ - key: 'your-api-key', - plugins: { vcdiff: vcdiffPlugin } -}); + if (sensorBuffer.length >= MAX_BATCH_SIZE) { + flushBuffer(); +} +} -const channel = realtime.channels.get('patient:7f3a9b2e:vitals', { - params: { delta: 'vcdiff' } -}); + function flushBuffer() { + if (sensorBuffer.length === 0) return; -channel.subscribe('vitals-update', (msg) => { - // SDK automatically reconstructs full state from delta - // New joiners receive full state on first message - updateVitalsDisplay(msg.data); + channel.publish('sensor-batch', { + readings: sensorBuffer.splice(0), + batchTimestamp: Date.now() }); -``` - - -See the [delta compression documentation](/docs/channels/options/deltas#subscribe) for plugin installation details and browser compatibility. - -**Bandwidth savings example:** -- Full payload: ~3KB (comprehensive vital signs with history) -- Delta payload: ~300-500 bytes (typical 80-85% compression) -- 1Hz updates × 10 subscribed devices = 30KB/s → 3-5KB/s -- **Result: 80-85% bandwidth reduction** - -**Key considerations:** -- First message after connection is always full state -- Connection recovery sends full state, then resumes delta mode -- Monitor compression ratios via [application statistics](/docs/metadata-stats/stats) -- For cost optimization, use the [persist last message](/docs/storage-history/storage#persist-last-message) rule to store only the final complete state rather than every delta - -## Ensuring data freshness with message conflation +} -Medical devices often publish vital signs faster than dashboards can render them. A device publishing at 10Hz (10 updates/second) overwhelms a dashboard UI that refreshes at 1-2 FPS due to complex chart rendering. Without optimization, 8-9 stale updates accumulate per second in the browser queue. + // Flush periodically even if batch isn't full + setInterval(flushBuffer, BATCH_INTERVAL_MS); + ``` + -### The staleness problem +This approach keeps you well within rate limits while still delivering timely updates. A 100ms batching interval means subscribers see data with at most 100ms additional latency, which is imperceptible for most dashboard use cases. -This creates three issues: -- Clinicians see delayed vital signs (dangerous in critical care) -- Dashboard becomes unresponsive (unusable during emergencies) -- Bandwidth wasted on data that's never displayed +### Server-side batching for cost efficiency -### Solution: Message conflation +During high-activity periods—a goal being scored, market volatility, or a major incident—message rates can spike dramatically. [Server-side batching](/docs/messages/batch#server-side) helps manage these spikes by grouping messages before delivery to subscribers. -[Message conflation](/docs/messages#conflation) aggregates messages on the server over a configured time window (e.g., 100ms), discards intermediate values, and delivers only the most current state to subscribers. +Configure server-side batching via [channel rules](/docs/channels#rules) in your Ably dashboard. Navigate to your app settings, create a new rule for your channel namespace (e.g., `match:*:stats`), enable server-side batching, and set the batching interval. For responsive dashboards, 100ms provides a good balance. For maximum cost optimization, intervals up to 1000ms can significantly reduce message counts. -Configure conflation through [channel rules](/docs/channels#rules) in your Ably dashboard: +The key benefit of server-side batching is that it reduces billable outbound message count during traffic spikes. If your source publishes 10 updates per second and you have 1000 subscribers, without batching you'd have 10,000 outbound messages per second. With 500ms batching, messages are grouped into 2 batches per second, resulting in 2,000 outbound messages per second—a 5x reduction. -1. Navigate to app settings → Channel Rules -2. Create rule for channel pattern: `patient:*:vitals` -3. Enable conflation with interval (e.g., 100ms) -4. Set conflation key pattern: `#{message.name}` +Unlike message conflation, server-side batching preserves all messages. Every update is delivered, just grouped together for efficiency. This makes it suitable for scenarios where you need complete data but want to smooth out traffic spikes. -**When to use conflation:** -- ✅ Display refresh rate < data publish rate -- ✅ Only current value matters (e.g., current heart rate, not historical sequence) -- ✅ Preventing UI performance degradation -- ✅ Cost-sensitive deployments where intermediate values aren't critical +### Message conflation for latest-value scenarios -**When NOT to use conflation:** -- ❌ Every update must be processed (e.g., waveform displays) -- ❌ Audit trail or compliance requirements (conflation permanently discards intermediate messages) -- ❌ Alert messages (small, infrequent, every alert matters) +For dashboards where only the current value matters—stock prices, sensor readings, vehicle positions—[message conflation](/docs/messages#conflation) delivers only the most recent value within each time window. -**Critical warning:** +Configure conflation through [channel rules](/docs/channels#rules) in your dashboard. You'll specify a conflation interval (e.g., 100ms) and a conflation key pattern that determines which messages are considered related. Messages with the same conflation key within the time window are conflated together, with only the latest delivered. -**Performance example:** -- Device publishes at 10Hz = 10 msg/sec -- With 100ms conflation = 10 msg/sec → ~10 msg/sec delivered (one per 100ms window) -- For 10 subscribed dashboards: 100 msg/sec → 100 msg/sec outbound -- **Result: Prevents dashboard overload while maintaining current state visibility** - -## Multi-device synchronization and connection recovery - -A single ICU patient may be monitored simultaneously by a bedside display, nurse station dashboard showing 10 patients, doctor's mobile app, and family viewing screen. All must show identical, current vital signs. Any desynchronization could lead to clinical errors. +Conflation dramatically reduces costs when publishers send updates faster than users can perceive. If a price feed publishes 100 updates per second but your dashboard refreshes at 10 FPS, you're wasting 90% of your message budget on updates users never see. With 100ms conflation, you reduce outbound messages by 10x while showing users the most current data available. -### Ably's synchronization guarantee +### Delta compression for large payloads -All devices subscribe to the same channel (`patient:7f3a9b2e:vitals`). Ably guarantees: -- **Message ordering** ensures all devices see updates in the same sequence -- **Exactly-once delivery** prevents any device from receiving duplicate readings -- **Automatic fanout** to all subscribers without application logic -- **Connection-independent state** allows devices to join/leave independently +When dashboard payloads are large but change incrementally between updates, [delta compression](/docs/channels/options/deltas) reduces bandwidth by transmitting only the changes. Publishers continue to send complete state—the delta calculation happens automatically in Ably's infrastructure—while subscribers receive compressed updates that the SDK reconstructs into full state. -```javascript -// Bedside monitor -const channel = realtime.channels.get('patient:7f3a9b2e:vitals'); -channel.subscribe('vitals-update', (msg) => { - updateBedsideMonitor(msg.data); -}); + ```javascript + // Publisher: Send full dashboard state (no code changes needed) + const channel = realtime.channels.get('dashboard:overview'); + + setInterval(async () => { + await channel.publish('state-update', { + activeUsers: 15234, + transactions: 892341, + revenue: 1234567.89, + serverLoad: { us: 45, eu: 52, asia: 38 }, + topProducts: [...], // Array of 50+ products + recentEvents: [...], // Array of recent activity + // Most fields change only slightly between updates + }); +}, 1000); -// Nurse station (same subscription, different UI) -channel.subscribe('vitals-update', (msg) => { - updatePatientTile('7f3a9b2e', msg.data); -}); + // Subscriber: Enable delta compression + const Vcdiff = require('@ably/vcdiff-decoder'); -// Mobile app (same subscription, optimized for mobile) -channel.subscribe('vitals-update', (msg) => { - updateMobileVitals(msg.data); + const realtime = new Ably.Realtime({ + key: 'your-api-key', + plugins: { vcdiff: Vcdiff } }); -``` - - -### Handling network disruptions - -Network disruptions occur frequently in hospitals: Wi-Fi dead zones in elevators and stairwells, cellular connectivity drops for mobile clinicians, network maintenance windows, and infrastructure failures. Dashboards must automatically reconnect and recover missed data without clinician intervention. -**Ably's automatic recovery:** - - -```javascript -realtime.connection.on('disconnected', () => { - // Ably SDK automatically attempts reconnection - showReconnectingIndicator(); + const channel = realtime.channels.get('dashboard:overview', { + params: { delta: 'vcdiff' } }); -realtime.connection.on('connected', () => { - // Connection restored, channel state recovered automatically - hideReconnectingIndicator(); - // Ably SDK has already backfilled missed messages + channel.subscribe('state-update', (message) => { + // SDK automatically reconstructs full state from deltas + updateDashboard(message.data); }); -``` + ``` -The SDK tracks the last received message ID. On reconnection, it requests missed messages from Ably and seamlessly backfills to current state. No application logic required. +Delta compression is particularly effective for dashboards that display comprehensive state where most values remain stable. A 5KB dashboard payload where only a few fields change each second might compress to 500 bytes—an 90% bandwidth reduction. Across 1000 subscribers receiving updates every second, that's the difference between 5MB/s and 500KB/s of outbound data. -If the primary region becomes unreachable, the SDK automatically tries fallback regions until connection is restored. This multi-region failover is transparent to your application. +## Authentication -**History backfill for extended disconnections:** +Authentication determines who can connect to your dashboard and what they can access. The approach differs significantly between fan engagement and critical monitoring scenarios, reflecting the different security requirements of each. -Ably stores message history for 2 minutes by default (sufficient for connection recovery and providing context when joining mid-stream). This can be extended up to 1 year if needed. For disconnections exceeding the history retention period, explicitly request history: +### Fan engagement: Anonymous and simple access + +For public dashboards where anyone can view the data, authentication focuses on preventing abuse rather than controlling access. You'll typically generate anonymous tokens that allow subscription but prevent publishing, ensuring viewers can watch but can't inject fake data. -```javascript -channel.subscribe('vitals-update', (msg) => { - updateDisplay(msg.data); -}); + ```javascript + // Server: Generate tokens for anonymous viewers + const jwt = require('jsonwebtoken'); + + function generateViewerToken() { + const header = { + typ: 'JWT', + alg: 'HS256', + kid: '{{ API_KEY_NAME }}' +}; + + const currentTime = Math.round(Date.now() / 1000); + + const claims = { + iat: currentTime, + exp: currentTime + 3600, // 1 hour expiration + 'x-ably-capability': JSON.stringify({ + 'match:*:stats': ['subscribe'], // Can only subscribe, not publish + 'match:*:reactions': ['subscribe', 'publish'] // Can send reactions +}), + 'x-ably-clientId': `viewer-${crypto.randomUUID()}` +}; + + return jwt.sign(claims, '{{ API_KEY_SECRET }}', { header }); +} -// After extended disconnection, backfill missed data -channel.history({ limit: 100 }, (err, resultPage) => { - if (err) { - console.error('Failed to retrieve history:', err); - return; - } - - // Display gap-fill indicator to user - resultPage.items.forEach((msg) => { - updateDisplayWithHistoricalData(msg.data); - }); + // Client: Connect with viewer token + const realtime = new Ably.Realtime({ + authCallback: async (tokenParams, callback) => { + const token = await fetch('/api/ably-token').then(r => r.text()); + callback(null, token); +} }); -``` + ``` -**Best practices:** -- Display connection state indicator in UI (connected/reconnecting/disconnected) -- Use history API for disconnections exceeding retention period -- Test recovery scenarios in realistic hospital network conditions -- Configure [connection timeout](/docs/connect/states#connection-state-recovery) based on application criticality - -See the [fault tolerance documentation](/docs/platform/architecture/fault-tolerance) for details on Ably's infrastructure resilience. - -## HIPAA compliance and security +The token capabilities restrict viewers to subscribing on stats channels while allowing them to both subscribe and publish on reaction channels. This enables interactive features like emoji reactions without compromising the integrity of the primary data stream. -Healthcare applications handling Protected Health Information (PHI) must comply with HIPAA's Security Rule. Ably provides the infrastructure and capabilities needed to build HIPAA-compliant applications. +### Critical monitoring: Strict access control -### HIPAA requirements overview +For sensitive dashboards where access must be carefully controlled, authentication ties directly into your authorization system. Each user receives a token that grants access only to the specific entities they're authorized to monitor. -- **Encryption:** Data encrypted at rest and in transit (§164.312(a)(2)(iv)) -- **Access Control:** Only authorized users access patient data (§164.312(a)(1)) -- **Audit Trails:** All data access must be logged (§164.312(b)) -- **Data Integrity:** Prevent unauthorized modification (§164.312(c)(1)) -- **Business Associate Agreement (BAA):** Required for vendors handling PHI - -### How Ably supports HIPAA compliance + + ```javascript + // Server: Generate tokens based on user's authorization + async function generateMonitoringToken(userId) { + // Look up user's authorized patients/equipment/entities + const authorizedEntities = await getAuthorizedEntities(userId); -**Encryption:** -- Transport encryption via TLS 1.2+ automatically enforced on all connections -- Perfect Forward Secrecy (PFS) enabled -- Message history encrypted at rest with AES-256 -- Encryption keys rotated regularly -- Certificate pinning available for mobile apps + // Build capability for only their authorized channels + const capability = {}; + authorizedEntities.forEach(entityId => { + capability[`vitals:${entityId}`] = ['subscribe']; + capability[`alerts:${entityId}`] = ['subscribe', 'publish']; // Can acknowledge alerts +}); -**Access Control:** + const header = { + typ: 'JWT', + alg: 'HS256', + kid: '{{ API_KEY_NAME }}' +}; -Implement token-based authentication with capability constraints: + const currentTime = Math.round(Date.now() / 1000); - -```javascript -// Generate token on secure backend (never in client code) -const Ably = require('ably'); -const rest = new Ably.Rest({ key: process.env.ABLY_API_KEY }); + const claims = { + iat: currentTime, + exp: currentTime + 1800, // 30 minute expiration for sensitive data + 'x-ably-capability': JSON.stringify(capability), + 'x-ably-clientId': userId +}; -async function generatePatientToken(nurseId, assignedPatientIds) { - const capability = {}; - - // Grant access only to assigned patients - assignedPatientIds.forEach(patientId => { - capability[`patient:${patientId}:vitals`] = ['subscribe']; - capability[`patient:${patientId}:alerts`] = ['subscribe', 'publish']; // Can acknowledge alerts - }); - - const tokenRequest = await rest.auth.createTokenRequest({ - clientId: nurseId, - capability: capability, - ttl: 3600000 // 1 hour expiration (short-lived tokens) - }); - - return tokenRequest; + return jwt.sign(claims, '{{ API_KEY_SECRET }}', { header }); } - -// Client uses token -const realtime = new Ably.Realtime({ - authCallback: async (tokenParams, callback) => { - // Fetch token from your secure backend - const tokenRequest = await fetch('/api/ably-token').then(r => r.json()); - callback(null, tokenRequest); - } -}); -``` + ``` -**Audit Logging:** - -Ably provides audit-ready logs via integrations. Export logs to your SIEM: -- Who accessed which channel (client ID) -- When access occurred (timestamp) -- What actions were performed (publish/subscribe) -- Connection source (IP address, user agent) - -Common integration patterns: -- Kafka integration → Elasticsearch → Kibana -- AWS Lambda → CloudWatch Logs -- Azure Functions → Azure Monitor +The shorter token expiration for sensitive dashboards ensures that if a user's access is revoked, their existing token expires quickly. The capabilities are built dynamically based on the user's current authorizations, so changes in your authorization system are reflected within the token expiration period. -See [Ably integrations](/docs/platform/integrations) for configuration details. +For production deployments, you should never expose API keys in client-side code. Always use token authentication, with tokens generated by your server based on the user's authenticated session. Apply the principle of least privilege, granting only the capabilities each user actually needs. For compliance scenarios, use [integration rules](/docs/platform/integrations) to log channel activity for audit trails. -**Data Integrity:** +### HIPAA compliance for healthcare dashboards -Ably guarantees: -- Message ordering (temporal accuracy for medical data) -- Exactly-once delivery (no duplicate vital readings) -- Checksum verification (detect data corruption) -- Immutable message history (tamper-proof audit trail) +For healthcare applications handling Protected Health Information (PHI), Ably provides the infrastructure needed to build HIPAA-compliant applications. All data is encrypted in transit via TLS 1.2+ and at rest with AES-256. Ably offers Business Associate Agreements (BAAs) for enterprise customers, and comprehensive audit logging is available through integration rules. When building patient monitoring dashboards, use de-identified patient IDs in channel names rather than actual patient identifiers, and ensure your token generation ties into your existing healthcare authorization systems. -**Business Associate Agreement (BAA):** +## Handling network disruption -Ably provides HIPAA BAA to enterprise customers, covering PHI transmission through Ably's infrastructure and committing to HIPAA security and privacy requirements. Contact [Ably sales](https://ably.com/contact) for BAA execution. +Network disruptions are inevitable—mobile devices lose signal, users switch networks, or infrastructure experiences issues. Dashboard applications must handle these gracefully, ensuring users understand what's happening and recover smoothly when connectivity returns. -### De-identification best practices +### Automatic reconnection and connection state -Use de-identified patient IDs in channel names and message payloads to reduce PHI exposure: +Ably's SDKs automatically handle reconnection and [connection state recovery](/docs/connect/states#connection-state-recovery). When a connection drops, the SDK will automatically attempt to reconnect using exponential backoff, trying multiple data centers if necessary. Your application should monitor connection state to provide appropriate user feedback: -```javascript -// ❌ Don't use PHI directly -const channelName = 'patient:John-Doe-DOB-1985-03-15:vitals'; - -// ✅ Use de-identified hash -const crypto = require('crypto'); -const patientHash = crypto - .createHash('sha256') - .update('internal-patient-id-12345') - .digest('hex') - .substring(0, 12); - -const channelName = `patient:${patientHash}:vitals`; - -// Store mapping in your secure database: -// internal-patient-id-12345 → 7f3a9b2e8c1d (hash) -``` - + ```javascript + // Monitor connection state for UI feedback + realtime.connection.on('connected', () => { + hideConnectionWarning(); + console.log('Connected to Ably'); +}); -**Benefits:** -- Reduces PHI exposure in Ably infrastructure -- Simplifies compliance audit (less PHI to track) -- Channel names can be safely logged and monitored + realtime.connection.on('disconnected', () => { + showReconnectingIndicator(); + console.log('Disconnected - attempting to reconnect...'); +}); -**Important:** De-identification does NOT eliminate HIPAA requirements. You still need BAA, encryption, access controls, and audit trails. + realtime.connection.on('suspended', () => { + showConnectionError('Connection suspended - will keep trying'); +}); -## External storage and long-term data retention + realtime.connection.on('failed', () => { + showConnectionError('Connection failed - please refresh'); +}); + ``` + -Ably's message history (2 minutes by default, extendable to 1 year) enables connection recovery and context on join but is not designed for long-term medical records. Healthcare organizations require years of retention for compliance, complex querying for trend analysis, and data warehouse integration for business intelligence. +The `disconnected` state indicates a temporary loss of connection where the SDK is actively trying to reconnect. The `suspended` state indicates that reconnection attempts have been unsuccessful for an extended period, but the SDK will continue trying. The `failed` state indicates that the connection cannot be recovered and manual intervention is required. -### Architecture pattern: Real-time + archival +### Message continuity after reconnection -Stream data through Ably for real-time delivery to dashboards while simultaneously archiving to your own storage infrastructure: +Ably maintains a 2-minute message buffer for each connection. When a client reconnects within this window, any messages published during the disconnection are automatically delivered, ensuring no data is lost during brief network interruptions. -``` -Medical Device → Ably Pub/Sub → [1] Real-time Dashboard (immediate display) - → [2] Ably Integration → Kafka → Time-series DB - → Data Lake (S3) - → Analytics Pipeline -``` +For longer disconnections, or when you need to backfill historical data, use the [history API](/docs/storage-history/history) to retrieve missed messages: -**Why this pattern:** -- Real-time path: Ably handles immediate delivery to dashboards -- Archival path: Ably integration forwards all messages to your storage -- Decoupled: Storage issues don't affect real-time monitoring -- Compliant: Meet your specific retention requirements without impacting live systems + + ```javascript + // Track the timestamp of the last received message + let lastReceivedTimestamp = Date.now(); -**Cost optimization:** If using delta compression, configure the [persist last message](/docs/storage-history/storage#persist-last-message) rule to store only the final complete state rather than every delta. This provides the latest state for reconnections while reducing storage costs, with full historical data captured in your external archive. + channel.subscribe('update', (message) => { + lastReceivedTimestamp = message.timestamp; + updateDashboard(message.data); +}); + + // After reconnection, backfill any potentially missed messages + realtime.connection.on('connected', async () => { + // Only backfill if this is a reconnection, not initial connection + if (wasConnectedBefore) { + try { + const history = await channel.history({ + start: lastReceivedTimestamp, + direction: 'forwards', + limit: 100 +}); -### Implementation: Ably integration to Kafka + history.items.forEach(message => { + if (message.timestamp > lastReceivedTimestamp) { + updateDashboard(message.data); + lastReceivedTimestamp = message.timestamp; +} +}); +} catch (error) { + console.error('Failed to backfill history:', error); +} +} + wasConnectedBefore = true; +}); + ``` + -Configure Ably integration via your dashboard: +For critical monitoring dashboards, this message continuity is essential. A nurse checking vital signs needs to know that the data displayed is current and complete, not missing updates from a brief network interruption. -1. Navigate to app settings → Integrations -2. Create Kafka integration -3. Configure rule: Trigger on all messages to `patient:*:vitals` -4. Kafka broker: `your-kafka-cluster.hospital.internal:9092` -5. Topic: `patient-vitals-archive` -6. Format: JSON (or Confluent Avro for schema registry) +### Graceful degradation during connectivity issues -**Kafka consumer to time-series database:** +Even with automatic reconnection, there will be periods where your dashboard doesn't have current data. Design your UI to communicate this clearly to users, indicating both the age of the displayed data and the connection status: -```go -// Kafka consumer to persist vitals to time-series database -package main - -import ( - "context" - "encoding/json" - "log" - "time" - "github.com/segmentio/kafka-go" -) - -type VitalsMessage struct { - Timestamp int64 `json:"timestamp"` - PatientID string `json:"patientId"` - DeviceID string `json:"deviceId"` - Vitals struct { - HeartRate struct { - Value int `json:"value"` - Quality string `json:"quality"` - } `json:"heartRate"` - BloodPressure struct { - Systolic int `json:"systolic"` - Diastolic int `json:"diastolic"` - Quality string `json:"quality"` - } `json:"bloodPressure"` - SpO2 struct { - Value int `json:"value"` - Quality string `json:"quality"` - } `json:"spO2"` - Temperature struct { - Value float64 `json:"value"` - Quality string `json:"quality"` - } `json:"temperature"` - RespiratoryRate struct { - Value int `json:"value"` - Quality string `json:"quality"` - } `json:"respiratoryRate"` - } `json:"vitals"` -} + ```javascript + // Track data freshness for UI indication + let lastUpdateTime = Date.now(); + const STALE_THRESHOLD_MS = 5000; // Consider data stale after 5 seconds + + channel.subscribe('update', (message) => { + lastUpdateTime = Date.now(); + markDataFresh(); + updateDashboard(message.data); +}); -func consumeVitals(db *sql.DB) { - reader := kafka.NewReader(kafka.ReaderConfig{ - Brokers: []string{"kafka.hospital.internal:9092"}, - Topic: "patient-vitals-archive", - GroupID: "vitals-archiver", - }) - defer reader.Close() - - for { - msg, err := reader.ReadMessage(context.Background()) - if err != nil { - log.Println("Error reading from Kafka:", err) - continue - } - - var vitals VitalsMessage - if err := json.Unmarshal(msg.Value, &vitals); err != nil { - log.Println("Error parsing vitals:", err) - continue - } - - storeVitals(db, vitals) - } -} + // Check freshness periodically and update UI accordingly + setInterval(() => { + const timeSinceUpdate = Date.now() - lastUpdateTime; -func storeVitals(db *sql.DB, vitals VitalsMessage) { - // Insert into TimescaleDB (PostgreSQL-based time-series DB) - query := ` - INSERT INTO patient_vitals - (patient_id, timestamp, heart_rate, blood_pressure_systolic, - blood_pressure_diastolic, spo2, temperature, respiratory_rate, - device_id, heart_rate_quality) - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) - ` - - _, err := db.Exec( - query, - vitals.PatientID, - time.UnixMilli(vitals.Timestamp), - vitals.Vitals.HeartRate.Value, - vitals.Vitals.BloodPressure.Systolic, - vitals.Vitals.BloodPressure.Diastolic, - vitals.Vitals.SpO2.Value, - vitals.Vitals.Temperature.Value, - vitals.Vitals.RespiratoryRate.Value, - vitals.DeviceID, - vitals.Vitals.HeartRate.Quality, - ) - - if err != nil { - log.Println("Error storing vitals:", err) - // Consider dead letter queue for failed writes - } + if (timeSinceUpdate > STALE_THRESHOLD_MS) { + markDataStale(); + showLastUpdateTime(lastUpdateTime); + // Display something like "Last updated 15 seconds ago" } -``` +}, 1000); + ``` -**Key decisions:** -- Storage technology: Time-series DB (TimescaleDB, InfluxDB) vs Data Lake (S3 + Parquet) -- Retention policy: Automated deletion based on your legal/business requirements -- Schema management: Avro or Protobuf for forward compatibility -- Failure handling: Dead letter queue for failed writes - -Your retention period should be determined by your specific legal and business requirements, which vary by jurisdiction and use case. +For fan engagement dashboards, showing slightly stale data with a "last updated" indicator is usually acceptable. For critical monitoring dashboards, you might want more aggressive staleness thresholds and clearer visual warnings when data isn't current. -## Alerting and threshold monitoring +## Presence and occupancy -Real-time clinical alerts ensure care teams respond immediately to critical changes in patient condition. +[Presence](/docs/presence-occupancy/presence) and [occupancy](/docs/presence-occupancy/occupancy) provide awareness of who's connected and how many viewers are engaged with your dashboard. The choice between them depends on whether you need to know _who_ is watching or just _how many_ are watching. -### Alert architecture patterns +### Presence: Knowing who's watching -**Option 1: Client-side alerting (simple, low latency)** - -Each dashboard implements threshold checks locally: +Presence is useful for collaborative dashboards where operators need to coordinate, or where seeing who else is viewing the same data provides context. Each client can enter a channel's presence set with associated data, and all presence subscribers receive notifications when members enter, leave, or update their data. -```javascript -channel.subscribe('vitals-update', (msg) => { - const vitals = msg.data; - - if (vitals.heartRate.value > 120) { - triggerAlert('HIGH_HEART_RATE', vitals); - } - - if (vitals.spO2.value < 90) { - triggerAlert('LOW_SPO2', vitals); - } - - updateDisplay(vitals); + ```javascript + // Enter presence when opening the dashboard + const channel = realtime.channels.get('ops:control-room'); + + await channel.presence.enter({ + name: 'Sarah Johnson', + role: 'supervisor', + station: 'Control Room A' }); -``` - -**Option 2: Server-side alerting (centralized, auditable)** - -A dedicated alert service monitors thresholds and publishes alerts to a separate channel: + // Subscribe to see who else is watching + channel.presence.subscribe('enter', (member) => { + addToActiveUsersList({ + clientId: member.clientId, + name: member.data.name, + role: member.data.role + }); +}); -``` -Medical Device → Ably → Alert Service (monitors thresholds) - ↓ - Alerts Channel → All Dashboards -``` + channel.presence.subscribe('leave', (member) => { + removeFromActiveUsersList(member.clientId); +}); -**Recommendation:** Use server-side alerting for: -- Centralized alert logic (single source of truth) -- Alert deduplication (prevent alert fatigue) -- Audit trail (regulatory requirement) -- Multi-user coordination (alert acknowledgment tracking) + // Get the current list of viewers + const members = await channel.presence.get(); + members.forEach(member => { + addToActiveUsersList({ + clientId: member.clientId, + name: member.data.name, + role: member.data.role + }); +}); -### Implementation: Alert service with acknowledgment + // Update presence data when status changes + await channel.presence.update({ + name: 'Sarah Johnson', + role: 'supervisor', + station: 'Control Room A', + status: 'handling incident' +}); + ``` + -Medical alerts often require acknowledgment to ensure clinical staff have seen critical notifications. Ably's [message annotations](/docs/messages/annotations) feature provides a reliable mechanism for alert acknowledgment. +Presence is powerful but expensive at scale. Every enter, leave, and update event generates messages delivered to all presence subscribers. For a channel with 1000 viewers all subscribed to presence, a single user joining triggers 1000 outbound messages. If users are frequently joining and leaving, this can quickly dominate your message costs. -**How it works:** -1. Alert service publishes alert messages to `patient:{id}:alerts` channel -2. Clinicians receive alerts on their dashboards -3. When acknowledged, dashboard publishes a `flag.v1` annotation with `name: 'acknowledged'` -4. Alert service monitors annotation summaries to track acknowledgment status -5. If not acknowledged within threshold, escalate to supervisor +### Occupancy: Counting viewers efficiently -**Dashboard implementation:** +For fan engagement dashboards where you want to show viewer counts without needing to know individual identities, occupancy provides an efficient alternative. Occupancy gives you aggregate metrics about channel connections without the per-event overhead of full presence. -```javascript -// Dashboard: Subscribe to alerts and handle acknowledgment -const alertChannel = realtime.channels.get('patient:7f3a9b2e:alerts', { - modes: ['MESSAGE_SUBSCRIBE', 'ANNOTATION_PUBLISH'] + ```javascript + // Enable occupancy updates on channel attachment + const channel = realtime.channels.get('match:12345:stats', { + params: { occupancy: 'metrics' } }); -alertChannel.subscribe((message) => { - if (message.action === 'message.create') { - const alert = message.data; - - // Display alert banner with audio notification - displayAlertBanner(alert); - playAlertSound(alert.severity); - - // Clinician clicks "Acknowledge" button - document.getElementById('ack-btn').onclick = async () => { - await alertChannel.annotations.publish(message, { - type: 'acknowledgment:flag.v1', - name: 'acknowledged' - }); - hideAlertBanner(); - }; - } - - // Track annotation summaries to show who acknowledged - if (message.action === 'message.summary') { - const ackSummary = message.annotations?.summary?.['acknowledgment:flag.v1']; - if (ackSummary) { - console.log(`Alert acknowledged by: ${ackSummary.clientIds.join(', ')}`); - } - } + // Subscribe to occupancy updates + channel.subscribe('[meta]occupancy', (message) => { + const metrics = message.data.metrics; + updateViewerCount(metrics.subscribers); + + // Additional metrics available: + // metrics.connections - total connections to channel + // metrics.publishers - connections with publish capability + // metrics.presenceMembers - members in presence set }); -``` + ``` -**Alert service implementation:** +Occupancy updates are debounced and delivered efficiently, making them suitable for channels with thousands or millions of viewers. The overhead is minimal compared to full presence, and you still get the "15,234 people watching" social proof that enhances fan engagement experiences. - -```go -// Alert service monitoring thresholds and acknowledgments -package main - -import ( - "context" - "log" - "time" - "github.com/ably/ably-go/ably" -) - -type AlertService struct { - realtime *ably.Realtime - thresholds map[string]VitalThresholds -} +### Scaling presence for large audiences -type VitalThresholds struct { - HeartRateMax int - HeartRateMin int - SpO2Min int -} +If you need presence-like functionality with thousands of users, consider a hybrid approach. Use occupancy for the aggregate viewer count that everyone sees, but enable full presence only for specific user groups who need to see individual identities. -type Alert struct { - Type string `json:"type"` - Severity string `json:"severity"` - PatientID string `json:"patientId"` - Value int `json:"value"` - Threshold int `json:"threshold"` - Timestamp int64 `json:"timestamp"` -} + + ```javascript + // Regular viewers: occupancy only, no presence overhead + const statsChannel = realtime.channels.get('match:stats', { + params: { occupancy: 'metrics' } +}); -func (s *AlertService) MonitorPatient(ctx context.Context, patientID string) { - vitalsChannel := s.realtime.Channels.Get("patient:" + patientID + ":vitals") - alertChannel := s.realtime.Channels.Get("patient:" + patientID + ":alerts") - - // Subscribe to vitals and check thresholds - _, err := vitalsChannel.Subscribe(ctx, "vitals-update", func(msg *ably.Message) { - vitals := msg.Data.(map[string]interface{}) - - // Check heart rate threshold - if hr, ok := getHeartRate(vitals); ok { - if hr > s.thresholds[patientID].HeartRateMax { - s.publishAlert(ctx, alertChannel, Alert{ - Type: "HIGH_HEART_RATE", - Severity: "WARNING", - PatientID: patientID, - Value: hr, - Threshold: s.thresholds[patientID].HeartRateMax, - Timestamp: time.Now().UnixMilli(), - }) - } - } - - // Check SpO2 threshold - if spO2, ok := getSpO2(vitals); ok { - if spO2 < s.thresholds[patientID].SpO2Min { - s.publishAlert(ctx, alertChannel, Alert{ - Type: "LOW_SPO2", - Severity: "CRITICAL", // SpO2 < 90% is critical - PatientID: patientID, - Value: spO2, - Threshold: s.thresholds[patientID].SpO2Min, - Timestamp: time.Now().UnixMilli(), - }) - } - } - }) - - if err != nil { - log.Fatal("Failed to subscribe to vitals:", err) - } - - // Monitor alert acknowledgments via annotation summaries - _, err = alertChannel.Subscribe(ctx, func(msg *ably.Message) { - if msg.Action == ably.MessageActionSummary { - ackSummary := msg.Annotations.Summary["acknowledgment:flag.v1"] - if ackSummary != nil { - log.Printf("Alert %s acknowledged by %v", msg.Serial, ackSummary.ClientIDs) - // Cancel escalation timer for this alert - cancelEscalation(msg.Serial) - } - } - }) - - if err != nil { - log.Fatal("Failed to subscribe to alert acknowledgments:", err) - } -} + // VIP/moderators: also join a separate presence channel + if (userIsVIP || userIsModerator) { + const vipChannel = realtime.channels.get('match:vip-presence'); + await vipChannel.presence.enter({ + name: user.name, + badge: user.badge +}); -func (s *AlertService) publishAlert(ctx context.Context, channel *ably.RealtimeChannel, alert Alert) { - err := channel.Publish(ctx, "clinical-alert", alert) - if err != nil { - log.Println("Failed to publish alert:", err) - // Critical: alerts must be delivered. Consider fallback mechanism - } + // Display VIP/moderator presence in UI + vipChannel.presence.subscribe((event) => { + updateVIPList(event); +}); } -``` + ``` -**Best practices:** -- Use separate alert channel (don't mix with vitals) -- Include severity levels (INFO, WARNING, CRITICAL) -- Implement alert acknowledgment via annotations (`flag.v1` type) -- Debounce alerts (prevent flooding during sustained abnormal readings) -- Escalation logic (if not acknowledged within threshold, page supervisor) -- Monitor annotation summaries to track who acknowledged which alerts +This pattern gives you the best of both worlds: efficient viewer counts for the masses, and detailed presence for the subset of users who need it. + +## Priced for scale -See the [message annotations documentation](/docs/messages/annotations) for complete details. +Realtime dashboards can involve significant message volumes, especially with large viewer counts. Understanding Ably's pricing model and implementing cost optimization strategies ensures your application remains economically sustainable as it grows. -## Cost optimization strategies +### Understanding the cost model -Understanding costs and implementing optimization techniques ensures sustainable economics as you scale. +Ably charges for messages, connections, and channels. For most dashboard applications, messages are the dominant cost component. Each message published counts as one inbound message, and each delivery to a subscriber counts as one outbound message. -### Healthcare dashboard cost profile +For a dashboard with 1 publisher and 1000 subscribers publishing 1 update per second, that's 1 inbound message plus 1000 outbound messages per second—86.4 million messages per day. Understanding this fanout effect is crucial for cost planning. -**Ably pricing components:** -- Messages: Inbound (from devices) + Outbound (to dashboards) -- Connections: Each device and dashboard connection -- Channels: Active channels with subscribers -- Storage: Message history retention +### Server-side batching for burst management -**Single ICU patient scenario:** -- 1 medical device publishing vitals at 1Hz = 86,400 inbound messages/day -- 4 subscribed dashboards = 345,600 outbound messages/day -- 5 concurrent connections = negligible cost -- 1 active channel = negligible cost +[Server-side batching](/docs/messages/batch#server-side) groups messages before fanout, dramatically reducing outbound message count during high-activity periods. This is particularly valuable when your data source publishes multiple updates per second. -**Daily cost (standard rates):** ~$0.86/patient/day or ~$26/patient/month +Let's calculate the impact for a live sports dashboard with 100,000 viewers and a data source publishing 10 updates per second during an exciting match: -**Cost scales with:** -- Number of monitored patients -- Number of subscribed displays per patient -- Publishing frequency (1Hz vs 10Hz) -- Payload size (basic vitals vs waveforms) +Without batching, outbound messages are 10 updates × 100,000 subscribers = 1,000,000 messages per second. Over a 2-hour match, that's 7.2 billion messages. -### Optimization techniques +With 500ms server-side batching, messages are grouped into 2 batches per second. Each batch counts as a single outbound message per subscriber, so outbound messages become 2 batches × 100,000 subscribers = 200,000 messages per second. Over the same 2-hour match, that's 1.44 billion messages—an 80% reduction. -**1. Delta compression (80-85% bandwidth reduction)** +The key insight is that batching's effectiveness scales with your inbound message rate. If you're only publishing once per second, batching provides minimal benefit. But during excitement spikes when updates accelerate, batching automatically smooths the fanout. -``` -Before: 3KB payload × 1Hz × 4 displays = 43.2KB/s = ~3.7GB/day -After: 500B payload × 1Hz × 4 displays = 7.2KB/s = ~0.6GB/day -Savings: ~85% reduction in data transfer -``` +### Delta compression for bandwidth efficiency -**2. Message conflation (alternative cost optimization)** +[Delta compression](/docs/channels/options/deltas) reduces the size of each message rather than the count. When dashboard payloads are large but change incrementally, deltas can achieve 80-90% bandwidth reduction. -For cost-sensitive deployments where intermediate values aren't critical: +For a dashboard payload of 5KB where most fields remain unchanged between updates, the delta might be only 500 bytes. Across 100,000 subscribers, that's 500MB/s versus 50MB/s of data transfer. While this doesn't directly reduce message count, it reduces data transfer costs and improves delivery latency. -``` -10 patients, 1Hz updates, 10 displays each: -Without conflation: 10 × 86,400 inbound + 10 × 864,000 outbound = ~$260/month -With 500ms conflation (2Hz effective): ~$105/month (80% outbound reduction) -``` +### Conflation for latest-value scenarios -**Trade-off:** Conflation permanently discards intermediate messages. Even external storage integrations receive only conflated data. Use conflation only if losing intermediate vital readings is acceptable. For audit compliance, use delta compression instead. +When only the current value matters and intermediate values can be discarded, [conflation](/docs/messages#conflation) provides the most aggressive cost reduction. A price feed publishing 100 updates per second, conflated to 10 updates per second, reduces outbound messages by 90%. -**Cost comparison (10 patients, 30 days):** -- No optimization: ~$260/month -- With deltas (85% bandwidth savings): ~$120/month (preserves all data) -- With conflation (80% outbound reduction): ~$105/month (discards intermediate data) +However, conflation should be used carefully. It permanently discards intermediate messages, which may not be appropriate for all use cases. For audit trails, historical analysis, or scenarios where users might want to see the full sequence of events, use batching instead. -**3. Connection management** +### Connection management + +While messages typically dominate costs, inefficient connection management can also impact your bill. Close connections when they're not needed: -```javascript -// Close connections when dashboard not in use -window.addEventListener('beforeunload', () => { - realtime.close(); // Graceful disconnect + ```javascript + // Close connections when dashboard is not visible + document.addEventListener('visibilitychange', () => { + if (document.hidden) { + realtime.connection.close(); +} else { + realtime.connection.connect(); +} }); -// For mobile apps, disconnect in background -document.addEventListener('visibilitychange', () => { - if (document.hidden) { - realtime.connection.close(); - } else { - realtime.connection.connect(); - } + // Always close cleanly on page unload + window.addEventListener('beforeunload', () => { + realtime.close(); }); -``` + ``` -**4. Selective subscriptions** +When a client disconnects abruptly without calling close(), Ably maintains the connection state for 2 minutes to enable reconnection. Calling close() explicitly releases resources immediately. -Nurse station dashboards showing multiple patients should subscribe only to visible patients: +### Selective subscriptions + +For dashboards with multiple panels or tabs, subscribe only to the data the user is currently viewing: -```javascript -// Subscribe only to currently visible patients -function updateVisiblePatients(visiblePatientIds) { - // Unsubscribe from off-screen patients - currentSubscriptions.forEach((sub, patientId) => { - if (!visiblePatientIds.includes(patientId)) { - sub.channel.detach(); - } - }); - - // Subscribe to newly visible patients - visiblePatientIds.forEach((patientId) => { - if (!currentSubscriptions.has(patientId)) { - subscribeToPatient(patientId); - } - }); + ```javascript + // When user switches dashboard panels, manage subscriptions efficiently + let currentChannel = null; + + function switchToPanel(panelId) { + // Detach from previous panel's channel + if (currentChannel) { + currentChannel.detach(); +} + + // Attach to new panel's channel + currentChannel = realtime.channels.get(`data:${panelId}`); + currentChannel.subscribe(updatePanelDisplay); } -``` + ``` -**5. REST API for infrequent publishing** +This approach is particularly important for dashboards with many data sources where the user typically focuses on a subset at any given time. -If a device publishes less than once per minute, use REST API instead of maintaining persistent connection: +## Production-ready checklist - -```javascript -// One-off publish (no persistent connection) -const rest = new Ably.Rest({ key: process.env.ABLY_API_KEY }); -rest.channels.get('patient:7f3a9b2e:vitals').publish('vitals-update', data); -``` - +Before deploying your realtime dashboard to production, verify that you've addressed these key areas: -**Cost optimization checklist:** -- [ ] Enable delta compression on vital signs channels (preserves all data) -- [ ] Consider conflation only if intermediate values aren't needed (discards data) -- [ ] Close connections when displays inactive -- [ ] Subscribe only to visible patients (nurse stations) -- [ ] Use REST for infrequent device updates (less than 1/min) -- [ ] Batch high-frequency samples to respect 50 msg/s channel limit -- [ ] Monitor costs via Ably dashboard and validate compression ratios - -## Production readiness checklist - -Before deploying to production: - -**Security & Compliance:** -- [ ] BAA executed with Ably -- [ ] Token-based authentication implemented (no API keys in clients) -- [ ] Capability-based access control configured -- [ ] Audit logging enabled and exporting to SIEM -- [ ] De-identified patient IDs used in channel names -- [ ] Encryption verified (TLS 1.2+) - -**Reliability & Performance:** -- [ ] Connection recovery tested (simulate network failures) -- [ ] Multi-device synchronization verified -- [ ] Load tested at expected scale (update frequency × patients × displays) -- [ ] Latency measured (p50, p95, p99) -- [ ] Alerting service deployed with failover - -**Data Management:** -- [ ] External storage integration configured and tested -- [ ] Retention policies defined and enforced -- [ ] History API tested for backfill scenarios -- [ ] Data integrity checks implemented (checksums) - -**Cost Management:** -- [ ] Delta compression enabled and compression ratio validated -- [ ] Connection management logic implemented -- [ ] Costs monitored in Ably dashboard -- [ ] Budget alerts configured - -**Operational:** -- [ ] Connection state indicators in UI -- [ ] Error handling and fallback mechanisms -- [ ] Monitoring and alerting for Ably service health -- [ ] Runbook for common failure scenarios -- [ ] Load testing at 2x expected peak +Ensure that your authentication implementation uses token authentication rather than API keys in client code. Tokens should have appropriate capabilities following the principle of least privilege, and expiration times should be appropriate for your security requirements—shorter for sensitive dashboards, longer for public content. -## Next steps +Verify that your message throughput is within channel limits, with batching implemented if your data source exceeds 50 messages per second. Choose and configure the appropriate optimization strategy based on your requirements—batching for burst management, conflation for latest-value scenarios, or delta compression for large payloads. -* Read the [Delta Compression documentation](/docs/channels/options/deltas) for complete implementation details -* Read the [Message Conflation documentation](/docs/messages#conflation) for configuration options -* Explore [Ably Integrations](/docs/platform/integrations) for external storage patterns -* Review [Message Annotations](/docs/messages/annotations) for alert acknowledgment implementation -* Learn about [Token Authentication](/docs/auth/token) for production security -* Understand [Channel Options](/docs/channels/options) for optimization features -* Review [Connection State Recovery](/docs/connect/states#connection-state-recovery) for resilience patterns +Test your connection state handling to ensure users see appropriate feedback during disconnections, and verify that message backfill works correctly for extended outages. Implement graceful degradation so the dashboard remains usable even when connectivity is impaired. -**Related Guides:** -- [Data Streaming and Distribution](/docs/guides/pub-sub/data-streaming) +Set up monitoring for connection counts, message rates, and latency percentiles. Configure budget alerts in the Ably dashboard to catch unexpected usage spikes before they become costly surprises. + +## Next steps -**Get Help:** -- [Contact Ably Sales](https://ably.com/contact) for BAA and enterprise support -- [Ably Community](https://ably.com/community) for technical questions \ No newline at end of file +* Explore the [Ably Pub/Sub documentation](/docs/pub-sub) for API details. +* Learn about [delta compression](/docs/channels/options/deltas) for bandwidth optimization. +* Understand [server-side batching](/docs/messages/batch#server-side) for cost control. +* Configure [message conflation](/docs/messages#conflation) for latest-value scenarios. +* Implement [presence](/docs/presence-occupancy/presence) for collaborative dashboards. +* Set up [occupancy](/docs/presence-occupancy/occupancy) for viewer counts. +* Review [authentication best practices](/docs/auth/token) before going to production. +* Learn about [connection state recovery](/docs/connect/states#connection-state-recovery) for reliable reconnection. From b52d46eba6be271713ece7dc61aa33c9aacbac07 Mon Sep 17 00:00:00 2001 From: Steven Lindsay Date: Thu, 8 Jan 2026 17:52:52 +0000 Subject: [PATCH 05/16] wip --- .../guides/pub-sub/health-tech-dashboard.mdx | 100 ++++++++++++++++++ 1 file changed, 100 insertions(+) diff --git a/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx b/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx index 0fcaac30d3..01368e434a 100644 --- a/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx +++ b/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx @@ -644,6 +644,104 @@ For dashboards with multiple panels or tabs, subscribe only to the data the user This approach is particularly important for dashboards with many data sources where the user typically focuses on a subset at any given time. +## Outbound data streaming + +While Ably excels at delivering realtime data to dashboards, many applications also need to stream that data to external systems for analytics, long-term storage, or audit compliance. Ably's [outbound streaming](/docs/platform/integrations/streaming) and [webhook](/docs/platform/integrations/webhooks) integrations enable you to route messages to your data infrastructure without building custom pipelines. + +### Why stream data externally? + +There are several compelling reasons to integrate outbound streaming into your dashboard architecture. Realtime analytics platforms like Apache Kafka, Amazon Kinesis, or Apache Pulsar can process your data streams for aggregation, anomaly detection, or machine learning inference. Long-term storage requirements often exceed Ably's default retention period—you may need to archive dashboard data for months or years to support business intelligence, historical trending, or regulatory requirements. Audit and compliance mandates, particularly in healthcare and financial services, frequently require immutable records of all data that passed through your system, with specific retention periods and tamper-evident storage. + +### Streaming vs webhooks + +Ably offers two approaches for sending data to external systems, each suited to different use cases. + +[Outbound streaming](/docs/platform/integrations/streaming) provides continuous, high-throughput data delivery to streaming platforms. Messages flow from Ably to your chosen service with minimal latency and are delivered in order. This is the right choice for high-volume data pipelines where you're processing thousands of messages per second, feeding data warehouses or data lakes, building realtime analytics on platforms like Kafka or Kinesis, or requiring guaranteed delivery with at-least-once semantics. + + + ```text + Supported streaming destinations: + - Apache Kafka + - Amazon Kinesis + - Amazon SQS + - AMQP (RabbitMQ, etc.) + - Apache Pulsar + ``` + + +[Outbound webhooks](/docs/platform/integrations/webhooks) trigger HTTP requests to your endpoints when events occur. They're more flexible but better suited to lower-volume, event-driven scenarios. Use webhooks when you're integrating with serverless functions like AWS Lambda or Google Cloud Functions, triggering workflows in systems like Zapier or custom APIs, processing events that don't require streaming-level throughput, or integrating with services that only accept HTTP endpoints. + +### Configuring outbound streaming for dashboards + +To stream your dashboard data externally, configure an integration rule in your [Ably dashboard](https://ably.com/dashboard) or via the [Control API](/docs/platform/account/control-api). You'll specify which channels to stream using a regular expression filter, what events to capture (messages, presence, lifecycle, or occupancy), and the destination service with its connection details. + +For a patient monitoring dashboard where audit trails are required, you might stream all vitals channels to Kafka for both real-time alerting and long-term storage: + + + ```text + Channel filter: ^vitals:.* + Event types: channel.message, channel.presence + Destination: Apache Kafka + Topic: patient-vitals-audit + ``` + + +For a fan engagement dashboard where you want to analyze viewing patterns, you might stream occupancy events to Kinesis for real-time analytics: + + + ```text + Channel filter: ^match:.*:stats + Event types: channel.message, channel.occupancy + Destination: Amazon Kinesis + Stream: sports-engagement-analytics + ``` + + +### Message format considerations + +When streaming to external systems, you can choose between [enveloped](/docs/platform/integrations/streaming#enveloped) and [non-enveloped](/docs/platform/integrations/streaming#non-enveloped) message formats. + +Enveloped messages wrap the payload with metadata including the channel name, timestamp, app ID, and data center that processed the event. This is recommended for most use cases because it provides context needed for routing, filtering, and debugging in your downstream systems. + + + ```json + { + "source": "channel.message", + "appId": "aBCdEf", + "channel": "vitals:patient-7f3a9b2e", + "site": "eu-central-1-A", + "timestamp": 1123145678900, + "messages": [{ + "id": "ABcDefgHIj:1:0", + "timestamp": 1123145678900, + "data": { + "heartRate": 72, + "bloodPressure": { "systolic": 120, "diastolic": 80 }, + "spO2": 98 + } + }] + } + ``` + + +Non-enveloped messages deliver just the raw payload, which is useful when your downstream system expects a specific format or you want to minimize data transfer. However, you'll lose the channel and metadata context that enveloped messages provide. + +### Building audit trails for compliance + +For regulated industries requiring comprehensive audit trails, outbound streaming enables you to capture every message with its full context. Stream to an immutable store like Amazon S3 (via Kinesis Firehose) or a compliance-focused database. + +Key considerations for audit-compliant streaming include ensuring your streaming destination provides durable, tamper-evident storage. Include timestamps from both the original publish and the streaming delivery. Retain channel names and client IDs to establish the complete chain of custody. Consider encrypting data at rest in your downstream storage, especially for healthcare (HIPAA) or financial (SOX, PCI-DSS) compliance. + +### Realtime analytics pipelines + +For dashboards that need to derive insights from their data streams, outbound streaming enables sophisticated analytics without impacting your dashboard's performance. A typical pattern streams raw events to Kafka, processes them with a stream processor like Apache Flink or Kafka Streams, stores aggregated results in a time-series database, and optionally publishes insights back to Ably for dashboard display. + +This architecture keeps your dashboard responsive—it displays raw data directly from Ably—while your analytics pipeline processes the same data asynchronously. The results can be published back to Ably on a separate channel for dashboards that display both real-time data and computed analytics. + +### Handling conflation with external storage + +If you're using [message conflation](#message-conflation-for-latest-value-scenarios) to optimize dashboard delivery costs, be aware that conflated messages are also what gets streamed to external systems. If you need complete data for analytics or audit purposes but want conflation for dashboard delivery, consider using separate channel namespaces—one with conflation for dashboard subscribers and one without for your streaming integration. + ## Production-ready checklist Before deploying your realtime dashboard to production, verify that you've addressed these key areas: @@ -666,3 +764,5 @@ Set up monitoring for connection counts, message rates, and latency percentiles. * Set up [occupancy](/docs/presence-occupancy/occupancy) for viewer counts. * Review [authentication best practices](/docs/auth/token) before going to production. * Learn about [connection state recovery](/docs/connect/states#connection-state-recovery) for reliable reconnection. +* Configure [outbound streaming](/docs/platform/integrations/streaming) for analytics and long-term storage. +* Set up [webhooks](/docs/platform/integrations/webhooks) for event-driven integrations. From 5d26e6b7095750bed93c05c198ecdb38e75f4bca Mon Sep 17 00:00:00 2001 From: Steven Lindsay Date: Thu, 8 Jan 2026 18:04:58 +0000 Subject: [PATCH 06/16] wip --- .../docs/guides/pub-sub/health-tech-dashboard.mdx | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx b/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx index 01368e434a..8847c8505a 100644 --- a/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx +++ b/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx @@ -645,18 +645,27 @@ For dashboards with multiple panels or tabs, subscribe only to the data the user This approach is particularly important for dashboards with many data sources where the user typically focuses on a subset at any given time. ## Outbound data streaming +## Audit, analysis and storage While Ably excels at delivering realtime data to dashboards, many applications also need to stream that data to external systems for analytics, long-term storage, or audit compliance. Ably's [outbound streaming](/docs/platform/integrations/streaming) and [webhook](/docs/platform/integrations/webhooks) integrations enable you to route messages to your data infrastructure without building custom pipelines. +While Ably excels at delivering realtime data to dashboards, many applications need to do more with that data beyond displaying it. Understanding when and why you might need to route data to external systems helps you design the right architecture from the start. ### Why stream data externally? +Realtime analytics enables you to derive insights from your data streams as they flow through your system. You might want to detect anomalies in sensor readings, aggregate viewer engagement metrics, or feed machine learning models with live data. Platforms like Apache Kafka, Amazon Kinesis, or Apache Pulsar can process these streams for aggregation, pattern detection, or triggering automated responses. There are several compelling reasons to integrate outbound streaming into your dashboard architecture. Realtime analytics platforms like Apache Kafka, Amazon Kinesis, or Apache Pulsar can process your data streams for aggregation, anomaly detection, or machine learning inference. Long-term storage requirements often exceed Ably's default retention period—you may need to archive dashboard data for months or years to support business intelligence, historical trending, or regulatory requirements. Audit and compliance mandates, particularly in healthcare and financial services, frequently require immutable records of all data that passed through your system, with specific retention periods and tamper-evident storage. +Long-term storage requirements often exceed Ably's default retention period. You may need to archive dashboard data for months or years to support business intelligence dashboards, historical trending analysis, or simply to maintain records beyond what Ably retains by default. ### Streaming vs webhooks +Audit and compliance mandates, particularly in healthcare (HIPAA) and financial services (SOX, PCI-DSS), frequently require immutable records of all data that passed through your system. These regulations often specify retention periods, tamper-evident storage, and the ability to reconstruct the complete sequence of events for any given time period. Ably offers two approaches for sending data to external systems, each suited to different use cases. +### Outbound streaming integrations [Outbound streaming](/docs/platform/integrations/streaming) provides continuous, high-throughput data delivery to streaming platforms. Messages flow from Ably to your chosen service with minimal latency and are delivered in order. This is the right choice for high-volume data pipelines where you're processing thousands of messages per second, feeding data warehouses or data lakes, building realtime analytics on platforms like Kafka or Kinesis, or requiring guaranteed delivery with at-least-once semantics. +Ably's [outbound streaming](/docs/platform/integrations/streaming) and [webhook](/docs/platform/integrations/webhooks) integrations enable you to route messages to your data infrastructure without building custom pipelines. You can capture every message flowing through your dashboard channels and deliver them to external systems automatically. + +[Outbound streaming](/docs/platform/integrations/streaming) provides continuous, high-throughput data delivery to streaming platforms. Messages flow from Ably to your chosen service with minimal latency and are delivered in order. This is the right choice for high-volume data pipelines where you're processing thousands of messages per second, feeding data warehouses or data lakes, or requiring guaranteed delivery with at-least-once semantics. ```text @@ -670,12 +679,15 @@ Ably offers two approaches for sending data to external systems, each suited to [Outbound webhooks](/docs/platform/integrations/webhooks) trigger HTTP requests to your endpoints when events occur. They're more flexible but better suited to lower-volume, event-driven scenarios. Use webhooks when you're integrating with serverless functions like AWS Lambda or Google Cloud Functions, triggering workflows in systems like Zapier or custom APIs, processing events that don't require streaming-level throughput, or integrating with services that only accept HTTP endpoints. +[Outbound webhooks](/docs/platform/integrations/webhooks) trigger HTTP requests to your endpoints when events occur. They're more flexible but better suited to lower-volume, event-driven scenarios. Use webhooks when integrating with serverless functions like AWS Lambda or Google Cloud Functions, triggering workflows in systems like Zapier or custom APIs, or integrating with services that only accept HTTP endpoints. ### Configuring outbound streaming for dashboards +### Configuring integrations To stream your dashboard data externally, configure an integration rule in your [Ably dashboard](https://ably.com/dashboard) or via the [Control API](/docs/platform/account/control-api). You'll specify which channels to stream using a regular expression filter, what events to capture (messages, presence, lifecycle, or occupancy), and the destination service with its connection details. For a patient monitoring dashboard where audit trails are required, you might stream all vitals channels to Kafka for both real-time alerting and long-term storage: +For a patient monitoring dashboard where audit trails are required, you might stream all vitals channels to Kafka: ```text @@ -687,6 +699,7 @@ For a patient monitoring dashboard where audit trails are required, you might st For a fan engagement dashboard where you want to analyze viewing patterns, you might stream occupancy events to Kinesis for real-time analytics: +For a fan engagement dashboard where you want to analyze viewing patterns, you might stream occupancy events to Kinesis: ```text @@ -739,6 +752,7 @@ For dashboards that need to derive insights from their data streams, outbound st This architecture keeps your dashboard responsive—it displays raw data directly from Ably—while your analytics pipeline processes the same data asynchronously. The results can be published back to Ably on a separate channel for dashboards that display both real-time data and computed analytics. ### Handling conflation with external storage +### Conflation and external storage If you're using [message conflation](#message-conflation-for-latest-value-scenarios) to optimize dashboard delivery costs, be aware that conflated messages are also what gets streamed to external systems. If you need complete data for analytics or audit purposes but want conflation for dashboard delivery, consider using separate channel namespaces—one with conflation for dashboard subscribers and one without for your streaming integration. From cb60cd5ef1a676fd1b46e843fb5e1350a443ad1e Mon Sep 17 00:00:00 2001 From: Steven Lindsay Date: Thu, 8 Jan 2026 19:18:25 +0000 Subject: [PATCH 07/16] wip --- .../guides/pub-sub/health-tech-dashboard.mdx | 108 ++++++------------ 1 file changed, 34 insertions(+), 74 deletions(-) diff --git a/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx b/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx index 8847c8505a..3180df0212 100644 --- a/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx +++ b/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx @@ -54,7 +54,7 @@ Channels are the foundation of your dashboard architecture. They determine how d For scenarios where all subscribers receive the same data stream, a single channel provides the simplest and most cost-effective architecture. This pattern works well for public dashboards like sports scores, stock prices, or weather updates where every viewer sees identical information. - ```javascript +```javascript // Publisher: Broadcast live match statistics to all viewers const channel = realtime.channels.get('match:12345:stats'); @@ -75,7 +75,7 @@ For scenarios where all subscribers receive the same data stream, a single chann channel.subscribe('stats-update', (message) => { updateDashboard(message.data); }); - ``` +``` The single channel pattern maximizes cost efficiency because Ably's fanout delivers each published message to all subscribers with a single outbound message charge per subscriber. There's no duplication of data across channels, and the architecture remains simple to reason about. @@ -85,7 +85,7 @@ The single channel pattern maximizes cost efficiency because Ably's fanout deliv When different subscribers need access to different data streams, or when fine-grained access control is required, per-entity channels provide natural isolation. This pattern is common in healthcare, where each patient has their own channel, or in multi-tenant SaaS platforms where each customer's data must remain separate. - ```javascript +```javascript // Publisher: Medical device publishing patient vitals const patientId = 'patient-7f3a9b2e'; // De-identified patient ID const channel = realtime.channels.get(`vitals:${patientId}`); @@ -109,7 +109,7 @@ When different subscribers need access to different data streams, or when fine-g updatePatientTile(patientId, message.data); }); }); - ``` +``` Per-entity channels enable you to apply different [capabilities](/docs/auth/capabilities) to each channel, ensuring that users can only subscribe to the data they're authorized to see. This pattern also provides natural isolation—a spike in activity on one channel doesn't affect others, and you can apply different optimization rules to different channel namespaces. @@ -119,7 +119,7 @@ Per-entity channels enable you to apply different [capabilities](/docs/auth/capa When your dashboard needs to support both overview and detailed views, hierarchical channels provide a flexible solution. A dispatcher might view aggregated fleet metrics until they need to focus on a specific vehicle, at which point they drill down to detailed telemetry. - ```javascript +```javascript // Channel naming convention for hierarchical data: // fleet:overview - Aggregated metrics for all vehicles // fleet:region:europe - Regional aggregations @@ -138,7 +138,7 @@ When your dashboard needs to support both overview and detailed views, hierarchi updateVehicleDetails(message.data); }); } - ``` +``` This pattern enables bandwidth optimization because you don't stream detailed telemetry for every vehicle until the user actually needs it. You can also apply different update frequencies to different levels of the hierarchy—overview data might update every 5 seconds while detailed vehicle data updates every 100ms. @@ -154,7 +154,7 @@ Ably applies rate limits to ensure platform stability. By default, channels acce For data sources generating more than 50 updates per second, batching multiple samples into single publishes is the recommended approach: - ```javascript +```javascript // Batch high-frequency sensor readings into periodic publishes const sensorBuffer = []; const BATCH_INTERVAL_MS = 100; // Publish every 100ms @@ -183,7 +183,7 @@ For data sources generating more than 50 updates per second, batching multiple s // Flush periodically even if batch isn't full setInterval(flushBuffer, BATCH_INTERVAL_MS); - ``` +``` This approach keeps you well within rate limits while still delivering timely updates. A 100ms batching interval means subscribers see data with at most 100ms additional latency, which is imperceptible for most dashboard use cases. @@ -215,7 +215,7 @@ Conflation dramatically reduces costs when publishers send updates faster than u When dashboard payloads are large but change incrementally between updates, [delta compression](/docs/channels/options/deltas) reduces bandwidth by transmitting only the changes. Publishers continue to send complete state—the delta calculation happens automatically in Ably's infrastructure—while subscribers receive compressed updates that the SDK reconstructs into full state. - ```javascript +```javascript // Publisher: Send full dashboard state (no code changes needed) const channel = realtime.channels.get('dashboard:overview'); @@ -247,7 +247,7 @@ When dashboard payloads are large but change incrementally between updates, [del // SDK automatically reconstructs full state from deltas updateDashboard(message.data); }); - ``` +``` Delta compression is particularly effective for dashboards that display comprehensive state where most values remain stable. A 5KB dashboard payload where only a few fields change each second might compress to 500 bytes—an 90% bandwidth reduction. Across 1000 subscribers receiving updates every second, that's the difference between 5MB/s and 500KB/s of outbound data. @@ -261,7 +261,7 @@ Authentication determines who can connect to your dashboard and what they can ac For public dashboards where anyone can view the data, authentication focuses on preventing abuse rather than controlling access. You'll typically generate anonymous tokens that allow subscription but prevent publishing, ensuring viewers can watch but can't inject fake data. - ```javascript +```javascript // Server: Generate tokens for anonymous viewers const jwt = require('jsonwebtoken'); @@ -294,7 +294,7 @@ For public dashboards where anyone can view the data, authentication focuses on callback(null, token); } }); - ``` +``` The token capabilities restrict viewers to subscribing on stats channels while allowing them to both subscribe and publish on reaction channels. This enables interactive features like emoji reactions without compromising the integrity of the primary data stream. @@ -304,7 +304,7 @@ The token capabilities restrict viewers to subscribing on stats channels while a For sensitive dashboards where access must be carefully controlled, authentication ties directly into your authorization system. Each user receives a token that grants access only to the specific entities they're authorized to monitor. - ```javascript +```javascript // Server: Generate tokens based on user's authorization async function generateMonitoringToken(userId) { // Look up user's authorized patients/equipment/entities @@ -334,7 +334,7 @@ For sensitive dashboards where access must be carefully controlled, authenticati return jwt.sign(claims, '{{ API_KEY_SECRET }}', { header }); } - ``` +``` The shorter token expiration for sensitive dashboards ensures that if a user's access is revoked, their existing token expires quickly. The capabilities are built dynamically based on the user's current authorizations, so changes in your authorization system are reflected within the token expiration period. @@ -354,7 +354,7 @@ Network disruptions are inevitable—mobile devices lose signal, users switch ne Ably's SDKs automatically handle reconnection and [connection state recovery](/docs/connect/states#connection-state-recovery). When a connection drops, the SDK will automatically attempt to reconnect using exponential backoff, trying multiple data centers if necessary. Your application should monitor connection state to provide appropriate user feedback: - ```javascript +```javascript // Monitor connection state for UI feedback realtime.connection.on('connected', () => { hideConnectionWarning(); @@ -373,7 +373,7 @@ Ably's SDKs automatically handle reconnection and [connection state recovery](/d realtime.connection.on('failed', () => { showConnectionError('Connection failed - please refresh'); }); - ``` +``` The `disconnected` state indicates a temporary loss of connection where the SDK is actively trying to reconnect. The `suspended` state indicates that reconnection attempts have been unsuccessful for an extended period, but the SDK will continue trying. The `failed` state indicates that the connection cannot be recovered and manual intervention is required. @@ -385,7 +385,7 @@ Ably maintains a 2-minute message buffer for each connection. When a client reco For longer disconnections, or when you need to backfill historical data, use the [history API](/docs/storage-history/history) to retrieve missed messages: - ```javascript +```javascript // Track the timestamp of the last received message let lastReceivedTimestamp = Date.now(); @@ -417,7 +417,7 @@ For longer disconnections, or when you need to backfill historical data, use the } wasConnectedBefore = true; }); - ``` +``` For critical monitoring dashboards, this message continuity is essential. A nurse checking vital signs needs to know that the data displayed is current and complete, not missing updates from a brief network interruption. @@ -427,7 +427,7 @@ For critical monitoring dashboards, this message continuity is essential. A nurs Even with automatic reconnection, there will be periods where your dashboard doesn't have current data. Design your UI to communicate this clearly to users, indicating both the age of the displayed data and the connection status: - ```javascript +```javascript // Track data freshness for UI indication let lastUpdateTime = Date.now(); const STALE_THRESHOLD_MS = 5000; // Consider data stale after 5 seconds @@ -448,7 +448,7 @@ Even with automatic reconnection, there will be periods where your dashboard doe // Display something like "Last updated 15 seconds ago" } }, 1000); - ``` +``` For fan engagement dashboards, showing slightly stale data with a "last updated" indicator is usually acceptable. For critical monitoring dashboards, you might want more aggressive staleness thresholds and clearer visual warnings when data isn't current. @@ -462,7 +462,7 @@ For fan engagement dashboards, showing slightly stale data with a "last updated" Presence is useful for collaborative dashboards where operators need to coordinate, or where seeing who else is viewing the same data provides context. Each client can enter a channel's presence set with associated data, and all presence subscribers receive notifications when members enter, leave, or update their data. - ```javascript +```javascript // Enter presence when opening the dashboard const channel = realtime.channels.get('ops:control-room'); @@ -502,7 +502,7 @@ Presence is useful for collaborative dashboards where operators need to coordina station: 'Control Room A', status: 'handling incident' }); - ``` +``` Presence is powerful but expensive at scale. Every enter, leave, and update event generates messages delivered to all presence subscribers. For a channel with 1000 viewers all subscribed to presence, a single user joining triggers 1000 outbound messages. If users are frequently joining and leaving, this can quickly dominate your message costs. @@ -512,7 +512,7 @@ Presence is powerful but expensive at scale. Every enter, leave, and update even For fan engagement dashboards where you want to show viewer counts without needing to know individual identities, occupancy provides an efficient alternative. Occupancy gives you aggregate metrics about channel connections without the per-event overhead of full presence. - ```javascript +```javascript // Enable occupancy updates on channel attachment const channel = realtime.channels.get('match:12345:stats', { params: { occupancy: 'metrics' } @@ -528,7 +528,7 @@ For fan engagement dashboards where you want to show viewer counts without needi // metrics.publishers - connections with publish capability // metrics.presenceMembers - members in presence set }); - ``` +``` Occupancy updates are debounced and delivered efficiently, making them suitable for channels with thousands or millions of viewers. The overhead is minimal compared to full presence, and you still get the "15,234 people watching" social proof that enhances fan engagement experiences. @@ -538,7 +538,7 @@ Occupancy updates are debounced and delivered efficiently, making them suitable If you need presence-like functionality with thousands of users, consider a hybrid approach. Use occupancy for the aggregate viewer count that everyone sees, but enable full presence only for specific user groups who need to see individual identities. - ```javascript +```javascript // Regular viewers: occupancy only, no presence overhead const statsChannel = realtime.channels.get('match:stats', { params: { occupancy: 'metrics' } @@ -557,7 +557,7 @@ If you need presence-like functionality with thousands of users, consider a hybr updateVIPList(event); }); } - ``` +``` This pattern gives you the best of both worlds: efficient viewer counts for the masses, and detailed presence for the subset of users who need it. @@ -601,7 +601,7 @@ However, conflation should be used carefully. It permanently discards intermedia While messages typically dominate costs, inefficient connection management can also impact your bill. Close connections when they're not needed: - ```javascript +```javascript // Close connections when dashboard is not visible document.addEventListener('visibilitychange', () => { if (document.hidden) { @@ -615,7 +615,7 @@ While messages typically dominate costs, inefficient connection management can a window.addEventListener('beforeunload', () => { realtime.close(); }); - ``` +``` When a client disconnects abruptly without calling close(), Ably maintains the connection state for 2 minutes to enable reconnection. Calling close() explicitly releases resources immediately. @@ -625,7 +625,7 @@ When a client disconnects abruptly without calling close(), Ably maintains the c For dashboards with multiple panels or tabs, subscribe only to the data the user is currently viewing: - ```javascript +```javascript // When user switches dashboard panels, manage subscriptions efficiently let currentChannel = null; @@ -639,55 +639,28 @@ For dashboards with multiple panels or tabs, subscribe only to the data the user currentChannel = realtime.channels.get(`data:${panelId}`); currentChannel.subscribe(updatePanelDisplay); } - ``` +``` This approach is particularly important for dashboards with many data sources where the user typically focuses on a subset at any given time. -## Outbound data streaming ## Audit, analysis and storage -While Ably excels at delivering realtime data to dashboards, many applications also need to stream that data to external systems for analytics, long-term storage, or audit compliance. Ably's [outbound streaming](/docs/platform/integrations/streaming) and [webhook](/docs/platform/integrations/webhooks) integrations enable you to route messages to your data infrastructure without building custom pipelines. While Ably excels at delivering realtime data to dashboards, many applications need to do more with that data beyond displaying it. Understanding when and why you might need to route data to external systems helps you design the right architecture from the start. -### Why stream data externally? -Realtime analytics enables you to derive insights from your data streams as they flow through your system. You might want to detect anomalies in sensor readings, aggregate viewer engagement metrics, or feed machine learning models with live data. Platforms like Apache Kafka, Amazon Kinesis, or Apache Pulsar can process these streams for aggregation, pattern detection, or triggering automated responses. +Often data displayed on dashboards needs to be archived for long-term storage, compliance, or historical analysis. Audit trails are essential in regulated industries like healthcare, where you must retain immutable records of all data that passed through your system. Business intelligence teams may want to analyze historical trends, user behavior, or system performance over time. -There are several compelling reasons to integrate outbound streaming into your dashboard architecture. Realtime analytics platforms like Apache Kafka, Amazon Kinesis, or Apache Pulsar can process your data streams for aggregation, anomaly detection, or machine learning inference. Long-term storage requirements often exceed Ably's default retention period—you may need to archive dashboard data for months or years to support business intelligence, historical trending, or regulatory requirements. Audit and compliance mandates, particularly in healthcare and financial services, frequently require immutable records of all data that passed through your system, with specific retention periods and tamper-evident storage. -Long-term storage requirements often exceed Ably's default retention period. You may need to archive dashboard data for months or years to support business intelligence dashboards, historical trending analysis, or simply to maintain records beyond what Ably retains by default. +To facilitate this, Ably provides [outbound integrations](/docs/platform/integrations) that let you stream data to external systems like Apache Kafka, Amazon Kinesis, or custom HTTP endpoints. This enables you to build comprehensive data pipelines without reinventing the wheel. -### Streaming vs webhooks -Audit and compliance mandates, particularly in healthcare (HIPAA) and financial services (SOX, PCI-DSS), frequently require immutable records of all data that passed through your system. These regulations often specify retention periods, tamper-evident storage, and the ability to reconstruct the complete sequence of events for any given time period. - -Ably offers two approaches for sending data to external systems, each suited to different use cases. ### Outbound streaming integrations [Outbound streaming](/docs/platform/integrations/streaming) provides continuous, high-throughput data delivery to streaming platforms. Messages flow from Ably to your chosen service with minimal latency and are delivered in order. This is the right choice for high-volume data pipelines where you're processing thousands of messages per second, feeding data warehouses or data lakes, building realtime analytics on platforms like Kafka or Kinesis, or requiring guaranteed delivery with at-least-once semantics. -Ably's [outbound streaming](/docs/platform/integrations/streaming) and [webhook](/docs/platform/integrations/webhooks) integrations enable you to route messages to your data infrastructure without building custom pipelines. You can capture every message flowing through your dashboard channels and deliver them to external systems automatically. - -[Outbound streaming](/docs/platform/integrations/streaming) provides continuous, high-throughput data delivery to streaming platforms. Messages flow from Ably to your chosen service with minimal latency and are delivered in order. This is the right choice for high-volume data pipelines where you're processing thousands of messages per second, feeding data warehouses or data lakes, or requiring guaranteed delivery with at-least-once semantics. - - - ```text - Supported streaming destinations: - - Apache Kafka - - Amazon Kinesis - - Amazon SQS - - AMQP (RabbitMQ, etc.) - - Apache Pulsar - ``` - - -[Outbound webhooks](/docs/platform/integrations/webhooks) trigger HTTP requests to your endpoints when events occur. They're more flexible but better suited to lower-volume, event-driven scenarios. Use webhooks when you're integrating with serverless functions like AWS Lambda or Google Cloud Functions, triggering workflows in systems like Zapier or custom APIs, processing events that don't require streaming-level throughput, or integrating with services that only accept HTTP endpoints. -[Outbound webhooks](/docs/platform/integrations/webhooks) trigger HTTP requests to your endpoints when events occur. They're more flexible but better suited to lower-volume, event-driven scenarios. Use webhooks when integrating with serverless functions like AWS Lambda or Google Cloud Functions, triggering workflows in systems like Zapier or custom APIs, or integrating with services that only accept HTTP endpoints. ### Configuring outbound streaming for dashboards -### Configuring integrations To stream your dashboard data externally, configure an integration rule in your [Ably dashboard](https://ably.com/dashboard) or via the [Control API](/docs/platform/account/control-api). You'll specify which channels to stream using a regular expression filter, what events to capture (messages, presence, lifecycle, or occupancy), and the destination service with its connection details. For a patient monitoring dashboard where audit trails are required, you might stream all vitals channels to Kafka for both real-time alerting and long-term storage: -For a patient monitoring dashboard where audit trails are required, you might stream all vitals channels to Kafka: ```text @@ -698,23 +671,11 @@ For a patient monitoring dashboard where audit trails are required, you might st ``` -For a fan engagement dashboard where you want to analyze viewing patterns, you might stream occupancy events to Kinesis for real-time analytics: -For a fan engagement dashboard where you want to analyze viewing patterns, you might stream occupancy events to Kinesis: - - - ```text - Channel filter: ^match:.*:stats - Event types: channel.message, channel.occupancy - Destination: Amazon Kinesis - Stream: sports-engagement-analytics - ``` - - ### Message format considerations When streaming to external systems, you can choose between [enveloped](/docs/platform/integrations/streaming#enveloped) and [non-enveloped](/docs/platform/integrations/streaming#non-enveloped) message formats. -Enveloped messages wrap the payload with metadata including the channel name, timestamp, app ID, and data center that processed the event. This is recommended for most use cases because it provides context needed for routing, filtering, and debugging in your downstream systems. +Enveloped messages wrap the payload with metadata including the channel name, timestamp, app ID, and data center that processed the event. This is recommended and enabled by default because it provides context needed for routing, filtering, and debugging in your downstream systems. ```json @@ -751,10 +712,9 @@ For dashboards that need to derive insights from their data streams, outbound st This architecture keeps your dashboard responsive—it displays raw data directly from Ably—while your analytics pipeline processes the same data asynchronously. The results can be published back to Ably on a separate channel for dashboards that display both real-time data and computed analytics. -### Handling conflation with external storage ### Conflation and external storage -If you're using [message conflation](#message-conflation-for-latest-value-scenarios) to optimize dashboard delivery costs, be aware that conflated messages are also what gets streamed to external systems. If you need complete data for analytics or audit purposes but want conflation for dashboard delivery, consider using separate channel namespaces—one with conflation for dashboard subscribers and one without for your streaming integration. +If you're using [message conflation](#message-conflation-for-latest-value-scenarios) to optimize dashboard delivery costs, be aware that conflated messages are also what gets streamed to external systems. If you need complete data for analytics or audit purposes but want conflation for dashboard delivery, you may have to consider other publish patterns. ## Production-ready checklist From 2e844b1d495d340af44ffcf3867ef70f3b38071d Mon Sep 17 00:00:00 2001 From: Steven Lindsay Date: Fri, 9 Jan 2026 10:41:16 +0000 Subject: [PATCH 08/16] wip --- src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx b/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx index 3180df0212..ab551469b6 100644 --- a/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx +++ b/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx @@ -208,7 +208,7 @@ Configure conflation through [channel rules](/docs/channels#rules) in your dashb Message conflation permanently discards intermediate messages. Even messages streamed to external storage via Ably integrations will only contain conflated data. Use conflation only when displaying the latest value is sufficient and historical accuracy isn't required. For audit trails or time-series analysis, use server-side batching or delta compression instead, which preserve all messages. -Conflation dramatically reduces costs when publishers send updates faster than users can perceive. If a price feed publishes 100 updates per second but your dashboard refreshes at 10 FPS, you're wasting 90% of your message budget on updates users never see. With 100ms conflation, you reduce outbound messages by 10x while showing users the most current data available. +Conflation dramatically reduces costs when publishers send updates faster than users can perceive. If a price feed publishes 10 updates per second but your dashboard refreshes only each second, you're wasting 90% of your message budget on updates users never see. With 100ms conflation, you reduce outbound messages by 10x while showing users the most current data available. ### Delta compression for large payloads @@ -376,7 +376,7 @@ Ably's SDKs automatically handle reconnection and [connection state recovery](/d ``` -The `disconnected` state indicates a temporary loss of connection where the SDK is actively trying to reconnect. The `suspended` state indicates that reconnection attempts have been unsuccessful for an extended period, but the SDK will continue trying. The `failed` state indicates that the connection cannot be recovered and manual intervention is required. +The `disconnected` state indicates a temporary loss of connection where the SDK is actively trying to reconnect. The `suspended` state indicates that reconnection attempts have been unsuccessful for an extended period, but the SDK will continue trying in 30 second intervals. The `failed` state indicates that the connection cannot be recovered and manual intervention is required. ### Message continuity after reconnection From 37680a2b70081af7ac858d83afdf92724caa9996 Mon Sep 17 00:00:00 2001 From: Steven Lindsay Date: Fri, 9 Jan 2026 11:20:38 +0000 Subject: [PATCH 09/16] refactor some text blocks to include lists/bullets --- .gitignore | 1 + src/data/nav/pubsub.ts | 9 ++ .../guides/pub-sub/health-tech-dashboard.mdx | 99 ++++++++++++++++--- 3 files changed, 94 insertions(+), 15 deletions(-) diff --git a/.gitignore b/.gitignore index 11c7f8ca85..ee5b381a1b 100644 --- a/.gitignore +++ b/.gitignore @@ -23,3 +23,4 @@ src/gatsby-types.d.ts .idea/* **/*.swp .claude +/.junio/ diff --git a/src/data/nav/pubsub.ts b/src/data/nav/pubsub.ts index 649d00dd04..b09482cc6a 100644 --- a/src/data/nav/pubsub.ts +++ b/src/data/nav/pubsub.ts @@ -325,6 +325,15 @@ export default { }, ], }, + { + name: 'Guides', + pages: [ + { + name: 'Dashboards and visualizations', + link: '/docs/guides/pub-sub/health-tech-dashboard', + } + ], + }, ], api: [ { diff --git a/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx b/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx index ab551469b6..18c78169a2 100644 --- a/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx +++ b/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx @@ -33,15 +33,39 @@ The key architectural decision is understanding which category your dashboard fa ### Fan engagement dashboards: Streaming to millions -Fan engagement dashboards prioritize broadcasting data to massive audiences where the experience is shared. These include live sports statistics streaming score updates, player stats, and match events to millions of fans. Stock tickers and market data platforms distribute price updates to trading platforms and financial apps. Live event platforms show concert statistics, voting results, and audience participation metrics. Gaming leaderboards deliver real-time rankings and achievement updates to large player bases. +Fan engagement dashboards prioritize broadcasting data to massive audiences where the experience is shared. Common examples include: -In these scenarios, the relationship is typically one publisher broadcasting to many subscribers. High message throughput with sub-second latency is usually sufficient, and eventual consistency is acceptable—not every intermediate value needs to be delivered. Cost optimization becomes critical due to message fanout, and users expect engaging, fast-updating displays rather than mission-critical accuracy. +* **Live sports statistics** streaming score updates, player stats, and match events to millions of fans +* **Stock tickers and market data platforms** distributing price updates to trading platforms and financial apps +* **Live event platforms** showing concert statistics, voting results, and audience participation metrics +* **Gaming leaderboards** delivering real-time rankings and achievement updates to large player bases + +In these scenarios: + +* The relationship is typically one publisher broadcasting to many subscribers +* High message throughput with sub-second latency is usually sufficient +* Eventual consistency is acceptable—not every intermediate value needs to be delivered +* Cost optimization becomes critical due to message fanout +* Users expect engaging, fast-updating displays rather than mission-critical accuracy ### Critical monitoring dashboards: Every message matters -Critical monitoring dashboards prioritize guaranteed delivery, data integrity, and low latency for operational decisions where lives or significant assets may depend on the data. Patient monitoring systems stream vital signs from ICU equipment to nurse stations and mobile devices. Industrial control systems track equipment telemetry, safety alerts, and process monitoring. Fleet management platforms show vehicle locations, driver alerts, and cargo conditions. Energy grid monitoring displays power generation, consumption, and grid stability metrics. +Critical monitoring dashboards prioritize guaranteed delivery, data integrity, and low latency for operational decisions where lives or significant assets may depend on the data. Common examples include: + +* **Patient monitoring systems** streaming vital signs from ICU equipment to nurse stations and mobile devices +* **Industrial control systems** tracking equipment telemetry, safety alerts, and process monitoring +* **Fleet management platforms** showing vehicle locations, driver alerts, and cargo conditions +* **Energy grid monitoring** displaying power generation, consumption, and grid stability metrics + +These scenarios have distinct requirements: -These scenarios often involve 1-to-1 or 1-to-few relationships, where one data source streams to specific authorized viewers. Every message must be delivered with guaranteed ordering, and sub-100ms latency may be required. Connection recovery and message continuity are essential, and audit trails and compliance requirements often apply. For healthcare applications, Ably is HIPAA-compliant and offers Business Associate Agreements (BAAs) for customers handling Protected Health Information (PHI). +* Often involve 1-to-1 or 1-to-few relationships, where one data source streams to specific authorized viewers +* Every message must be delivered with guaranteed ordering +* Sub-100ms latency may be required +* Connection recovery and message continuity are essential +* Audit trails and compliance requirements often apply + +For healthcare applications, Ably is HIPAA-compliant and offers Business Associate Agreements (BAAs) for customers handling Protected Health Information (PHI). Understanding which category your dashboard falls into—or whether it combines elements of both—is fundamental to making the right architectural decisions throughout this guide. @@ -339,11 +363,22 @@ For sensitive dashboards where access must be carefully controlled, authenticati The shorter token expiration for sensitive dashboards ensures that if a user's access is revoked, their existing token expires quickly. The capabilities are built dynamically based on the user's current authorizations, so changes in your authorization system are reflected within the token expiration period. -For production deployments, you should never expose API keys in client-side code. Always use token authentication, with tokens generated by your server based on the user's authenticated session. Apply the principle of least privilege, granting only the capabilities each user actually needs. For compliance scenarios, use [integration rules](/docs/platform/integrations) to log channel activity for audit trails. +For production deployments: + +* Never expose API keys in client-side code +* Always use token authentication, with tokens generated by your server based on the user's authenticated session +* Apply the principle of least privilege, granting only the capabilities each user actually needs +* For compliance scenarios, use [integration rules](/docs/platform/integrations) to log channel activity for audit trails ### HIPAA compliance for healthcare dashboards -For healthcare applications handling Protected Health Information (PHI), Ably provides the infrastructure needed to build HIPAA-compliant applications. All data is encrypted in transit via TLS 1.2+ and at rest with AES-256. Ably offers Business Associate Agreements (BAAs) for enterprise customers, and comprehensive audit logging is available through integration rules. When building patient monitoring dashboards, use de-identified patient IDs in channel names rather than actual patient identifiers, and ensure your token generation ties into your existing healthcare authorization systems. +For healthcare applications handling Protected Health Information (PHI), Ably provides the infrastructure needed to build HIPAA-compliant applications: + +* All data is encrypted in transit via TLS 1.2+ and at rest with AES-256 +* Business Associate Agreements (BAAs) are available for enterprise customers +* Comprehensive audit logging is available through integration rules + +When building patient monitoring dashboards, use de-identified patient IDs in channel names rather than actual patient identifiers, and ensure your token generation ties into your existing healthcare authorization systems. ## Handling network disruption @@ -376,7 +411,11 @@ Ably's SDKs automatically handle reconnection and [connection state recovery](/d ``` -The `disconnected` state indicates a temporary loss of connection where the SDK is actively trying to reconnect. The `suspended` state indicates that reconnection attempts have been unsuccessful for an extended period, but the SDK will continue trying in 30 second intervals. The `failed` state indicates that the connection cannot be recovered and manual intervention is required. +Understanding each connection state helps you provide appropriate user feedback: + +* **`disconnected`** — A temporary loss of connection where the SDK is actively trying to reconnect +* **`suspended`** — Reconnection attempts have been unsuccessful for an extended period, but the SDK will continue trying in 30-second intervals +* **`failed`** — The connection cannot be recovered and manual intervention is required ### Message continuity after reconnection @@ -648,13 +687,22 @@ This approach is particularly important for dashboards with many data sources wh While Ably excels at delivering realtime data to dashboards, many applications need to do more with that data beyond displaying it. Understanding when and why you might need to route data to external systems helps you design the right architecture from the start. -Often data displayed on dashboards needs to be archived for long-term storage, compliance, or historical analysis. Audit trails are essential in regulated industries like healthcare, where you must retain immutable records of all data that passed through your system. Business intelligence teams may want to analyze historical trends, user behavior, or system performance over time. +Common reasons for routing data externally include: + +* **Long-term storage** — Archiving dashboard data for compliance or historical analysis +* **Audit trails** — Retaining immutable records of all data, essential in regulated industries like healthcare +* **Business intelligence** — Analyzing historical trends, user behavior, or system performance over time To facilitate this, Ably provides [outbound integrations](/docs/platform/integrations) that let you stream data to external systems like Apache Kafka, Amazon Kinesis, or custom HTTP endpoints. This enables you to build comprehensive data pipelines without reinventing the wheel. ### Outbound streaming integrations -[Outbound streaming](/docs/platform/integrations/streaming) provides continuous, high-throughput data delivery to streaming platforms. Messages flow from Ably to your chosen service with minimal latency and are delivered in order. This is the right choice for high-volume data pipelines where you're processing thousands of messages per second, feeding data warehouses or data lakes, building realtime analytics on platforms like Kafka or Kinesis, or requiring guaranteed delivery with at-least-once semantics. +[Outbound streaming](/docs/platform/integrations/streaming) provides continuous, high-throughput data delivery to streaming platforms. Messages flow from Ably to your chosen service with minimal latency and are delivered in order. This is the right choice when: + +* Processing thousands of messages per second in high-volume data pipelines +* Feeding data warehouses or data lakes +* Building realtime analytics on platforms like Kafka or Kinesis +* Requiring guaranteed delivery with at-least-once semantics ### Configuring outbound streaming for dashboards @@ -704,11 +752,21 @@ Non-enveloped messages deliver just the raw payload, which is useful when your d For regulated industries requiring comprehensive audit trails, outbound streaming enables you to capture every message with its full context. Stream to an immutable store like Amazon S3 (via Kinesis Firehose) or a compliance-focused database. -Key considerations for audit-compliant streaming include ensuring your streaming destination provides durable, tamper-evident storage. Include timestamps from both the original publish and the streaming delivery. Retain channel names and client IDs to establish the complete chain of custody. Consider encrypting data at rest in your downstream storage, especially for healthcare (HIPAA) or financial (SOX, PCI-DSS) compliance. +Key considerations for audit-compliant streaming: + +* Ensure your streaming destination provides durable, tamper-evident storage +* Include timestamps from both the original publish and the streaming delivery +* Retain channel names and client IDs to establish the complete chain of custody +* Consider encrypting data at rest in your downstream storage, especially for healthcare (HIPAA) or financial (SOX, PCI-DSS) compliance ### Realtime analytics pipelines -For dashboards that need to derive insights from their data streams, outbound streaming enables sophisticated analytics without impacting your dashboard's performance. A typical pattern streams raw events to Kafka, processes them with a stream processor like Apache Flink or Kafka Streams, stores aggregated results in a time-series database, and optionally publishes insights back to Ably for dashboard display. +For dashboards that need to derive insights from their data streams, outbound streaming enables sophisticated analytics without impacting your dashboard's performance. A typical pattern involves: + +1. Streaming raw events to Kafka +2. Processing them with a stream processor like Apache Flink or Kafka Streams +3. Storing aggregated results in a time-series database +4. Optionally publishing insights back to Ably for dashboard display This architecture keeps your dashboard responsive—it displays raw data directly from Ably—while your analytics pipeline processes the same data asynchronously. The results can be published back to Ably on a separate channel for dashboards that display both real-time data and computed analytics. @@ -720,13 +778,24 @@ If you're using [message conflation](#message-conflation-for-latest-value-scenar Before deploying your realtime dashboard to production, verify that you've addressed these key areas: -Ensure that your authentication implementation uses token authentication rather than API keys in client code. Tokens should have appropriate capabilities following the principle of least privilege, and expiration times should be appropriate for your security requirements—shorter for sensitive dashboards, longer for public content. +**Authentication:** +* Use token authentication rather than API keys in client code +* Tokens have appropriate capabilities following the principle of least privilege +* Expiration times are appropriate for your security requirements—shorter for sensitive dashboards, longer for public content -Verify that your message throughput is within channel limits, with batching implemented if your data source exceeds 50 messages per second. Choose and configure the appropriate optimization strategy based on your requirements—batching for burst management, conflation for latest-value scenarios, or delta compression for large payloads. +**Message throughput:** +* Verify throughput is within channel limits (50 messages/second by default) +* Implement batching if your data source exceeds rate limits +* Choose and configure the appropriate optimization strategy—batching for burst management, conflation for latest-value scenarios, or delta compression for large payloads -Test your connection state handling to ensure users see appropriate feedback during disconnections, and verify that message backfill works correctly for extended outages. Implement graceful degradation so the dashboard remains usable even when connectivity is impaired. +**Connection handling:** +* Test connection state handling to ensure users see appropriate feedback during disconnections +* Verify message backfill works correctly for extended outages +* Implement graceful degradation so the dashboard remains usable even when connectivity is impaired -Set up monitoring for connection counts, message rates, and latency percentiles. Configure budget alerts in the Ably dashboard to catch unexpected usage spikes before they become costly surprises. +**Monitoring:** +* Set up monitoring for connection counts, message rates, and latency percentiles +* Configure budget alerts in the Ably dashboard to catch unexpected usage spikes before they become costly surprises ## Next steps From 9d12abf26babd4aaaf3c66be77635e224f7f5b44 Mon Sep 17 00:00:00 2001 From: Steven Lindsay Date: Fri, 9 Jan 2026 13:54:35 +0000 Subject: [PATCH 10/16] update health tech guide with channel rules and optimization strategies --- .../guides/pub-sub/health-tech-dashboard.mdx | 127 ++++++++++++------ 1 file changed, 88 insertions(+), 39 deletions(-) diff --git a/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx b/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx index 18c78169a2..3f73179490 100644 --- a/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx +++ b/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx @@ -222,6 +222,20 @@ The key benefit of server-side batching is that it reduces billable outbound mes Unlike message conflation, server-side batching preserves all messages. Every update is delivered, just grouped together for efficiency. This makes it suitable for scenarios where you need complete data but want to smooth out traffic spikes. +### Configuring channel rules + +Both server-side batching and message conflation are configured through [channel rules](/docs/channels#rules) in your Ably dashboard. Channel rules allow you to apply optimizations to specific channels or entire namespaces using regex pattern matching, making it easy to configure behavior across related channels without code changes. + +To configure a channel rule: + +1. Navigate to your app settings in the [Ably dashboard](https://ably.com/dashboard) and select the **Channel Rules** tab. +2. Create a new rule with a channel name pattern (e.g., `^match:.*:stats` to match all match stats channels, or `^vitals:.*` for all patient vitals). +3. Enable the desired optimization—server-side batching, message conflation, or both. +4. For batching, set the interval (100ms for responsive dashboards, up to 1000ms for maximum cost savings). +5. For conflation, configure the conflation interval and key pattern to determine which messages are merged. + +Using namespace patterns means you can apply different rules to different parts of your application. For example, you might enable conflation on price ticker channels (`^prices:.*`) while using batching on event feeds (`^events:.*`) where complete data is required. + ### Message conflation for latest-value scenarios For dashboards where only the current value matters—stock prices, sensor readings, vehicle positions—[message conflation](/docs/messages#conflation) delivers only the most recent value within each time window. @@ -276,13 +290,41 @@ When dashboard payloads are large but change incrementally between updates, [del Delta compression is particularly effective for dashboards that display comprehensive state where most values remain stable. A 5KB dashboard payload where only a few fields change each second might compress to 500 bytes—an 90% bandwidth reduction. Across 1000 subscribers receiving updates every second, that's the difference between 5MB/s and 500KB/s of outbound data. +#### Pairing with persist last message + +For state-based dashboards using delta compression, the [persist last message](/docs/storage-history/storage#persist-last-message) channel rule provides instant state recovery for new subscribers. When enabled, Ably stores the most recent message published to a channel for up to a year. New clients can then attach with `rewind=1` to immediately receive the last published state—even if the channel has been inactive. + +This combination is powerful: a new subscriber receives the complete current state immediately on connection, and subsequent updates arrive as efficient deltas. Without persist last message, a new subscriber would need to wait for the next publish to receive any data. + + +```javascript + // Subscriber: Get the latest state immediately on connection + const channel = realtime.channels.get('dashboard:overview', { + params: { + delta: 'vcdiff', + rewind: '1' // Retrieve the last message on attach +} +}); + + channel.subscribe('state-update', (message) => { + // First message received will be the persisted state + // Subsequent messages will be deltas against that baseline + updateDashboard(message.data); +}); +``` + + +Enable persist last message via [channel rules](/docs/channels#rules) in your Ably dashboard. Note that this stores a single message—if you publish multiple messages in a single `publish()` call, all of them are stored and returned on rewind. For state-based dashboards, publish complete state snapshots as single messages rather than partial updates. + ## Authentication Authentication determines who can connect to your dashboard and what they can access. The approach differs significantly between fan engagement and critical monitoring scenarios, reflecting the different security requirements of each. -### Fan engagement: Anonymous and simple access +### Fan engagement: Simple and shared access -For public dashboards where anyone can view the data, authentication focuses on preventing abuse rather than controlling access. You'll typically generate anonymous tokens that allow subscription but prevent publishing, ensuring viewers can watch but can't inject fake data. +For public dashboards where anyone can view the data, authentication focuses on preventing abuse rather than fine-grained access control. You'll typically generate tokens with the same or similar access patterns for all clients. + +These tokens might allow broad subscription to many channels, such as stats data for any in-progress match, but likely prevent publishing and other actions, ensuring viewers can watch but can't inject fake data or interact with other clients. ```javascript @@ -302,10 +344,10 @@ For public dashboards where anyone can view the data, authentication focuses on iat: currentTime, exp: currentTime + 3600, // 1 hour expiration 'x-ably-capability': JSON.stringify({ - 'match:*:stats': ['subscribe'], // Can only subscribe, not publish - 'match:*:reactions': ['subscribe', 'publish'] // Can send reactions + 'match:*:stats': ['subscribe'], // Restricted access, but to any channel in this namespace + 'match:*:reactions': ['subscribe', 'annotation-publish'] // Allow publishing reactions }), - 'x-ably-clientId': `viewer-${crypto.randomUUID()}` + 'x-ably-clientId': `viewer-123` }; return jwt.sign(claims, '{{ API_KEY_SECRET }}', { header }); @@ -321,7 +363,9 @@ For public dashboards where anyone can view the data, authentication focuses on ``` -The token capabilities restrict viewers to subscribing on stats channels while allowing them to both subscribe and publish on reaction channels. This enables interactive features like emoji reactions without compromising the integrity of the primary data stream. +The token capabilities restrict viewers to subscribing on stats channels while allowing them to both subscribe and annotate existing messages on reaction channels. This enables interactive features like emoji reactions without compromising the integrity of the primary data stream – for example, a trusted source could publish match updates or scores, to which viewers can react but not modify. + +Overall, the access patterns are likely to be broad on channels but limited in terms of actions, focusing on read-only access with controlled interactivity. ### Critical monitoring: Strict access control @@ -337,8 +381,8 @@ For sensitive dashboards where access must be carefully controlled, authenticati // Build capability for only their authorized channels const capability = {}; authorizedEntities.forEach(entityId => { - capability[`vitals:${entityId}`] = ['subscribe']; - capability[`alerts:${entityId}`] = ['subscribe', 'publish']; // Can acknowledge alerts + capability[`vitals:${entityId}`] = ['subscribe', 'history']; + capability[`alerts:${entityId}`] = ['subscribe', 'publish', 'history']; // Can acknowledge alerts }); const header = { @@ -363,11 +407,15 @@ For sensitive dashboards where access must be carefully controlled, authenticati The shorter token expiration for sensitive dashboards ensures that if a user's access is revoked, their existing token expires quickly. The capabilities are built dynamically based on the user's current authorizations, so changes in your authorization system are reflected within the token expiration period. +Channels are also tightly scoped to individual entities, and broad access patterns are avoided. A nurse can only subscribe to the vitals and alerts channels for their assigned patients, ensuring strict data isolation. + +When granting clients access to a namespace, if a new channel was created, clients could automatically subscribe without re-authenticating. In the critical monitoring scenario, namespaces should be applied cautiously and new channels should generally require updated tokens to ensure access remains tightly controlled. + For production deployments: * Never expose API keys in client-side code * Always use token authentication, with tokens generated by your server based on the user's authenticated session -* Apply the principle of least privilege, granting only the capabilities each user actually needs +* Apply the principle of least privilege, granting only the capabilities each user actually needs and on a per-channel basis * For compliance scenarios, use [integration rules](/docs/platform/integrations) to log channel activity for audit trails ### HIPAA compliance for healthcare dashboards @@ -377,6 +425,7 @@ For healthcare applications handling Protected Health Information (PHI), Ably pr * All data is encrypted in transit via TLS 1.2+ and at rest with AES-256 * Business Associate Agreements (BAAs) are available for enterprise customers * Comprehensive audit logging is available through integration rules +* Regional data constrains ensure all traffic can be routed through specific geographic regions to meet data residency requirements When building patient monitoring dashboards, use de-identified patient IDs in channel names rather than actual patient identifiers, and ensure your token generation ties into your existing healthcare authorization systems. @@ -411,7 +460,7 @@ Ably's SDKs automatically handle reconnection and [connection state recovery](/d ``` -Understanding each connection state helps you provide appropriate user feedback: +Understanding each connection state helps you provide appropriate user feedback, some examples include: * **`disconnected`** — A temporary loss of connection where the SDK is actively trying to reconnect * **`suspended`** — Reconnection attempts have been unsuccessful for an extended period, but the SDK will continue trying in 30-second intervals @@ -615,11 +664,10 @@ For a dashboard with 1 publisher and 1000 subscribers publishing 1 update per se [Server-side batching](/docs/messages/batch#server-side) groups messages before fanout, dramatically reducing outbound message count during high-activity periods. This is particularly valuable when your data source publishes multiple updates per second. -Let's calculate the impact for a live sports dashboard with 100,000 viewers and a data source publishing 10 updates per second during an exciting match: +To understand the impact, consider a live sports dashboard with 100,000 viewers and a data source publishing 10 updates per second during an exciting match: -Without batching, outbound messages are 10 updates × 100,000 subscribers = 1,000,000 messages per second. Over a 2-hour match, that's 7.2 billion messages. - -With 500ms server-side batching, messages are grouped into 2 batches per second. Each batch counts as a single outbound message per subscriber, so outbound messages become 2 batches × 100,000 subscribers = 200,000 messages per second. Over the same 2-hour match, that's 1.44 billion messages—an 80% reduction. +* **Without batching:** 10 updates × 100,000 subscribers = 1,000,000 messages per second. Over a 2-hour match, that totals 7.2 billion messages. +* **With 500ms batching:** Messages are grouped into 2 batches per second. Each batch counts as a single outbound message per subscriber, so you get 2 batches × 100,000 subscribers = 200,000 messages per second. Over the same 2-hour match, that's 1.44 billion messages—an 80% reduction. The key insight is that batching's effectiveness scales with your inbound message rate. If you're only publishing once per second, batching provides minimal benefit. But during excitement spikes when updates accelerate, batching automatically smooths the fanout. @@ -706,18 +754,15 @@ To facilitate this, Ably provides [outbound integrations](/docs/platform/integra ### Configuring outbound streaming for dashboards -To stream your dashboard data externally, configure an integration rule in your [Ably dashboard](https://ably.com/dashboard) or via the [Control API](/docs/platform/account/control-api). You'll specify which channels to stream using a regular expression filter, what events to capture (messages, presence, lifecycle, or occupancy), and the destination service with its connection details. +To stream your dashboard data externally, configure an integration rule in your [Ably dashboard](https://ably.com/dashboard) or via the [Control API](/docs/platform/account/control-api): -For a patient monitoring dashboard where audit trails are required, you might stream all vitals channels to Kafka for both real-time alerting and long-term storage: +1. Navigate to your app settings and select the **Integrations** tab. +2. Create a new integration rule and choose your destination service (e.g., Apache Kafka, Amazon Kinesis). +3. Specify which channels to stream using a regular expression filter (e.g., `^vitals:.*` for all patient vitals channels). +4. Select the event types to capture—messages, presence, lifecycle, or occupancy depending on your needs. +5. Configure the destination connection details, such as the Kafka topic or Kinesis stream name. - - ```text - Channel filter: ^vitals:.* - Event types: channel.message, channel.presence - Destination: Apache Kafka - Topic: patient-vitals-audit - ``` - +For a patient monitoring dashboard where audit trails are required, you might stream all vitals channels to Kafka (channel filter: `^vitals:.*`, event types: `channel.message` and `channel.presence`, topic: `patient-vitals-audit`) for both real-time alerting and long-term storage. ### Message format considerations @@ -778,24 +823,28 @@ If you're using [message conflation](#message-conflation-for-latest-value-scenar Before deploying your realtime dashboard to production, verify that you've addressed these key areas: -**Authentication:** -* Use token authentication rather than API keys in client code -* Tokens have appropriate capabilities following the principle of least privilege -* Expiration times are appropriate for your security requirements—shorter for sensitive dashboards, longer for public content +**Authentication:** Proper authentication protects your data and prevents abuse. + +* Use token authentication rather than API keys in client code. +* Ensure tokens have appropriate capabilities following the principle of least privilege. +* Set expiration times appropriate to your security requirements—shorter for sensitive dashboards, longer for public content. + +**Message throughput:** Understanding your message volumes ensures you stay within limits and control costs. + +* Verify that your throughput is within channel limits (50 messages/second by default). +* Implement batching if your data source exceeds rate limits. +* Choose and configure the appropriate optimization strategy—batching for burst management, conflation for latest-value scenarios, or delta compression for large payloads. + +**Connection handling:** Users should always understand the state of their connection and data. -**Message throughput:** -* Verify throughput is within channel limits (50 messages/second by default) -* Implement batching if your data source exceeds rate limits -* Choose and configure the appropriate optimization strategy—batching for burst management, conflation for latest-value scenarios, or delta compression for large payloads +* Test connection state handling to ensure users see appropriate feedback during disconnections. +* Verify that message backfill works correctly for extended outages. +* Implement graceful degradation so the dashboard remains usable even when connectivity is impaired. -**Connection handling:** -* Test connection state handling to ensure users see appropriate feedback during disconnections -* Verify message backfill works correctly for extended outages -* Implement graceful degradation so the dashboard remains usable even when connectivity is impaired +**Monitoring:** Proactive monitoring helps you catch issues before they impact users or costs. -**Monitoring:** -* Set up monitoring for connection counts, message rates, and latency percentiles -* Configure budget alerts in the Ably dashboard to catch unexpected usage spikes before they become costly surprises +* Set up monitoring for connection counts, message rates, and latency percentiles. +* Configure budget alerts in the Ably dashboard to catch unexpected usage spikes before they become costly surprises. ## Next steps From 359f5d7e69f2112639ba22bdf5c646d8e17f06a6 Mon Sep 17 00:00:00 2001 From: Steven Lindsay Date: Fri, 9 Jan 2026 14:27:47 +0000 Subject: [PATCH 11/16] rename health tech dashboard guide to dashboards and visualizations --- src/data/nav/pubsub.ts | 2 +- ...lth-tech-dashboard.mdx => dashboards-and-visualizations.mdx} | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename src/pages/docs/guides/pub-sub/{health-tech-dashboard.mdx => dashboards-and-visualizations.mdx} (100%) diff --git a/src/data/nav/pubsub.ts b/src/data/nav/pubsub.ts index b09482cc6a..18e337c2a1 100644 --- a/src/data/nav/pubsub.ts +++ b/src/data/nav/pubsub.ts @@ -330,7 +330,7 @@ export default { pages: [ { name: 'Dashboards and visualizations', - link: '/docs/guides/pub-sub/health-tech-dashboard', + link: '/docs/guides/pub-sub/dashboards-and-visualizations', } ], }, diff --git a/src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx b/src/pages/docs/guides/pub-sub/dashboards-and-visualizations.mdx similarity index 100% rename from src/pages/docs/guides/pub-sub/health-tech-dashboard.mdx rename to src/pages/docs/guides/pub-sub/dashboards-and-visualizations.mdx From 0141f9933939b67d84db2e187087c3391f2def21 Mon Sep 17 00:00:00 2001 From: Steven Lindsay Date: Fri, 9 Jan 2026 17:23:07 +0000 Subject: [PATCH 12/16] updated recommendations on channel rules, optimizations, and batch processing strategies. --- .../pub-sub/dashboards-and-visualizations.mdx | 118 ++++++++---------- 1 file changed, 52 insertions(+), 66 deletions(-) diff --git a/src/pages/docs/guides/pub-sub/dashboards-and-visualizations.mdx b/src/pages/docs/guides/pub-sub/dashboards-and-visualizations.mdx index 3f73179490..ccaaba655a 100644 --- a/src/pages/docs/guides/pub-sub/dashboards-and-visualizations.mdx +++ b/src/pages/docs/guides/pub-sub/dashboards-and-visualizations.mdx @@ -21,7 +21,7 @@ Ably is trusted by organizations delivering realtime data to millions of users s ![Ably Architecture Overview Diagram](../../../../images/content/diagrams/architecture-overview.png) -Delivering dashboard updates in realtime is key to an engaging and effective user experience. Ably's [serverless architecture](/docs/platform/architecture) eliminates the need for you to manage websocket servers. It automatically scales to handle millions of concurrent connections without provisioning or maintenance. Ably also handles all of the edge-cases around delivery, failover and scaling. +Delivering dashboard updates in realtime is key to an engaging and effective user experience. Ably's [serverless architecture](/docs/platform/architecture) eliminates the need for you to manage websocket servers. It automatically scales to handle millions of concurrent connections without provisioning or maintenance. Ably also handles all of the edge-cases around delivery, failover, and scaling. Despite the challenges of delivering these guarantees, Ably is designed to keep costs predictable. Using features such as server-side batching, delta compression, and efficient connection management, along with Ably's consumption-based pricing model, ensures costs are kept as low as possible, no matter the scale. @@ -44,9 +44,9 @@ In these scenarios: * The relationship is typically one publisher broadcasting to many subscribers * High message throughput with sub-second latency is usually sufficient -* Eventual consistency is acceptable—not every intermediate value needs to be delivered +* Eventual consistency is acceptable—intermediate values can be discarded to reduce outbound messages * Cost optimization becomes critical due to message fanout -* Users expect engaging, fast-updating displays rather than mission-critical accuracy +* Access control is often simple, with public read access ### Critical monitoring dashboards: Every message matters @@ -75,7 +75,7 @@ Channels are the foundation of your dashboard architecture. They determine how d ### Single channel for broadcast -For scenarios where all subscribers receive the same data stream, a single channel provides the simplest and most cost-effective architecture. This pattern works well for public dashboards like sports scores, stock prices, or weather updates where every viewer sees identical information. +For scenarios where all subscribers receive the same data stream, a single channel provides the simplest and most cost-effective architecture. This pattern works well for public dashboards like sports scores, stock prices, or match state where every viewer sees identical information. ```javascript @@ -106,12 +106,12 @@ The single channel pattern maximizes cost efficiency because Ably's fanout deliv ### Per-entity channels for isolation and access control -When different subscribers need access to different data streams, or when fine-grained access control is required, per-entity channels provide natural isolation. This pattern is common in healthcare, where each patient has their own channel, or in multi-tenant SaaS platforms where each customer's data must remain separate. +When different subscribers need access to different data streams, or when fine-grained access control is required, per-entity channels provide natural isolation. This pattern is common in industries with strict data compliance, like healthcare, where each patient may have their own dedicated channel, or in multi-tenant SaaS platforms where each customer's data must remain separate. ```javascript // Publisher: Medical device publishing patient vitals - const patientId = 'patient-7f3a9b2e'; // De-identified patient ID + const patientId = 'patient-7f3a9b2e'; const channel = realtime.channels.get(`vitals:${patientId}`); setInterval(async () => { @@ -124,7 +124,7 @@ When different subscribers need access to different data streams, or when fine-g }); }, 1000); - // Subscriber: Nurse station subscribes only to assigned patients + // Subscriber: Nurse monitoring station subscribes only to assigned patients const assignedPatients = ['patient-7f3a9b2e', 'patient-3c8d1a4f']; assignedPatients.forEach(patientId => { @@ -173,9 +173,9 @@ Dashboard applications face varying throughput demands, from steady streams of u ### Understanding rate limits -Ably applies rate limits to ensure platform stability. By default, channels accept up to 50 inbound messages per second. Enterprise plans can request higher limits for specific use cases. When working with high-frequency data sources, you'll need to either batch updates or distribute across multiple channels. +Ably applies rate limits to ensure platform stability. By default, channels accept up to 50 inbound messages per second. Enterprise plans can request higher limits for specific use cases. When working with high-frequency data sources, consider batching multiple updates into single messages to stay within these limits. -For data sources generating more than 50 updates per second, batching multiple samples into single publishes is the recommended approach: +For example, data sources generating more than 50 updates per second could be batched into periodic publishes: ```javascript @@ -210,37 +210,21 @@ For data sources generating more than 50 updates per second, batching multiple s ``` -This approach keeps you well within rate limits while still delivering timely updates. A 100ms batching interval means subscribers see data with at most 100ms additional latency, which is imperceptible for most dashboard use cases. +This approach helps keeps you well within rate limits while still delivering timely updates. A 100ms batching interval means subscribers see data with at most 100ms additional latency, which is imperceptible for most dashboard use cases. ### Server-side batching for cost efficiency -During high-activity periods—a goal being scored, market volatility, or a major incident—message rates can spike dramatically. [Server-side batching](/docs/messages/batch#server-side) helps manage these spikes by grouping messages before delivery to subscribers. +During high-activity periods—a goal being scored, market volatility, or a major incident—message rates can spike dramatically. [Server-side batching](/docs/messages/batch#server-side) helps manage these spikes by grouping messages before delivery to subscribers. Ordering and delivery guarantees are preserved, but billable outbound message counts are reduced. -Configure server-side batching via [channel rules](/docs/channels#rules) in your Ably dashboard. Navigate to your app settings, create a new rule for your channel namespace (e.g., `match:*:stats`), enable server-side batching, and set the batching interval. For responsive dashboards, 100ms provides a good balance. For maximum cost optimization, intervals up to 1000ms can significantly reduce message counts. +The key benefit of server-side batching is that it reduces billable outbound message count, especially during traffic spikes. If your source publishes 10 updates per second and you have 1000 subscribers, without batching you'd have 10,000 outbound messages per second. With 500ms batching, messages are grouped into 2 batches per second, resulting in 2,000 outbound messages per second—a 5x reduction. -The key benefit of server-side batching is that it reduces billable outbound message count during traffic spikes. If your source publishes 10 updates per second and you have 1000 subscribers, without batching you'd have 10,000 outbound messages per second. With 500ms batching, messages are grouped into 2 batches per second, resulting in 2,000 outbound messages per second—a 5x reduction. - -Unlike message conflation, server-side batching preserves all messages. Every update is delivered, just grouped together for efficiency. This makes it suitable for scenarios where you need complete data but want to smooth out traffic spikes. - -### Configuring channel rules - -Both server-side batching and message conflation are configured through [channel rules](/docs/channels#rules) in your Ably dashboard. Channel rules allow you to apply optimizations to specific channels or entire namespaces using regex pattern matching, making it easy to configure behavior across related channels without code changes. - -To configure a channel rule: - -1. Navigate to your app settings in the [Ably dashboard](https://ably.com/dashboard) and select the **Channel Rules** tab. -2. Create a new rule with a channel name pattern (e.g., `^match:.*:stats` to match all match stats channels, or `^vitals:.*` for all patient vitals). -3. Enable the desired optimization—server-side batching, message conflation, or both. -4. For batching, set the interval (100ms for responsive dashboards, up to 1000ms for maximum cost savings). -5. For conflation, configure the conflation interval and key pattern to determine which messages are merged. - -Using namespace patterns means you can apply different rules to different parts of your application. For example, you might enable conflation on price ticker channels (`^prices:.*`) while using batching on event feeds (`^events:.*`) where complete data is required. +Unlike message conflation, server-side batching preserves all messages and message order. Every update is delivered, just grouped together for efficiency. This makes it suitable for scenarios where you need complete data, but can tolerate some latency in exchange for cost savings. ### Message conflation for latest-value scenarios -For dashboards where only the current value matters—stock prices, sensor readings, vehicle positions—[message conflation](/docs/messages#conflation) delivers only the most recent value within each time window. +For dashboards where only the current state over some time window matters—stock prices, sensor readings, vehicle positions—[message conflation](/docs/messages#conflation) delivers only the most recent value within each time window. -Configure conflation through [channel rules](/docs/channels#rules) in your dashboard. You'll specify a conflation interval (e.g., 100ms) and a conflation key pattern that determines which messages are considered related. Messages with the same conflation key within the time window are conflated together, with only the latest delivered. +Configure conflation through [rules](/docs/channels#rules) in your dashboard. You'll specify a conflation interval and a conflation key pattern that determines which messages are considered related. Messages with the same conflation key within the time window are conflated together, with only the latest delivered. Multiple conflated messages are then delivered in a single batch at the end of the interval.