🎓 Educational Project: This repository is a comprehensive learning journey into Distributed Hash Tables (DHT) using practical examples and hands-on experimentation.
This project documents a complete learning journey from DHT fundamentals to production-ready distributed systems. Perfect for developers new to peer-to-peer networking who want hands-on experience with real DHT implementation.
- 🔗 P2P Network Fundamentals - Bootstrap nodes, routing tables, peer discovery
- 🗂️ Distributed Storage - Content addressing, data replication, fault tolerance
- 🏗️ Network Architecture - Kademlia routing, NAT traversal, censorship resistance
- 💾 Production Systems - Large-scale networks, JSON storage, enterprise patterns
- 🛡️ Security Concepts - Attack vectors, defense strategies, network resilience
| Phase | Focus Area | Key Achievements |
|---|---|---|
| 1-2 | Core Concepts & Network Setup | Bootstrap nodes, peer discovery, routing tables |
| 3 | Connection Process | Node lifecycle, bootstrap handshake, network integration |
| 4 | Security Analysis | Bootstrap attack vectors, censorship vulnerabilities |
| 5 | Storage Architecture | Request handlers, data persistence, command protocols |
| 6 | Advanced Features | Warehouse nodes, metadata tracking, analytics |
| 7 | Production Scale | 63-node networks, JSON storage pipeline, multi-node replication |
| 8 | Enterprise Storage ⭐ | Production warehouse system, CRUD operations, comprehensive testing |
step by step learning.md- Complete learning journey with discoveries and insights
verbose_bootstrap.mjs- Enhanced bootstrap with detailed network activity loggingnew_node.mjs- Educational node showing complete bootstrap processfresh_node.mjs- Debug version for connection troubleshooting
storage_node.mjs- Basic storage-enabled DHT nodewarehouse_node.mjs- Enterprise storage with metadata, analytics, and CRUD operationsinsert_json.mjs- JSON file insertion with validation and hash generationretrieve_json.mjs- JSON retrieval with integrity checking and formattingsample.json- Test data for storage experiments
test_warehouse.mjs- NEW! Comprehensive 7-test validation suite for warehouse storageworking_demo.mjs- Functional DHT demonstrationsimple_demo.mjs- Basic DHT operationssmall_network.mjs- Controlled network testingfixed_insert.mjs- Corrected insertion logic with proper bootstrapping
debug_insert.mjs- Insertion debugging and troubleshootingtest_*.mjs- Various testing scenarios for network behavior
# Start bootstrap node (terminal 1)
node examples/bootstrap.mjs
# Create 63-node storage network (terminal 2)
node examples/network.mjs# Insert JSON file into DHT
node insert_json.mjs sample.json
# Returns hash: fe55bb60efb0b18d59e644a55263b4be25d8c5217dfd55a7d18eb37f5958a494# Retrieve by hash
node retrieve_json.mjs fe55bb60efb0b18d59e644a55263b4be25d8c5217dfd55a7d18eb37f5958a494
# Save to file
node retrieve_json.mjs <hash> retrieved_data.json# Start multiple warehouse nodes for distributed storage
node warehouse_node.mjs # Warehouse 1
node warehouse_node.mjs # Warehouse 2 (different terminal)
node warehouse_node.mjs # Warehouse 3 (different terminal)
# Test the complete warehouse system
node test_warehouse.mjs
# Runs 7 comprehensive tests:
# ✅ Storage operations with metadata
# ✅ Retrieval with integrity validation
# ✅ Inventory listing and management
# ✅ Access tracking and analytics
# ✅ Deletion and lifecycle management
# ✅ Error handling and edge cases
# ✅ Multi-node compatibilityThe warehouse system has been completely overhauled with enterprise-grade features for production distributed storage networks.
- Fixed DHT Protocol Handling: Proper token management and request/response cycles
- Advanced Metadata Tracking: Access counts, timestamps, storage sizes, source tracking
- Complete CRUD Operations: Store, Retrieve, List, Delete with full lifecycle management
- Production-Ready Logging: Comprehensive operation tracking and debugging
- Multi-Node Compatibility: Seamless operation across distributed warehouse networks
| Command | Code | Purpose | Features |
|---|---|---|---|
| STORE | 100 | Store data with metadata | Size tracking, source attribution, timestamp |
| GET | 101 | Retrieve with access tracking | Increments access count, logs retrieval |
| LIST | 102 | Inventory management | Complete metadata listing, JSON export |
| DELETE | 103 | Lifecycle management | Secure deletion with confirmation |
The test_warehouse.mjs provides complete validation:
🧪 WAREHOUSE STORAGE TEST RESULTS
==================================================
✅ Passed: 7/7 - ALL TESTS PASSED!
🎉 Warehouse storage is working correctly!
Tests Cover:
1. Storage Operations - Multi-format data storage
2. Retrieval Integrity - Data consistency validation
3. Inventory Management - Metadata listing and tracking
4. Access Analytics - Usage pattern monitoring
5. Deletion Lifecycle - Secure data removal
6. Error Handling - Edge case resilience
7. Multi-Node Support - Distributed network compatibilityFault Tolerance: Data survives individual node failures
# Run multiple warehouses for redundancy
node warehouse_node.mjs # Port auto-assigned
node warehouse_node.mjs # Port auto-assigned
node warehouse_node.mjs # Port auto-assigned
# Data automatically distributed across all nodesLoad Distribution: Requests balanced across network
// Clients automatically find optimal warehouse
const warehouses = await findAvailableWarehouses()
// DHT routes to nearest/fastest warehouse automaticallyGeographic Scaling: Deploy warehouses globally
# Different geographic locations
Server-US: node warehouse_node.mjs --location="us-east"
Server-Europe: node warehouse_node.mjs --location="eu-west"
Server-Asia: node warehouse_node.mjs --location="ap-south"
# Users connect to nearest warehouse automatically- Enterprise Storage: Replace centralized databases with distributed warehouses
- Content Distribution: Media files replicated across geographic locations
- IoT Data Collection: Sensor data stored in regional warehouse clusters
- Microservices Storage: Each service gets its own warehouse network
- Web3 Applications: Decentralized storage without central authorities
Store Any Data Type:
// JSON data
await store('user-config', JSON.stringify({theme: 'dark', lang: 'en'}))
// Binary files
const fileBuffer = fs.readFileSync('document.pdf')
await store('document-123', fileBuffer)
// Plain text
await store('session-data', 'user-session-abc123')Advanced Queries:
// Get complete inventory with metadata
const inventory = await listWarehouseInventory()
console.log(inventory)
// Shows: keys, sizes, access counts, timestamps
// Track data usage
const stats = await getAccessStatistics('user-data-key')
console.log(`Accessed ${stats.count} times, last: ${stats.lastAccess}`)Multi-Node Replication:
// Store in multiple warehouses for redundancy
const warehouses = await findWarehouseNodes()
for (const warehouse of warehouses.slice(0, 3)) {
await storeInWarehouse(warehouse, criticalData)
}
// Data now exists in 3 different locations- Bootstrap Dependency: New nodes cannot join without accessible bootstrap nodes
- Censorship Vulnerability: Authoritarian states can kill P2P networks by blocking bootstraps
- Automatic Replication: Data is stored on multiple nodes automatically for fault tolerance
- Content Addressing: Same content always has same hash-based address globally
- Single Point of Failure: Bootstrap nodes are critical infrastructure
- Defense Strategies: Multiple bootstraps, domain fronting, bridge networks needed
- Attack Vectors: Network partitioning, bootstrap blocking, routing manipulation
- Storage Node = Regular Node + Request Handlers + Local Storage
- Warehouse Node = Storage Node + Metadata + Analytics + Lifecycle Management
- Enterprise DHT = Basic DHT + Audit Trails + Performance Monitoring
- JSON stored across multiple nodes simultaneously
- Fault-tolerant retrieval from any storing node
- Network resilience with node failures
- SHA256-based content addressing
- Global content discovery by hash
- Data integrity verification
- Advanced Metadata Tracking: Timestamps, sizes, access counts, source attribution
- Complete CRUD Operations: Store, GET, LIST, DELETE with lifecycle management
- Real-time Analytics: Usage patterns, performance monitoring, access statistics
- Comprehensive Logging: Operation audit trails and debug information
- Multi-Node Orchestration: Automatic distribution and load balancing
- Production Testing: 7-test validation suite ensuring reliability
This project serves as a complete DHT learning curriculum with:
- 📚 Theory: Step-by-step concept building from basics to advanced topics
- 🔧 Practice: Working code examples for every concept
- 🧪 Experimentation: Tools for testing ideas and exploring edge cases
- 📊 Analysis: Real network behavior observation and documentation
- 🏭 Production: Scalable patterns for real-world applications
Perfect for:
- Computer Science Students learning distributed systems
- Developers transitioning to P2P/blockchain development
- System Architects designing decentralized applications
- Researchers studying network resilience and censorship resistance
This study project is built upon the excellent dht-rpc library:
- Original Author: Mathias Buus (@mafintosh)
- Original Repository: holepunchto/dht-rpc
- License: MIT
- Description: Production-grade Kademlia DHT implementation for Node.js
- Educational layer: Step-by-step learning documentation
- Practical examples: 19+ working demo files with different use cases
- Production patterns: Enterprise storage architectures and scaling examples
- Security analysis: Real-world attack vectors and defense strategies
- JSON pipeline: Complete content storage and retrieval system
- Created by: storyteller
- Learning Documentation: Original educational content and analysis
- Example Scripts: Custom implementations for learning objectives
- Architecture Patterns: Original warehouse and enterprise storage designs
The core DHT-RPC library provides the foundation for this study. For complete API documentation, see the original repository.
- Remote IP / firewall detection
- Easily add any command to your DHT
- Streaming queries and updates
- Production-ready Kademlia implementation
- NAT traversal and network resilience
import DHT from 'dht-rpc'
// Create bootstrap node
const bootstrap = DHT.bootstrapper(10001, '127.0.0.1')
// Create DHT node
const node = new DHT({ bootstrap: ['localhost:10001'] })
// Add request handling
node.on('request', (req) => {
// Handle storage/retrieval requests
})After completing this study, you will understand:
- DHT node creation and configuration
- P2P network bootstrap and discovery
- Distributed storage implementation
- Content addressing and hash-based routing
- Decentralized architecture patterns
- Fault tolerance and replication strategies
- Network security and attack mitigation
- Production scaling and monitoring
- P2P vs client-server trade-offs
- Distributed consensus challenges
- Censorship resistance mechanisms
- Real-world deployment considerations
This study foundation enables building:
- Distributed file systems
- Censorship-resistant messaging
- Peer-to-peer social networks
- Blockchain light clients
- Custom routing algorithms
- Encrypted storage systems
- Cross-network bridging
- Performance optimization
- Original dht-rpc library: MIT License (Mathias Buus)
- Study project additions: MIT License
- Educational content: Available for learning and reference
🎉 Ready to explore the decentralized future? Start with Phase 1 in step by step learning.md!