Distributed File Storage System

A decentralized, fully distributed, content-addressable file storage system built in Go. This project implements peer-to-peer file sharing with AES encryption, allowing files to be stored across multiple nodes and retrieved from the network.

Features

• Peer-to-Peer Architecture: Multiple nodes communicate directly without a central server
• Content-Addressable Storage: Files are stored using SHA1 hashes for deduplication
• AES Encryption: Files are encrypted before being transmitted over the network
• Automatic Replication: Files stored on one node are automatically replicated to connected peers
• Large File Streaming: Efficient handling of large files through streaming

Project Structure

Project/
├── main.go           # Entry point - sets up and runs file servers
├── server.go         # FileServer implementation and message handling
├── store.go          # Local file storage with content-addressable paths
├── crypto.go         # Encryption/decryption utilities
├── p2p/              # Peer-to-peer networking layer
│   ├── transport.go      # Transport interface
│   ├── tcp_transport.go  # TCP implementation
│   ├── message.go        # RPC message types
│   ├── encoding.go       # Message encoding/decoding
│   └── handshake.go      # Peer handshake logic
├── store_test.go     # Storage tests
├── Makefile          # Build commands
└── README.md         # This file

Prerequisites

• Go 1.21 or higher

Installation

1. Clone the repository:

   git clone <your-repo-url>
   cd Project

2. Install dependencies:

   go mod tidy

Building

# Build the project
go build -o bin/fs .

# Or use the Makefile
make build

Running

Using Go directly

go run .

Using the compiled binary

./bin/fs

Using Makefile

make run

How It Works

Default Configuration

The default main.go starts 3 interconnected nodes:

Node	Port	Connects To
s1	3000	-
s2	7000	-
s3	5000	s1, s2

Demo Flow

1. Three servers start and establish connections
2. Server s3 stores 20 test files (picture_0.png through picture_19.png)
3. Files are encrypted and replicated to s1 and s2
4. s3 deletes its local copies
5. s3 fetches files back from the network
6. Retrieved content is printed to console

Customization

Adding Your Own Files

Modify main.go to store your own files:

func main() {
    s1 := makeServer(":3000", "")
    s2 := makeServer(":4000", ":3000")

    go func() { log.Fatal(s1.Start()) }()
    time.Sleep(500 * time.Millisecond)
    go s2.Start()
    time.Sleep(2 * time.Second)

    // Store a file
    data := bytes.NewReader([]byte("Your file content here"))
    s2.Store("myfile.txt", data)

    // Retrieve a file
    reader, err := s2.Get("myfile.txt")
    if err != nil {
        log.Fatal(err)
    }
    
    content, _ := ioutil.ReadAll(reader)
    fmt.Println(string(content))
}

Changing Ports

Modify the makeServer calls in main.go:

s1 := makeServer(":8000", "")           // Listen on port 8000
s2 := makeServer(":8001", ":8000")      // Listen on 8001, connect to 8000

Adding More Nodes

s1 := makeServer(":3000", "")
s2 := makeServer(":4000", ":3000")
s3 := makeServer(":5000", ":3000", ":4000")
s4 := makeServer(":6000", ":3000", ":4000", ":5000")

API Reference

FileServer Methods

Method	Description
Store(key string, r io.Reader)	Store a file with the given key
Get(key string) (io.Reader, error)	Retrieve a file by key (local or network)
Start() error	Start the file server
Stop()	Stop the file server

Store Methods

Method	Description
Has(id, key string) bool	Check if a file exists locally
Read(id, key string) (int64, io.Reader, error)	Read a file from disk
Write(id, key string, r io.Reader) (int64, error)	Write a file to disk
Delete(id, key string) error	Delete a file from disk
Clear() error	Remove all stored files

Testing

# Run all tests
go test -v ./...

# Run only store tests
go test -v -run TestStore

# Run only p2p tests
go test -v ./p2p/...

Troubleshooting

Port Already in Use

If you see "address already in use" errors, change the ports in main.go or kill existing processes:

lsof -i :3000
kill <PID>

Connection Refused

Ensure nodes are started in the correct order - bootstrap nodes must be running before other nodes try to connect.

Architecture Overview

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Node 1    │────▶│   Node 2    │────▶│   Node 3    │
│  (Port 3000)│◀────│  (Port 7000)│◀────│  (Port 5000)│
└─────────────┘     └─────────────┘     └─────────────┘
      │                   │                   │
      ▼                   ▼                   ▼
┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Storage   │     │   Storage   │     │   Storage   │
│  :3000_net  │     │  :7000_net  │     │  :5000_net  │
└─────────────┘     └─────────────┘     └─────────────┘

File Storage Path Structure

Files are stored using content-addressable paths:

<storage_root>/<server_id>/<hash_path>/<filename>

Example:
:3000_network/abc123.../68044/29f74/181a6/.../6804429f74181a63c50c3d81d733a12f14a353ff

License

MIT License