RCP-260211A: S3 Storage Backend Support for RGB Persistence Layer

[stash-labs]

Type: C-stdlib (Standard Library Change Proposal)
Author: Stash Labs
Related: RCP-240814A, RCP-240725A

Summary

This proposal adds an S3-compatible storage backend to the existing rgb-std standard library.

Motivation

The current rgb-persist-fs filesystem implementation lacks:

• Multi-device synchronization capability, such as using the same wallet across mobile and PC
• Reliable backup/recovery mechanism, such as automatic backup and quick recovery from data loss
• Enterprise-level availability requirements, which involve the ability to manage the file data of tens of thousands or even hundreds of thousands of users simultaneously.

S3 protocol provides:

• Standard object storage interface, which has become the de facto cloud storage standard
• Wide ecosystem support (AWS, MinIO, Filebase), enabling immediate cloud storage implementation
• Path to decentralized storage (IPFS, Arweave via S3 gateways). Through S3 gateways, file data can be stored on decentralized remote servers, where contract can be stored in Arweave

The Core Challenge

The primary concern is how to resolve the conflict between synchronous and asynchronous methods.

// RGB current interface: synchronous
pub trait Stockpile {
    fn contract(&self, id: ContractId) -> Option<Contract>;
}

// S3 SDK: asynchronous
impl S3Client {
    async fn get_object(&self, ...) -> Result<...>;
}

After carefully studying other RFC documents and related code, This issue has been previously discussed. Dr. Orlovsky proposed a solution in #12, implementing an object with the "persistent provider" trait. This object would contain a dedicated thread that uses async I/O to store data to filesystem or database. However, this approach can also cause integration challenges for downstream products. For example, this may introduce additional complexity in asynchronous runtime

After reviewing other related discussions, we concluded that the consensus reached was to have RGB protocol integrators implement remote storage themselves. While this alleviates the problem, it means every developer or company integrating the RGB protocol needs to write a complete set of storage code from scratch. Moreover, implementing this storage solution requires significant time and effort to learn and read RGB's source code, which is not conducive to the development of the RGB protocol ecosystem.

Proposal

Our latest exploration has found that async programming is not necessarily required to integrate S3. Synchronous approaches can also integrate S3, such as using the rust-s3 library: https://github.com/durch/rust-s3

Recommended Solution: Using rust-s3 Synchronous Mode for S3 Integration

After thorough analysis of the rust-s3 library (v0.37.1), It provides robust synchronous support suitable for RGB integration:

Synchronous Features Available:

• sync-native-tls - Synchronous mode with OpenSSL/Native TLS
• sync-rustls-tls - Synchronous mode with Rustls (pure Rust TLS)
• blocking - Generates *_blocking variants for all Bucket methods
• Uses attohttpc as the underlying synchronous HTTP client (287K+ monthly downloads)

Architecture Compatibility:
The synchronous API directly matches RGB's Stockpile trait requirements:

// RGB Interface (synchronous)
pub trait Stockpile {
    fn contract(&self, id: ContractId) -> Option<Contract>;
}

// rust-s3 Synchronous API
bucket.get_object_blocking("/path/to/file")?;  // No async/await needed
bucket.put_object_blocking("/path/to/file", data)?;

Compilation Impact Analysis

A critical concern is whether rust-s3's async code will affect RGB's compilation when using sync mode.

How rust-s3 Compilation Works:

rust-s3 uses conditional compilation via maybe-async macro. When configured correctly, async code is completely excluded from compilation:

[dependencies]
rust-s3 = {
    version = "0.37.1",
    default-features = false,  # Disables default async features
    features = ["sync-native-tls"]  # Only enables sync mode
}

Key Points:

• ✅ With default-features = false, async runtime (tokio/async-std) code is NOT compiled
• ✅ No async dependencies are linked (tokio, futures, etc.)
• ✅ Smaller binary size (sync-only code)
• ✅ The maybe-async macro strips all async/await keywords at compile time for sync builds

---

Alternative Solution

@crisdut proposed an alternative approach that requires neither changing nor implementing a new storage provider, but simply extending the existing storage API to support the above goals (remote backup/recovery).

Our Position: The API extension approach is simpler for basic backup/recovery, but we believe a dedicated storage provider offers better separation of concerns and long-term scalability for ecosystem application implementation needs. However, if the community prefers to extend the existing API, we are willing to start with API extension as Phase 1, with the understanding that as requirements grow, It may be necessary to evolve toward a dedicated provider.

WASM Compatibility Challenge

Concern: Blocking calls cannot work in WASM/browser environments due to event-driven architecture (event loop). Browsers require non-blocking, asynchronous calls to external APIs. Attempting blocking calls in browser environments will encounter numerous runtime errors as they freeze the main thread (Proposed by @crisdut)

Reference: wasm-bindgen/wasm-bindgen#2111

Response: This fundamental constraint is acknowledged. Resolving WASM compatibility is beyond the scope of this proposal. As the implementation difficulty and workload cannot currently be assessed, this proposal focuses on native solutions.

Proof of Concept (PoC) Plan

Based on community feedback, a minimal PoC will be implemented to validate the approach. The PoC will focus on several key validation points:

1. Extend Existing Storage API to Support This Proposal (Remote Backup/Recovery)

• Provide a simple, effective, and reliable way for S3 remote data backup and recovery
• Test data backup integrity and recovery in different scenarios

2. Implement Dedicated Storage Provider and Verify Synchronization Mechanism with Data Consistency

• How data and state validation consistency works
• Test conflict detection for multi-device scenarios

Implementation Commitment:
This proposal may not be the most suitable storage solution for all RGB use cases, but at the current stage, it serves as a transitional solution for enterprise-level users requiring cloud storage capabilities. We will continuously share PoC progress with the community. After PoC validation is complete and this proposal is approved, Stash Labs is willing to implement this solution under RGB consortium guidance, providing production-grade code, comprehensive tests, documentation, and ongoing support services.

---

Acknowledgments

Sincere gratitude is extended to Armando Dutra and Adam Borco for their invaluable criticisms and suggestions during the drafting of this RFC. Their insights have been instrumental in clarifying the core objectives of this proposal. Continued criticisms and suggestions from the community on this RFC are warmly welcomed.