CAR Content Locator

Explorer UI for tracking content stored in Content Addressable aRchives (CARs) on Filecoin and IPFS networks. See demo.

Originally developed for USC (University of Southern California) Libraries to keep track of private files stored on the Filecoin network and to explore Deal information, the CAR Content Locator now supports:

• Storing file metadata
• Storing file location within a Piece and Deal information on the Filecoin network (requires Singularity)
• Looking up copies of a file on IPFS using public/private delegated routing endpoints

Screenshot

Installation

Option 1: Docker Compose (Recommended)

1. Clone or download the CAR Content Locator files
2. Run the application with Docker:

   docker-compose up -d

3. Access the application at http://localhost:8080
4. MariaDB is accessible on port 3306 with:
   • Database: singularity
   • Username: caruser
   • Password: carpassword

You'll need to populate the database by pointing Singularity at it. You can use the connection string for this:

export DATABASE_CONNECTION_STRING="mysql://caruser:carpassword@tcp(127.0.0.1:3306)/singularity?parseTime=true"

Option 2: Manual Setup

Prerequisites

• LAMP stack (Linux, Apache, MySQL, PHP) or equivalent server environment
• Singularity CLI tool installed and configured using MySQL
• MySQL/MariaDB database server
• PHP 7.4 or higher

Setup

1. Clone or download the CAR Content Locator files to your web server directory
2. Create a _config.php file in the root directory with the following content:

    <?php
    $sql_dbname = "[[ DBNAME ]]";       // Your database name
    $sql_username = "[[ USERNAME ]]";   // Database username
    $sql_password = "[[ PASSWORD ]]";   // Database password
    $sql_servername = "[[ HOSTNAME ]]"; // Database hostname (localhost or IP)

3. Initialize the Singularity database using the admin command:
   singularity admin init
4. Create the additional required database tables (see SQL section below)
5. Navigate to the application URL in your web browser to verify the installation

Database Schema

The application requires the following additional tables beyond the standard Singularity schema.

Data Table

Tracks metadata for files in the dataset:

CREATE TABLE `usc_data` (
  `usc_data_id` int(11) NOT NULL AUTO_INCREMENT,
  `file_path` varchar(1000) NOT NULL,
  `sha1` varchar(100) NOT NULL,
  `title_id` varchar(50) NOT NULL,
  `file_id` int(11) NOT NULL,
  PRIMARY KEY (`usc_data_id`),
  KEY `index_file_path` (`file_path`(768)),
  KEY `usc_data_title_id` (`title_id`)
) ENGINE=InnoDB AUTO_INCREMENT=1224999 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;

Files Table

Stores details about files:

CREATE TABLE `usc_files` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `relative_path` text DEFAULT NULL,
  `sha1` text DEFAULT NULL,
  `size` bigint(11) DEFAULT NULL,
  `key_fp` text DEFAULT NULL,
  `encrypted_key` text DEFAULT NULL,
  `title_id` varchar(50) NOT NULL DEFAULT '',
  `usc_file_id` int(11) NOT NULL,
  `tar_source` int(11) NOT NULL DEFAULT 0,
  PRIMARY KEY (`id`),
  KEY `index_relative_path` (`relative_path`(768)),
  KEY `files_title_id` (`title_id`)
) ENGINE=InnoDB AUTO_INCREMENT=1284775 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;

File Range CAR Table

Maps file ranges to CAR files:

CREATE TABLE `file_range_car` (
  `file_range_id` int(11) DEFAULT NULL,
  `car_id` int(11) DEFAULT NULL,
  KEY `idx_file_range_car_file_range_id` (`file_range_id`),
  KEY `idx_file_range_car_car_id` (`car_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

Mapping files to CARs

You will need to run singularity-mapping on the database to provide the UI will all the data it needs.

Usage in the USC Libraries' Filecoin Pipeline

USC Libraries uses the following Singularity-based pipeline to prepare encrypted data for storing onto their Filecoin node, as well as nodes of their storage partners. The CAR Content Locator relies on data produced at various steps to keep track of file metadata, file ranges, Piece CIDs, and Deal information.

flowchart LR
    %% Data Input
    A[/Collection of Files/] --> B[Pre-Preparation Step]
    
    %% Processing Pipeline
    B --> C[/Encrypted Files + Validated Hashes/]
    C --> D[Singularity: Preparation and Packing]
    D --> E[/CAR Files/]
    E --> F[Storage Provider: Store as Deals]
    
    %% Metadata Tracking
    C --> H[Car-File Mapping]
    
    %% Final Storage
    F --> G[/Stored Deals/]

Loading

* Data archived by USC Libraries is generally not publicly accessible, and not published to IPFS, so the IPFS features are not used by USC Libraries

Adding your own IPFS providers

When possible, the UI shows information about where the file is stored on the IPFS network and roughly how many copies of it exist. By editing providers.js, you can add your own delegated routing node. This could be used to provide visibiliy into your own private IPFS network, for example.

To do this, simply edit the providerData variable at the top of the file:

const providerData = [
  { url: "https://indexer.pinata.cloud", name: "Pinata" },
  { url: "https://routingv1.storacha.network", name: "Storacha" },
  // etc..
  { url: "https://my.secret.indexer.com", name: "Secret Network" },
];