Bulk import PDFs from a local folder

[dakl]

Summary

Support importing an entire folder of PDFs into PaperShelf. This is the main onboarding path for users who already have a local library of papers.

Current state

PaperShelf is arXiv-centric: every paper requires an arxivId as its primary identifier. Metadata is fetched from the arXiv API, and citations are resolved via Semantic Scholar using that arXiv ID. There is no support for papers that don't have an arXiv entry.

Questions to resolve

1. Matching local PDFs to arXiv entries

Given a folder of PDFs with arbitrary filenames, how do we find the corresponding arXiv paper?

Possible approaches:

• Extract text from PDF → search arXiv by title. Use the existing pdf-parse text extraction, pull the title from the first page, and query the arXiv API. Fragile — titles may not match exactly (formatting, special characters, line breaks).
• Extract DOI from PDF metadata or text → resolve to arXiv ID. Many PDFs embed a DOI in their metadata or first page. Use CrossRef or Semantic Scholar to map DOI → arXiv ID.
• Use Semantic Scholar title search as a fuzzy fallback. S2 has a paper search endpoint that returns arXiv IDs when available. Could be more tolerant of title variations than the arXiv API.
• Fingerprint-based matching. Hash the PDF content or extract a unique identifier (e.g., first N characters of abstract) and match against known databases.

We likely need a pipeline that tries multiple strategies in order and presents uncertain matches to the user for confirmation.

2. Handling papers with no arXiv entry

Many papers (conference proceedings, journals, theses, internal reports, non-CS fields) are not on arXiv at all. The current data model requires arxivId as a UNIQUE non-null column.

Questions:

• Schema change: Should arxivId become nullable? Or should we introduce a more general identifier system (DOI, S2 ID, ISBN, or a synthetic internal ID)?
• Metadata sources for non-arXiv papers: CrossRef (via DOI), Semantic Scholar (via title search or DOI), Google Scholar, or manual entry?
• Citation graph: Semantic Scholar can resolve citations by DOI or S2 ID, not just arXiv ID. Should we generalize the citation fetching to support multiple identifier types?
• User experience: What does a "paper without metadata" look like in the UI? Minimal card with just filename + extracted title? A prompt to manually fill in metadata?

3. Import UX

• Progress feedback: Importing hundreds of PDFs with text extraction + metadata resolution will be slow. Need a progress indicator and the ability to cancel.
• Conflict resolution: What if a PDF matches a paper already in the library? Skip, overwrite, or ask?
• Batch review: After matching, show the user a list of matched/unmatched papers so they can confirm or correct before committing to the database.
• Folder watching (future): Should we support watching a folder for new PDFs and auto-importing? Or is one-time import enough for v1?

Proposed approach (high-level)

1. User selects a folder via native file dialog
2. App scans for *.pdf files
3. For each PDF:
   a. Extract text and metadata (title, DOI if present)
   b. Try to match: DOI → S2/CrossRef → arXiv ID, then title search as fallback
   c. Classify as: matched (high confidence), uncertain (needs review), unmatched
4. Show import review screen with results
5. User confirms → papers are saved to database

Related work needed

• Generalize the data model to support non-arXiv papers (nullable arxivId or a new identifier system)
• Add DOI resolution (CrossRef API or Semantic Scholar DOI lookup)
• Add a bulk import IPC channel and progress reporting mechanism

[dakl]

Implementation Plan

Context

PaperShelf is currently arXiv-only: every paper requires an arxivId (UNIQUE NOT NULL), metadata comes from the arXiv API, and citations resolve via Semantic Scholar using that arXiv ID. There is no way to import local PDFs, and papers without an arXiv entry cannot exist in the system.

This plan adds folder-based PDF import with a metadata resolution pipeline, and generalizes the data model so papers without any public identifier (no DOI, no arXiv) can live in the library as first-class citizens.

---

Phase 1: Schema Evolution (foundation — everything depends on this)

1a. Add migration infrastructure to src/main/db/connection.ts

• Add schema_migrations table to track applied versions
• Add runMigrations() called after createSchema()
• Each migration is a numbered function that runs inside a transaction

1b. Migration 1: Generalize the papers table

SQLite can't alter column constraints, so we recreate the table:

CREATE TABLE papers_new (
  id TEXT PRIMARY KEY,
  arxiv_id TEXT,                          -- was UNIQUE NOT NULL, now nullable
  doi TEXT,                                -- NEW
  s2_id TEXT,                              -- NEW
  source TEXT NOT NULL DEFAULT 'arxiv',    -- NEW: 'arxiv' | 'doi' | 'local'
  title TEXT NOT NULL,
  authors TEXT NOT NULL DEFAULT '[]',
  abstract TEXT NOT NULL DEFAULT '',
  published_date TEXT NOT NULL DEFAULT '',
  updated_date TEXT NOT NULL DEFAULT '',
  categories TEXT NOT NULL DEFAULT '[]',
  arxiv_url TEXT NOT NULL DEFAULT '',
  pdf_url TEXT NOT NULL DEFAULT '',
  pdf_path TEXT,
  full_text TEXT,
  is_favorite INTEGER NOT NULL DEFAULT 0,
  created_at TEXT NOT NULL DEFAULT (datetime('now'))
);

-- Partial unique indexes (unique when not null)
CREATE UNIQUE INDEX idx_papers_arxiv_id ON papers(arxiv_id) WHERE arxiv_id IS NOT NULL;
CREATE UNIQUE INDEX idx_papers_doi ON papers(doi) WHERE doi IS NOT NULL;

Existing data migrated with source = 'arxiv', new columns as NULL. Also update citation_fetch_log to use a generic paper_identifier + identifier_type instead of just arxiv_id.

1c. Update types — src/shared/types.ts

export type PaperSource = 'arxiv' | 'doi' | 'local';

export interface LibraryPaper {
  // ... existing fields ...
  arxivId: string | null;   // was: string
  doi: string | null;        // NEW
  s2Id: string | null;       // NEW
  source: PaperSource;       // NEW
}

1d. Update DB layer — src/main/db/papers.ts, connection.ts

• Update PaperRow interface with new nullable columns
• Update rowToLibraryPaper mapping
• Update insertPaper to accept optional doi, s2Id, source
• Add getPaperByDoi(doi), checkPaperExists({ arxivId?, doi?, pdfPath? })

1e. Fix all null-safety issues throughout codebase

Every paper.arxivId reference must handle null:

• resolvers.ts — formatPaper, generateBibtex: omit arXiv fields when null
• paper-tools.ts — guard arXiv-specific operations
• citation-cache.ts — support DOI-based cache lookups
• citations.ts — getS2IdsByArxivIds needs DOI fallback
• PaperDetail.tsx — conditional arXiv/DOI display
• ipc-handlers.ts — citations:fetch should try DOI if no arxivId

---

Phase 2: PDF Metadata Extraction

2a. Enhance src/main/pdf-processor.ts

Add metadata extraction using the existing pdf-parse v2 library (supports getInfo()):

export interface PdfMetadata {
  title: string | null;
  authors: string[];
  doi: string | null;
}

export async function extractPdfMetadata(buffer: Buffer): Promise<PdfMetadata>

• Extract Title and Author from PDF document properties via parser.getInfo()
• Extract DOI from text content using regex: /\b(10\.\d{4,9}\/[-._;()/:A-Z0-9]+)/i
• Title fallback chain: PDF metadata Title → first prominent line from text → filename

2b. Add local PDF import function

export async function importLocalPdf(sourcePath: string, paperId: string):
  Promise<{ pdfPath: string; fullText: string | null; metadata: PdfMetadata }>

---

Phase 3: Identifier Resolution Pipeline

3a. New file: src/main/services/resolve-paper.ts

The core matching logic. Given extracted PDF metadata, tries to find the paper online:

1. DOI found in PDF → CrossRef lookup by DOI → get full metadata (authoritative for DOI)
2. DOI found → Semantic Scholar DOI:<doi> lookup → get arXiv ID + S2 ID + citation graph access
3. Title extracted → Semantic Scholar /paper/search/match?query=<title> → verify title similarity
4. Title extracted → arXiv API title search → verify match
5. No match → paper auto-imports as source: 'local' with whatever metadata was extracted from the PDF itself

Unmatched papers are auto-imported — the user can delete them later if unwanted.

3b. New file: src/main/crossref/client.ts

Simple CrossRef API client (free, no API key):

• lookupByDoi(doi) and searchByTitle(title)

3c. Extend src/main/semantic-scholar/client.ts

Add DOI lookup and title search using existing rate limiter:

• lookupByDoi(doi) — uses DOI:<doi> prefix
• searchByTitle(title) — uses /paper/search/match

3d. New file: src/main/services/title-similarity.ts

Normalized Jaccard similarity on word sets. Threshold ~0.8 for auto-match to prevent false positives on generic titles.

---

Phase 4: Import Service + IPC

4a. New file: src/main/services/import-papers.ts

Single-pass flow (scan → resolve → import automatically):

async function importPdfs(
  filePaths: string[],
  onProgress: (progress: ImportProgress) => void,
): Promise<ImportResult>

4b. New IPC channels in src/main/ipc-handlers.ts

• import:selectFolder — native folder dialog
• import:selectFiles — native file picker, PDF filter
• import:run — runs importPdfs, sends progress via webContents.send('import:progress', ...)

4c. Types

interface ImportProgress {
  total: number;
  processed: number;
  matched: number;
  unmatched: number;
  duplicates: number;
  errors: number;
  phase: 'scanning' | 'resolving' | 'importing' | 'done';
  currentFile: string;
}

interface ImportResult {
  imported: number;
  matched: number;       // resolved via arXiv/DOI/S2
  unmatched: number;     // imported as local papers
  duplicates: number;    // skipped (already in library)
  errors: Array<{ file: string; error: string }>;
}

---

Phase 5: Import UI

5a. Add "Import PDFs..." button to Sidebar or Toolbar.

5b. New file: src/renderer/components/ImportDialog.tsx

Modal with three phases:

1. Select — "Select Folder" / "Select Files" buttons
2. Processing — progress bar showing resolution phase + current file. All unmatched papers auto-import as local.
3. Done — summary: "Imported X papers (Y matched via arXiv/DOI, Z as local papers, N skipped as duplicates)"

5c. New Zustand store src/renderer/stores/importStore.ts for import state + progress.

5d. Visual differentiation: papers with source: 'local' show "Local paper" badge; DOI papers show DOI link instead of arXiv ID.

---

What's deferred

• Manual metadata editing in import flow
• Folder watching / auto-import
• Drag-and-drop PDF import
• MCP tool for bulk import
• Auto-tagging imported papers by source folder