getscipapers

---

Table of Contents

1. Description
2. Prerequisites
3. Installation
4. Usage
5. Running in GitHub Codespace
6. Docker Container
7. Documentation
8. Remarks

Description

getscipapers is a Python package designed for searching and requesting scientific papers from multiple sources. This project is a work in progress and primarily intended for personal use. It is not a comprehensive solution for accessing scientific papers. Portions of the code were developed with assistance from GitHub Copilot and ChatGPT Codex.

Prerequisites

• (Optional) Install IPFS Kubo to access the Nexus Search database:

  wget https://dist.ipfs.tech/kubo/v0.35.0/kubo_v0.35.0_linux-amd64.tar.gz
  tar -xvzf kubo_v0.35.0_linux-amd64.tar.gz
  cd kubo
  sudo ./install.sh

  Verify installation:

  ipfs --version

  Alternatively, you can interact with the Nexus Telegram bot. To do so, create a Telegram account and obtain your API ID and API hash from my.telegram.org.

• (Optional) Obtain free API keys from Elsevier, Wiley, or IEEE (IEEE support not yet implemented).

• (Optional) Create accounts at Sci-Net, AbleSci, Science Hub Mutual Aid, Z-Library or Facebook to request or download papers/books. For Facebook, join the relevant group after creating your account.

• Install Python (version 3.10 or later).

Installation

It is recommended to use a virtual environment to avoid conflicts with other Python packages. You can use venv or virtualenv. To set up the environment and install dependencies:

# Clone the repository
git clone https://github.com/hoanganhduc/getscipapers.git
cd getscipapers

# Create and activate a virtual environment (change the path if desired)
python -m venv ~/.getscipapers
source ~/.getscipapers/bin/activate

# Upgrade pip and install dependencies
pip install --upgrade pip
pip install build
pip install -r requirements.txt

# Build and install the package in editable mode
python -m build
pip install -e .

# Clean up build artifacts
rm -rf build/ dist/ *.egg-info/
find . -type d -name __pycache__ -exec rm -rf {} +
find . -type f -name "*.pyc" -delete

Usage

To use the Nexus Search database, start the IPFS daemon (if this is your first time, run ipfs init first) in one terminal:

ipfs daemon

In another terminal, use the getscipapers command to search for and request scientific papers. For usage details, run:

getscipapers --help

Quick CLI examples

Common end-to-end invocations of the CLI:

# Search by keyword, limit to 5 results, and download PDFs to the default folder
getscipapers getpapers --search "graph neural network" --limit 5

# Download a specific DOI using Unpaywall first (non-interactive to avoid prompts)
GETSCIPAPERS_EMAIL=you@example.com \
getscipapers getpapers --doi 10.1371/journal.pone.0245581 --db unpaywall --non-interactive

# Process a list of DOIs from a text file, saving PDFs to a custom folder
getscipapers getpapers --doi-file dois.txt --download-folder ./pdfs

# Extract DOIs from a PDF without downloading anything
getscipapers getpapers --extract-doi-from-pdf paper.pdf --no-download

# Show metadata only (no downloads) for a single DOI across all services
getscipapers getpapers --doi 10.48550/arXiv.1706.03762 --no-download --verbose

# Use environment-provided credentials and skip prompts entirely
export GETSCIPAPERS_EMAIL=you@example.com
export GETSCIPAPERS_ELSEVIER_API_KEY=your_elsevier_key
getscipapers getpapers --search "quantum error correction" --non-interactive

# Route every service through a JSON-defined proxy or force direct connections
getscipapers getpapers --doi 10.1371/journal.pcbi.1007128 --proxy ./sample_proxy_config.json
getscipapers getpapers --search "graph neural networks" --no-proxy
getscipapers getpapers --search "graph neural networks" --auto-proxy  # fetch a working proxy if one is missing

Graphical wrapper

Prefer a point-and-click flow? Launch the Tkinter GUI wrapper (works on both Windows and Linux) and trigger searches or DOI list downloads without remembering every flag:

getscipapers gui

The window exposes the same options as the CLI—database selection (toggle one or many services), metadata-only runs, custom download folders, and verbose logging—and reuses the CLI logic under the hood so behavior stays consistent.

The GUI also shows the running version with author and support links, accepts comma-separated DOIs directly or from a text file, disables actions while background work is in progress, offers Clear output and Open folder buttons to reset logs or jump to your downloads, and lets you type or browse for a credentials JSON file when you want to point at a custom location. You can likewise toggle proxy usage, browse for a proxy JSON configuration, or let the app automatically fetch a working proxy so every service call follows the same connectivity rule. Settings live in their own tab so search actions stay uncluttered, an About tab lists the author, description, source code, donation links, and license, and the output pane now has a Detailed log tab with timestamped entries alongside the main console. A dedicated Remove metadata control can scrub PDF metadata after downloads, either overwriting the original or writing a cleaned copy alongside it. Inline validation hints appear beside inputs when something is missing, a shared progress bar and toast notifications make long downloads and proxy discovery visible, and Save/Reload profile buttons persist your preferred folders, credentials, and proxy settings between sessions.

Running getscipapers in GitHub Codespace

The fastest way to run getscipapers is via GitHub Codespaces. This provides a preconfigured environment, eliminating local setup. To use it:

1. Fork the repository to your GitHub account.

2. (Optional) Set up codespace secrets for your API keys and configurations. See .devcontainer/set-secrets.sh for an example using GitHub CLI.

3. Create a new codespace from your forked repository. This will automatically set up the environment with all dependencies installed. You can also use GitHub CLI to create a codespace, for example:

   gh codespace create --repo hoanganhduc/getscipapers --branch master --machine basicLinux32gb

   The basicLinux32gb machine type provides 2 cores, 8GB RAM, and 32GB storage. See GitHub Codespaces documentation for more machine types such as standardLinux32gb, premiumLinux, and largePremiumLinux.

4. Once the codespace is ready, open a terminal and run getscipapers commands directly.

Docker Container for Running getscipapers

Overview

This guide explains how to use getscipapers inside a Docker container. The container includes all dependencies, so you can start downloading scientific papers immediately—no manual setup required.

Quick Start

1. Pull and Run the Prebuilt Image

To get started quickly, pull the latest image from GitHub Container Registry and run it:

docker pull ghcr.io/hoanganhduc/getscipapers:latest
docker run -it --rm -v $(pwd):/workspace ghcr.io/hoanganhduc/getscipapers:latest

This mounts your current directory to /workspace inside the container for easy file access.

2. Build and Run Locally

To build the image yourself:

docker build -t getscipapers .
docker run -it --rm -v $(pwd):/workspace getscipapers

3. Run in Detached Mode with Persistent Storage

To keep the container running in the background and ensure downloads and configuration persist:

docker run -d \
    --name getscipapers-container \
    --restart always \
    -v $HOME/Downloads:/home/getscipaper/Downloads \
    -v $HOME/.config/getscipapers:/home/getscipaper/.config/getscipapers \
    ghcr.io/hoanganhduc/getscipapers:latest

This setup saves downloaded papers and settings to your host machine. Adjust folder paths as needed.

Optional: Integrate with IPFS

To use IPFS with getscipapers, run an IPFS Kubo daemon in a separate container:

docker pull ipfs/kubo:latest
sudo ufw allow 4001
sudo ufw allow 8080
sudo ufw allow 5001

export ipfs_staging=$HOME/.ipfs
export ipfs_data=$HOME/.ipfs

docker run -d \
    --name ipfs_host \
    --restart always \
    -v $ipfs_staging:/export \
    -v $ipfs_data:/data/ipfs \
    -p 4001:4001 \
    -p 8080:8080 \
    -p 5001:5001 \
    ipfs/kubo:latest

This starts the IPFS daemon with persistent storage and required ports. Adjust folder paths as needed.

Running getscipapers Commands

To run getscipapers inside the container:

docker exec -it getscipapers-container getscipapers --help

Optional: Create a Convenience Script

For easier access, create a script at ~/.local/bin/getscipapers:

#!/bin/bash
CONTAINER_NAME="getscipapers-container"

if [ $# -lt 1 ]; then
    echo "Usage: $0 [arguments...]"
    exit 1
fi

COMMAND=("getscipapers" "$@")

if ! command -v docker &> /dev/null; then
    echo "Error: Docker is not installed"
    exit 1
fi

if ! docker ps -q -f name="$CONTAINER_NAME" | grep -q .; then
    echo "Error: Container '$CONTAINER_NAME' is not running"
    exit 1
fi

docker exec -i "$CONTAINER_NAME" "${COMMAND[@]}"

Make it executable:

chmod +x ~/.local/bin/getscipapers

Now you can run getscipapers directly from your terminal:

getscipapers --help

---

For more information, see the official documentation or repository.

Running Locally with Docker

You can run getscipapers locally using Docker without installing Python or dependencies on your system.

1. Ensure Docker is installed.
2. Pull the latest image:

docker pull ghcr.io/hoanganhduc/getscipapers:latest

3. Run the container, mounting a local directory for downloads or configuration:

docker run --rm -it -v /path/to/local/dir:/data ghcr.io/hoanganhduc/getscipapers:latest --output /data

Replace /path/to/local/dir with your preferred local directory.

This setup allows you to use getscipapers in an isolated environment, keeping your files accessible on your host machine.

Documentation

The repository ships with a comprehensive overview of the architecture, configuration model, and command-line workflows in docs/PROJECT_DOCUMENTATION.md. Refer to it for details on how each module collaborates to search, request, and download papers across supported services.

For a browsable HTML site, build the Sphinx documentation:

pip install -r docs/requirements.txt
sphinx-build -b html docs/source docs/_build/html

Open docs/_build/html/index.html in your browser to explore the CLI usage guides, configuration notes, and API reference pages generated from the source modules.

Remarks

• This package is a work in progress and may not always function as expected.
• The code is not yet fully clean or easy to follow.
• Searching with StcGeck is slow and generally best avoided, except in specific scenarios (such as when the Nexus bot is maintained). If you do not wish to use StcGeck, do not start the IPFS Desktop App or run ipfs daemon. In this case, the script will return errors, but StcGeck will not be used.
• Many features in the ablesci, scinet, libgen, wosonhj, and facebook modules depend on Selenium and may break if the target websites change.
  • Some features in the facebook module may work locally but fail in GitHub Codespace or Docker containers (Docker not yet tested). Logging in from Codespace may trigger Facebook verification due to unfamiliar IP addresses. To resolve this, run the Facebook login for the first time with the --no-headless option and use your browser via noVNC to verify your login. Subsequent logins should work without issues. The noVNC access address will look like https://<your-github-codespace-machine-name>-6080.app.github.dev.
  • Uploading to libgen may occasionally fail; retrying usually resolves the issue.
• The nexus module may not work reliably when using a proxy. Issues such as 307 Temporary Redirect errors may occur, and downloads may fail if the Nexus Search server or Telegram bot is unavailable.
• The first time you log in to Telegram (for using Nexus Search bots), you may be required to enter a verification code and password.