Skip to content

README.md

Latest commit

 

History

History
199 lines (152 loc) · 6.78 KB

README.md

File metadata and controls

199 lines (152 loc) · 6.78 KB

LLM Token Explorer

An interactive web application for exploring OpenAI language model token probabilities and generation alternatives. This tool provides real-time visualization of how AI models make token choices during text generation.

LLM Token Explorer React Node.js OpenAI

Features

Core Functionality

  • Token Probability Visualization: View actual token probabilities returned by OpenAI's API using the logprobs parameter
  • Interactive Token Exploration: Click on any generated token to see alternative choices the model considered
  • Color-coded Probability Display: Visual representation of token confidence levels from high (green) to low (red)
  • Real-time Generation: Watch as the model generates text with visible probability distributions

Model Controls

  • Temperature Adjustment: Control randomness in token selection (0.0 to 2.0)
  • Max Tokens Setting: Limit response length
  • Model Selection: Choose between GPT-3.5 Turbo and GPT-4
  • Token Counter: Estimate API usage before generation

Security & Access

  • Passkey Authentication: Secure access control for shared environments
  • Environment-based Configuration: API keys and secrets managed through environment variables
  • Session Persistence: Login state remembered across browser sessions

User Experience

  • Responsive Design: Optimized for desktop, tablet, and mobile devices
  • Compact Interface: Side-by-side prompt and response layout
  • Copy Functionality: Easy copying of generated responses
  • Performance Metrics: Response time and token usage tracking

Technology Stack

  • Frontend: React 18, TypeScript, Vite
  • UI Components: shadcn/ui, Radix UI primitives
  • Styling: Tailwind CSS with custom theming
  • Backend: Node.js, Express, TypeScript
  • API Integration: OpenAI API with logprobs support
  • State Management: TanStack Query for data fetching
  • Build & Dev: Vite with HMR support

Quick Start

Prerequisites

  • Node.js 18+ installed
  • OpenAI API account and key
  • Git (for cloning)

Installation

  1. Clone the repository

    git clone <your-repo-url>
cd llm-token-explorer

  2. Install dependencies

    npm install

  3. Set up environment variables Create a .env file in the root directory:

    OPENAI_API_KEY=your_openai_api_key_here
EXPLORER_PASSKEY=your_custom_passkey_here

  4. Start the development server

    npm run dev

  5. Access the application Open your browser to http://localhost:5000

Usage

Authentication

  1. Enter your custom passkey on the login screen
  2. The passkey is set via the EXPLORER_PASSKEY environment variable

Exploring Token Probabilities

  1. Enter a prompt in the left panel
  2. Adjust model parameters (temperature, max tokens, model)
  3. Click "Generate" to get the response
  4. View color-coded tokens in the response panel
  5. Click any token to see alternative choices and their probabilities

Understanding the Interface

  • Green tokens: High probability (>70%)
  • Yellow tokens: Medium probability (30-70%)
  • Red tokens: Low probability (<30%)
  • Token alternatives panel: Shows top 5 alternative tokens with probability bars

API Integration

The application leverages OpenAI's chat completions API with specific parameters:

{
  model: "gpt-4", // or "gpt-3.5-turbo"
  messages: [...],
  temperature: 0.7,
  max_tokens: 150,
  logprobs: true,
  top_logprobs: 5
}

The logprobs parameter returns probability information for each token, which is then processed and visualized in the interface.

Project Structure

├── client/                 # Frontend React application
│   ├── src/
│   │   ├── components/     # Reusable UI components
│   │   ├── pages/          # Page components
│   │   ├── lib/           # Utility functions and API calls
│   │   └── hooks/         # Custom React hooks
├── server/                # Backend Express application
│   ├── index.ts           # Server entry point
│   ├── routes.ts          # API route definitions
│   ├── openai.ts          # OpenAI integration
│   └── storage.ts         # Data persistence layer
├── shared/                # Shared TypeScript types and schemas
└── package.json           # Dependencies and scripts

Development

Available Scripts

  • npm run dev: Start development server with hot reload
  • npm run build: Build for production
  • npm run type-check: Run TypeScript type checking

Key Components

  • InputPanel: Prompt input and model parameter controls
  • ResultsPanel: Token visualization and probability display
  • Home: Main application logic and state management

Deployment

Environment Variables

Set these in your deployment environment:

  • OPENAI_API_KEY: Your OpenAI API key
  • EXPLORER_PASSKEY: Custom passkey for access control

Production Build

npm run build

The application serves both frontend and backend from a single Express server, making deployment straightforward.

Use Cases

Educational

  • AI/ML Courses: Demonstrate how language models make token choices
  • Research Projects: Analyze model behavior and confidence patterns
  • Interactive Learning: Hands-on exploration of AI text generation

Professional

  • Model Evaluation: Compare token probability distributions across models
  • Prompt Engineering: Understand how different prompts affect model confidence
  • AI Transparency: Show clients/stakeholders how AI systems make decisions

Contributing

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

  • OpenAI for providing the API with logprobs support
  • shadcn/ui for the excellent component library
  • The React and TypeScript communities for robust tooling

Support

If you encounter issues or have questions:

  1. Check the existing issues in the repository
  2. Create a new issue with detailed information
  3. Include steps to reproduce any bugs

Built with dedication for exploring the inner workings of AI language models.