-
Notifications
You must be signed in to change notification settings - Fork 99
feat(llms.txt): Add llms.txt Generation for LLM-Friendly Site Documentation #1836
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Question: why do we want this feature? |
The purpose of llms.txt is to provide Large Language Models (LLMs) with a structured, machine-readable guide to our website's most important content, summaries, and structure, helping AI understand, access, and accurately represent the site's information, improving discoverability and ensuring better, more relevant AI-generated answers and citations. It's like a curated highlight reel for AI, directing it to valuable resources. This will increase our discoverability with GEO and allows us (if we choose to) in the future to have a Slackbot that members can ask about US-RSE questions. Plus, it costs us nothing to add it. |
| pull_request: | ||
| branches: | ||
| - main | ||
| workflow_dispatch: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you actually want this line? If so, I don't think it's complete. This is what I normally do:
workflow_dispatch:
inputs:
git-ref:
description: Git Hash (Optional)
required: false
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Gotcha! I made some updates to the GH Actions to make the workflow more robust.
…improved validation for llms.txt generation
Add llms.txt Generation for LLM-Friendly Site Documentation
Summary
This PR implements automatic generation of an
llms.txtfile from the US-RSE Jekyll site, making the website content more accessible to Large Language Models (LLMs) following the llmstxt.org specification.Changes
This PR adds three new files:
1.
.github/workflows/llms-txt.yml- GitHub Actions Workflowmain, pull requests, and manual workflow dispatchllms.txt2.
scripts/generate_llms_txt.py- Generation Script_sitedirectory after Jekyll buildllms.txtwith organized sections3.
scripts/requirements.txt- Python Dependenciespyyaml>=6.0(for parsing_config.yml)Features
Use Case
The
llms.txtfile provides LLMs with a structured, text-based overview of the entire US-RSE website, making it easier for AI assistants to understand and reference site content when answering questions about US-RSE.Testing
The workflow can be manually triggered via workflow dispatch to test the generation process. Generated files are uploaded as artifacts for review.
Technical Details