LLM Generator
Generate llms.txt files for websites to improve their visibility to large language models
Project Overview
LLMGenerator is a service that generates llms.txt files for websites to improve their visibility to large language models. The service uses Firecrawl's map + scrape approach with OpenAI enhancement to create high-quality, LLM-optimized content.
What is llms.txt?
llms.txt is a standardized format for making website content more accessible to Large Language Models (LLMs). It provides:
- llms.txt: A concise index of all pages with AI-enhanced titles and descriptions
- llms-full.txt: Complete content of all pages for comprehensive access
What is llms.txt?
llms.txt is a standardized format for making website content more accessible to Large Language Models (LLMs). It provides:
- llms.txt: A concise index of all pages with AI-enhanced titles and descriptions
- llms-full.txt: Complete content of all pages for comprehensive access
Key Features
- 🚀 Fast Generation: Direct map + scrape approach with background processing
- ⚡ Reliable Background Jobs: Trigger.dev integration for long-running tasks with automatic retries
- 🤖 AI-Enhanced Content: OpenAI-powered titles and descriptions for paid plans
- 💰 Cost-Optimized Tiers: 95% cheaper than competitors - Free and paid plans with different generation methods
- 📊 Comprehensive Analytics: Track generation stats, mapped URLs, and usage
- 🔄 Real-time Updates: Live progress tracking with dedicated status pages
- 🛡️ Enterprise Ready: Rate limiting, API keys, and robust error handling
- 📖 API Documentation: OpenAPI/Swagger 3.0 with interactive documentation
- 🌐 Background Processing: Trigger.dev handles long-running tasks with retries
- 🔒 Flexible Pricing: Credit packages (never expire) + subscription plans
Tech Stack
Frontend
- React 19 with TypeScript
- Vite for fast builds
- Tanstack Router & Query
- Tailwind CSS & shadcn/ui
- React Hook Form with Zod validation
Backend
- Hono on Cloudflare Workers
- Cloudflare D1 (SQLite) with Drizzle ORM
- Better Auth for authentication
- Trigger.dev for background jobs
- tRPC for type-safe APIs
Integrations
- Firecrawl API for web scraping
- OpenAI API for content enhancement
- Stripe for payments
- Cloudflare KV for session caching
- Sentry for error tracking
Development & Testing
- Biome for linting/formatting
- Bun for package management
- Wrangler for deployments
- Vitest with 135 tests
- OpenAPI/Swagger documentation
Generation Methods
🆓 Simple Generation (1x Credits)
- ✓ Fast generation (15-30 seconds)
- ✓ Uses Firecrawl's /llmstxt endpoint
- ✓ Cost-effective for free tier users
- ✓ Locked for free plans, optional for paid
⭐ Enhanced Generation (2x Credits)
- ✓ AI-enhanced titles (3-4 words) and descriptions (9-10 words)
- ✓ Uses map + scrape + OpenAI integration
- ✓ Premium quality for paid plans
- ✓ Default for paid users
Business Model & Revenue
This SaaS platform serves developers, agencies, and businesses who want to improve their website's visibility to large language models. The credit-based pricing system provides flexibility while maintaining healthy margins.
Revenue Streams:
- Credit packages (pay-as-you-go)
- Subscription plans (monthly recurring)
- API access (all paid tiers)
- Enterprise custom solutions
Competitive Advantages:
- 95% cheaper than competitors
- Credits never expire (packages)
- API-first architecture
- Real-time background processing
Project Timeline
Start Date: June 2025
Status: 🔨 Work in Progress (v2.0.0)
Type: Personal SaaS Project