LLMs Optimization (AI SEO)
Get indexed in AI (LLMs) world
Hi there! We all know that LLMs are becoming a key information and education source. Want your project to get noticed in your client’s ChatGPT response? Let’s see how to make that happen!
Let’s figure out how to work with big names like ChatGPT, Perplexity, and Gemini, and set you up for success. Ready to get started?
David Pierce captures the shift of search patterns perfectly in his recent article, "Obviously the Future of Google Search" for The Verge. He explains that Google’s AI Overviews are just the beginning — the company is already preparing for a world where all search is AI search.
For content creators, that means it's no longer just about ranking on page one of Google — it's about being summarized, cited, and surfaced by AI models.
This Substack is reader-supported. To receive new posts and support my work, consider becoming a free or paid subscriber.
# What’s LLM Optimization (and Why It’s Urgent)?
LLM optimization is all about making your site or content easy for big language models (like ChatGPT, Perplexity, or Gemini) to understand, index, and quote. It’s not just about search rankings—it’s about how AI answers when someone asks, “What’s [your project]?” or “What are the best tools for [your niche]?”
Based on StatCounter’s highlights, recent data shows ChatGPT’s dominance in the AI model market, as shown below:
According to the article on businesstechweekly.com and new data from Similar’s latest market analysis, ChatGPT holds approximately 80.1% of the AI search market, outperforming competitors like DeepSeek (6.5%) and Google AI Search (5.6%).
Google Search: Despite the rising popularity of AI tools, Google remains the dominant player in traditional search, processing around 8.7 billion queries daily and receiving over 85.5 billion visits monthly.
But this is just the beginning, AI is rapidly gaining momentum and gearing up to compete with traditional search engines.
Growth in AI Chatbot Traffic: According to a fresh study from onelittleweb.com, chatbot traffic grew by 81% from April 2024 to March 2025, reaching 55.2 billion visits, with ChatGPT leading at 47.7 billion visits.
In the article "The Future is Now: Why AI-Driven Search Will Transform Your Marketing," published on March 14, 2025, on Pallas Advisory, they mentioned that by 2026, traditional search could lose 25% (if not more) of its volume to AI alternatives.
Current data indicates significant growth in the use of AI tools like ChatGPT and a gradual decline in the traditional search share. This shows AI, especially ChatGPT, is taking over, making it easier and quicker to find what we need online.
# AI Traffic Real Life Case
Some projects have already raked in thousands of clicks just by becoming an Al favorite.
One standout example is Traveler Pro a luggage store, which gained attention after being featured and recommended by AI-driven platforms. Their detailed and helpful product reviews, combined with AI-powered marketing strategies, have helped them reach a wider audience and boost their online traffic dramatically. This shows how integrating AI can transform traditional retail businesses and amplify their growth.
# How LLMs Actually Find and Quote You
LLMs like Perplexity, ChatGPT, Gemini, and next-gen models decide which content to quote. Your site needs to be machine-readable and spot-on to get noticed!
ChatGPT
You’ll usually get a summary in ChatGPT’s own words, sometimes with references if web browsing is used, but rarely with direct quotes unless you ask. ChatGPT can quote sources if you specifically request direct quotes or if it retrieves something from the web. Quotes will usually be brief, and ChatGPT cites the source (e.g., “According to the Creators AI website: ‘Our platform helps creators…’”).
Perplexity
Perplexity always provides a brief, sourced summary, pulling from multiple web sources in real-time. ou get bullet points, short summaries, and always a list of source links at the bottom. Perplexity is more likely to include direct quotes and always cites the source inline (with numbers or hyperlinks), so you can see exactly where the info is from. It can also fetch longer direct quotes if you ask.
Sharing is caring! Refer someone who recently started a learning journey in AI. Make them more productive and earn rewards!
Gemini
Gemini summarizes info like ChatGPT, but since it’s Google’s model, it often pulls the latest web info and may show a “snapshot” or preview of web results. You get a short, easy-to-read summary, sometimes with links or sources depending on how Gemini is set up (web-enabled or not). Gemini can provide direct quotes if you specify, and will typically attribute quotes to the source (e.g., “According to TheCreatorsAI.com: ‘[quoted text]’”).
TL;DR
Perplexity is best known for its transparent quoting and sourcing, functioning like a mini research assistant by providing summaries with direct references and often including actual quotes from web sources. In contrast, ChatGPT and Gemini typically summarize information in their own words, offering clear and concise overviews, but will quote and cite specific sources if explicitly requested or when web search is enabled.
# llms.txt—The Robots.txt for the AI Era
llms.txt is a special file you can add to your website, similar to the "robots.txt" file used for search engines. It’s a guide for LLMs (like ChatGPT, Perplexity, or Gemini) to understand which parts of your site to read, which to quote in their answers, and which to skip.
How Does It Work?
To help large language models (LLMs) like ChatGPT provide accurate and relevant answers about your site, simply create a text file called “llms.txt” and upload it to your website’s root folder. This file lets you set clear rules for LLMs, such as highlighting which content to focus on (like your blog posts), what information to quote (like key statistics), and what to ignore (like contact forms or ads). By giving LLMs this guidance, you boost your chances of being cited correctly and ensure better, more useful responses about your content.
A couple of things to keep in mind: not every LLM supports llms.txt yet since it’s a pretty new idea, and you’ll need to update the file regularly as your site changes. Also, while this helps steer LLMs in the right direction, it’s not a guarantee you’ll always get quoted. LLMs still pick what’s most relevant and high quality.
# Step-by-Step Tutorial on Indexing Your Project with llms.txt
1. Create the File
To create an llms.txt file, open a text editor such as Notepad or VS Code and add your rules for LLM crawlers in a simple text format. For example, thecreatorsai.com template:
# Creators' AI
> Creators' AI is a Substack publication offering AI insights, tools, and guides for creators and entrepreneurs.
## About
- [About Creators' AI](https://thecreatorsai.com/about): Learn about the mission and team behind Creators' AI.
## Key Sections
- [Home](https://thecreatorsai.com/): Main page featuring the latest posts and updates.
- [Tools](https://thecreatorsai.com/t/tools): Explore AI tools and resources curated for creators.
- [Creators Cases](https://thecreatorsai.com/t/cases): Real-world applications and case studies of AI in creative endeavors.
- [News Feed](https://thecreatorsai.com/t/news): Stay updated with the latest news and developments in AI.
- [Sponsor](https://thecreatorsai.com/sponsor): Information on sponsorship opportunities.
- [Archive](https://thecreatorsai.com/archive): Access the complete list of past articles and posts.
## Featured Articles
- ["AI Tools I Can't Live Without"](https://thecreatorsai.com/p/ai-tools-i-cant-live-without): A comprehensive guide to essential AI tools for creators.
- ["OpenAI’s U-turn to Non-profit | Weekly Edition"](https://thecreatorsai.com/p/openais-u-turn-to-non-profit-weekly): Analysis of OpenAI's structural changes and their implications.
- ["Are You Thinking WITH AI?"](https://thecreatorsai.com/p/are-you-thinking-with-ai): A guide on integrating AI into your thought processes and note-taking.
- ["Why Activate ChatGPT Memory"](https://thecreatorsai.com/p/why-activate-chatgpt-memory): Insights into ChatGPT's memory feature and its benefits.
- ["23 GPTs For Marketing & Startups"](https://thecreatorsai.com/p/23-gpts-for-marketing-and-startups): A curated list of GPTs beneficial for marketing and startup endeavors.
## Tags & Topics
- AI Tools
- AI News
- Case Studies
- Tutorials
- Productivity
- Content Creation
- Marketing
- Startups
- ChatGPT
- GPTs
- Memory
- Monetization
- Automation
- No-Code
- Prompt Engineering
- AI Ethics
- AI Safety
- AI in Business
- AI in Education
## Subscription
- [Subscribe to Creators' AI](https://thecreatorsai.com/): Join over 10,000 subscribers receiving weekly AI insights and tools.
## Contact
- [Contact Creators' AI](https://thecreatorsai.com/about): Reach out for inquiries or collaborations.2. Save and Upload the File
Save the file as llms.txt (making sure it’s not named llms.txt.txt), then use your website hosting panel, FTP, or file manager to upload it to the root directory of your website.
3. Verify Accessibility
Open your browser and navigate to https://thecreratorsai.com/llms.txt; you should see the content of your file displayed in the browser—if you don’t, double-check the file permissions and make sure it’s placed correctly in your site’s root directory.
4. Test with LLMs and Crawlers
Currently, there’s no universal testing tool for llms.txt, but you can monitor your server logs for crawler user-agents such as openai-crawler and google-extended to see if they are respecting your llms.txt rules; as adoption of the standard increases, major AI companies like OpenAI, Google, and Perplexity are expected to honor these directives more consistently. Or just use them.
5. Keep the File Updated
Update your llms.txt file whenever your website structure changes or if you want to adjust your LLM crawling preferences, and be sure to check documentation from LLM companies regularly for any updates to user-agent names or changes in standards.
Also, here is an example of how to create llms.txt in less than a minute with
LLMs.txt generator + Firecrawl
Firecrawl actively scans the most relevant web sources in real-time, while the generator processes this raw data to ensure consistency. It then compiles the refined information into an LLMs.txt file, which is subsequently uploaded and used to train large language models, enabling them to accurately respond to client queries.
You will find this generated llms.txt file at the end of the article.
Thanks for reading! Feel free to share it.
# My LLM Optimization Testing
You can test how AI sees your project with three simple prompts:
What’s [your project]?Best [niche] tools in 2025?Compare [your project] vs. [competitor].
A simple way to see if Perplexity or ChatGPT are noticing your site. Go to their website or app and ask a question that matches your content. Check the answer and see if your site gets mentioned. Look for small links or numbers that show where the info is coming from, and see if your website’s address shows up. If Perplexity are quoting you, great, your setup is working. If not, you might need to improve your content or get more good backlinks.
# LLMs Optimization is a new SEO
The old SEO game, like fighting for Google’s first page, is fading. Now, the goal is to be the clearest, most concise answer in ChatGPT or Perplexity. When LLMs mention your site, users find you easily, bringing pure traffic and trust! Also, AI agents may use information from LLMs and interact with your site and creating a snowball of information about your project.
If you lost your chance to become #1 in the SEO game, you may have a chance now with LLMs indexing or AI. If you are building organic growth for a product or exploring new services with AI, this may be the right time to double down on it
As I promised, generated LLMs.txt :
*Note: This is an incomplete result, please enable full generation by entering a Firecrawl key.
# https://thecreatorsai.com/ llms.txt
- [AI Tools and Insights](https://thecreatorsai.com/): Explore AI tools, news, and insights for creators.
- [AI Tools Archive](https://thecreatorsai.com/archive): Explore the latest AI tools, tutorials, and discussions.
- [Creators AI Community](https://thecreatorsai.com/about): Explore AI tools and community for creators and entrepreneurs.
- [AI in Healthcare](https://thecreatorsai.com/p/chatgpt-will-be-your-doctor): Exploring AI's role in healthcare and diagnostics.
- [AI Health and Tools](https://thecreatorsai.com/p/the-era-without-doctors-weekly-edition): Weekly AI updates on healthcare, tools, and innovations.
- [ChatGPT Memory Benefits](https://thecreatorsai.com/p/why-activate-chatgpt-memory): Explore how ChatGPT Memory enhances personal insights and feedback.
- [OpenAI's Nonprofit Shift](https://thecreatorsai.com/p/openais-u-turn-to-non-profit-weekly): OpenAI's shift to nonprofit amidst legal challenges and AI advancements.
- [AI Startup Ideas](https://thecreatorsai.com/p/ai-startups-ideas-by-yc-summer-25): Explore innovative AI startup ideas from Y Combinator's latest insights.
- [MCP Guide for Founders](https://thecreatorsai.com/p/mcp-full-guide-for-founders-and-makers): Comprehensive guide on Model Context Protocol for creators.
- [Meta AI App Overview](https://thecreatorsai.com/p/meta-ai-app-the-new-chatgpt-competitor): Explore Meta's new AI app, a ChatGPT competitor.This article was first published in the Creators AI newsletter. View the original edition.
