Google has quietly updated its list of user-triggered fetchers with new documentation for Google NotebookLM. The importance of this seemingly minor change is that it’s clear that Google NotebookLM will not obey robots.txt.
Google NotebookLM
NotebookLM is an AI research and writing tool that enables users to add a web page URL, which will process the content and then enable them to ask a range of questions and generate summaries based on the content.
Google’s tool can automatically create an interactive mind map that organizes topics from a website and extracts takeaways from it.
User-Triggered Fetchers Ignore Robots.txt
Google User-Triggered Fetchers are web agents that are triggered by users and by default ignore the robots.txt protocol.
According to Google’s User-Triggered Fetchers documentation:
“Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules.”
