Google has clarified that Mediapartners-Google should no longer be treated as merely the AdSense crawler. Crawling preferences aimed at that user agent can affect multiple Google advertising products, including Google Ad Manager, making old robots.txt assumptions potentially broader in impact than publishers and technical teams realized.
The change appears in Google’s official crawling documentation changelog dated September 17, 2026. Google says it generalized the Mediapartners-Google documentation specifically to make clear that the crawler affects “various ad-related Google products,” not just AdSense. The company also renamed the relevant documentation heading from “AdSense” to Mediapartners-Google.
Google gives both AdSense and Google Ad Manager as examples of affected products. The clarification does not announce a new crawler or say that its production behavior suddenly changed on September 17. Instead, Google is correcting the mental model site owners should use when deciding whether to allow or restrict this user agent.
Mediapartners-Google is an advertising crawler, not an AdSense-only crawler
Google’s current special-case crawler reference says Mediapartners-Google visits participating sites to provide relevant ads. More importantly, it now states that crawling preferences addressed to the user agent affect various advertising-related Google products, explicitly naming AdSense and Google Ad Manager.
That wording changes how publishers should interpret a robots.txt rule written years ago under the assumption that it controlled only AdSense contextual crawling. A deliberate restriction may have consequences for other Google advertising systems that rely on the same crawler identity.
The distinction is operational rather than semantic. If several products share a crawler, blocking the crawler is effectively a shared infrastructure decision. The site cannot assume that the rule remains isolated to the product name historically associated with the documentation section.
Google changed the documentation because the old label was too narrow
Google is unusually direct about why it made the update. The September 17 changelog says the documentation was generalized because preferences addressed to Mediapartners-Google affect multiple ad-related products. It also says the old section heading was changed from “AdSense” to the crawler’s actual user-agent name.
That is a useful documentation pattern because infrastructure names often outlive the products with which users first associate them. A crawler can become shared plumbing across several services even while old tutorials, configuration files and internal runbooks continue describing it through a single product.
For publishers, the immediate task is therefore not to assume that Google launched a new ad-crawling system. It is to audit whether existing crawler rules were written with an obsolete product-specific assumption.
A robots.txt rule can have a wider advertising blast radius than expected
Large publishers often maintain detailed robots.txt files containing rules accumulated over many years. Some are designed for search crawlers, some for AI systems, some for commercial bots and some for specialized Google crawlers. The reason for a particular directive can become unclear long after the person who added it has moved on.
A rule such as Disallow for Mediapartners-Google may therefore be sitting in production because a team once wanted to control AdSense behavior. Google’s clarification means that team should now evaluate the rule against the wider set of advertising products that may depend on the crawler.
This does not mean every restriction automatically breaks every Google advertising feature. Google’s documentation says the preferences affect various ad-related products, not that every product behaves identically or that a single blocked URL necessarily produces a specific revenue outcome. The correct conclusion is that the scope is broader than AdSense and should be reviewed accordingly.
The global robots.txt wildcard does not control this crawler
There is another technical detail that makes Mediapartners-Google different from ordinary crawling assumptions. Google classifies it as a special-case crawler and says the global * user-agent group in robots.txt is ignored for this crawler.
Site owners who want to express preferences for it need to use the specific Mediapartners-Google token. Google’s documentation provides an example in which a dedicated group allows one archive path while disallowing another.
This means a publisher cannot reliably infer Mediapartners-Google behavior simply by looking at a generic User-agent: * block. An audit needs to search specifically for rules targeting Mediapartners-Google and then determine why those rules exist.
This is not a Google Search ranking change
Because the update appears in Google’s crawling infrastructure documentation, it can easily be mistaken for an SEO crawling change. Google’s explanation is about advertising products, not a new organic Search ranking factor or a change to Googlebot indexing behavior.
Mediapartners-Google is separate from the common Googlebot crawler used for Search. A site can therefore have one set of preferences for Search crawling and another for the advertising crawler.
The practical overlap with SEO comes through technical governance. The same robots.txt file can contain directives for many Google systems, so an SEO team may own or deploy a configuration whose effects extend into advertising operations. That makes cross-team review more important than treating robots.txt as a Search-only file.
Publishers should audit the intent behind existing rules
The most useful response to the documentation change is a configuration audit rather than an immediate removal of restrictions. A publisher should identify whether it has a dedicated Mediapartners-Google group, determine when and why it was added, and check which paths are allowed or blocked.
Advertising and monetization teams should be involved because the rule can affect systems beyond AdSense. Google Ad Manager is now explicitly named in Google’s own explanation, so a crawler policy should not be changed solely by an SEO or infrastructure team without understanding the advertising setup.
Teams should also distinguish deliberate restrictions from legacy copy-and-paste rules. Robots.txt files are frequently inherited from templates or previous site versions. A directive that once solved a specific monetization problem may no longer match the publisher’s current stack.
The clarification reflects Google’s broader crawling infrastructure model
Google has increasingly documented crawling as shared infrastructure rather than something belonging only to Search. In late 2025, the company moved crawler documentation away from Search Central into a dedicated crawling infrastructure site, explaining that its crawlers support products including Search, Shopping, News, Gemini and AdSense.
The Mediapartners-Google change follows the same logic. Product-centric labels can obscure the fact that a crawler identity may serve several systems. Naming the documentation after the user agent rather than AdSense makes the technical boundary clearer.
That is especially relevant as Google’s product portfolio becomes more interconnected. Site owners increasingly need to understand not merely whether a Google crawler is allowed, but which Google products depend on that crawler and what functionality a restriction may affect.
Crawler governance is becoming a cross-functional responsibility
For many organizations, robots.txt historically belonged to SEO. That ownership model is increasingly incomplete. A single configuration can affect search indexing, advertising, AI features, shopping systems and other automated services depending on the user agent involved.
The September 17 update provides a concrete example. A developer may see Mediapartners-Google and think “AdSense.” An ad operations team may rely on Google Ad Manager. An SEO team may control the robots.txt deployment. The technical rule sits at the intersection of all three.
Better governance means documenting each crawler-specific rule with an owner and business rationale. When Google changes or clarifies the product scope of a crawler, teams can then identify the affected configuration without reconstructing years of historical decisions.
The safest interpretation is broader scope, not newly announced behavior
Google’s changelog describes this as a documentation generalization. That wording matters because it does not establish that Mediapartners-Google began serving Google Ad Manager for the first time on September 17. Google is saying that the old AdSense-centric documentation did not accurately communicate the crawler’s broader role.
Publishers should therefore avoid treating the update as evidence of a sudden algorithmic or advertising-platform change. The concrete new information is that Google now officially documents multiple ad products behind the crawler and wants site owners to configure it with that broader scope in mind.
That clarification is enough to justify reviewing old rules. A crawler directive written for one assumed product can become a hidden dependency for several products when the underlying infrastructure is shared.
The crawler name now matters more than the old product label
The practical lesson from Google’s update is simple: configure Mediapartners-Google according to what the crawler actually serves, not according to an outdated assumption that its only purpose is AdSense.
Google now explicitly lists AdSense and Google Ad Manager as examples of products affected by crawling preferences addressed to this user agent. It has renamed the documentation section accordingly and retained the crawler as a special-case user agent with its own robots.txt behavior.
For publishers, that makes a small documentation edit worth an infrastructure review. Blocking Mediapartners-Google may still be intentional, but the decision should be made with an understanding that the consequences can extend beyond AdSense.