Why Google ignores many backlinks, the 4 rules robots.txt reads, AI Overviews on 88% of buyer searches, server logs for AI bots, and a GTM fix for GA4 bot data.
This week’s HighDegree* Marketing by SerpLabSEO issue is brought to you by our official r/SerpLabeSEO sub-reddit; Join here and start fresh marketing & SEO discussions.
Hi Marketers,
Welcome to the new issue of HighDegree*: Cutting Through the Noise in SEO & Digital Marketing
This week covers what Google, AI bots, and your analytics can see on your site. Cyrus Shepard explains why Google ignores many backlinks and how to earn the ones it counts. Google’s Gary Illyes walks through the four rules a robots.txt file can hold. Semrush finds AI Overviews on 88% of the commercial searches it tracked, with review queries rising fastest. A Moz guide shows how server logs reveal whether AI bots can reach your pages, and Dana DiTomaso shares a Google Tag Manager setup that moves bot traffic out of GA4. Each story ends with something you can check on your own site this week.
Let’s jump in.
➞ Digital Marketing This Week
- Why Google Ignores Most Backlinks and How to Make Yours Count
- What Google Says You Can Put in a Robots.txt File
- AI Overviews Now Show on 88% of Buyer Searches
- What Your Server Logs Reveal About AI Crawlers
- How One GTM Variable Keeps Bots Out of GA4
➞ Subscribe
If you are not subscribed to HighDegree* yet, please subscribe to us by clicking the “Subscribe” button below. I will post every issue to HighDegree* substack version, so you can get it instantly in your inbox every time I hit publish!
➞ Why Google Ignores Most Backlinks and How to Make Yours Count
Cyrus Shepard of Zyppy argues that Google ignores or discounts many of the links you earn, even when you follow the usual anchor text advice. Drawing on Google patents, the API leak, and trial testimony, he lists what makes a link count. One trusted link can carry more weight than a hundred weak ones, and Google’s leaked documentation calls anchors with PageRank above 51,000 “the best anchor.” The words around a link matter too, so a sentence describing what you do beats a bare “good plumber” link.
Deliberately varied anchors such as “cheap flights” and “affordable flights” can be flagged, while natural anchors like “this flight-price study” hold up. Sitewide sidebar links barely count, since Google keeps at most 200 anchors from one domain. Redirecting an old article to a broader category page can also cost you its links. Before your next outreach push, aim for one link from a respected site, ask for it high in the body copy, and check that the linking page, the anchor, and your page all cover the same topic.
Read the full report from zyppy.com ➞
Links only help when crawlers can reach the pages they point to, and Google just explained the file that controls that access.
➞ What Google Says You Can Put in a Robots.txt File
Google’s Gary Illyes and Cherry Prommawin used the second episode of their Non-Canonical Cabinet video series to explain what a robots.txt file can hold. Google’s parser follows the RFC 9309 standard, reads the file line by line, and understands four rules: user-agent, allow, disallow, and sitemap. Anything else is dropped without warning, so a line like “don’t crawl secret page” does nothing and the page may still be crawled. Comments starting with # are ignored.
User-agent names can only contain letters and dashes, so an emoji there fails. For paths with emojis or non-English characters, Google encodes them for you, but older crawlers may not, so write the percent-encoded version. Illyes also said robots.txt only controls access. It cannot stop a company from training on content it already collected, and blocking a specific AI crawler means naming its user-agent token. Check your file this week: remove anything that isn’t one of the four rules, confirm each AI crawler you block by name, and test real URLs against your rules.
Read the full report from seroundtable.com ➞
Once crawlers get in, AI Overviews decide what buyers see first, and Semrush measured how far they now reach.
➞ AI Overviews Now Show on 88% of Buyer Searches
Semrush tracked 400 commercial keywords in banking, software, and ecommerce from September 2025 to August 2026, each with a modifier buyers use near a decision: alternatives, brand versus brand, best X for Y, pricing, and reviews. AI Overviews appeared on 88% of them by August, up from 45.75%, and on 97.08% once weighted by search volume. Review searches jumped from 30% to about 90%, the biggest rise, while “best X for Y” queries finished highest at 96.12%.
Ecommerce went from 32.62% to 85.11%, and all three industries ended between 85% and 91%. A few segments trail, such as ecommerce price searches at 71.43%, up from 4.76% a year earlier. In a “mailchimp vs klaviyo” example, the overview drew on Klaviyo’s own page, Zapier, and a YouTube video, so your comparison and pricing pages still feed the answer alongside review sites and creators. Track which sources get cited for your top buyer queries, keep prices and feature lists current, and ask happy customers to review you on the sites those overviews cite.
Read the full report from semrush.com ➞
To be cited in those answers, AI bots first have to fetch your pages, and your server logs show whether they can.
➞ What Your Server Logs Reveal About AI Crawlers
Zach Chahalis of iPullRank says many teams spend on AI search without checking whether AI bots can reach their key pages. Server logs show it, because each line records the IP address, time, page, status code, bytes sent, and user-agent. In his sample, ChatGPT-User gave up on a pricing page after 31 seconds and logged a 499 (client closed request), GPTBot got a near-empty page because the content loaded with JavaScript, and a firewall refused Claude-User with a 403 in six milliseconds.
Ask your developer or host for a few hundred lines from your content delivery network (CDN) first, then two weeks of data. Open them in a tool such as Screaming Frog’s Log File Analyser with bot verification turned on, since anyone can fake a user-agent. Blocking GPTBot only stops training, but blocking OAI-SearchBot can keep you out of ChatGPT search answers. In an iPullRank and Profound study of 700,000 pages, pages that often timed out for OpenAI got 18 times fewer citations. Start with 403s and repeated 499s on your most important pages.
Read the full report from moz.com ➞
Bots that do reach your site can also end up in your analytics, and Dana DiTomaso found a way to move them out.
➞ How One GTM Variable Keeps Bots Out of GA4
Dana DiTomaso noticed a Realtor client in Edmonton suddenly getting traffic from China and Singapore, with sessions that opened every page two to ten times. These looked like AI scrapers running headless browsers, and GA4 counted them as engaged sessions. GA4’s built-in filter only removes known bots, and its own filters can’t send traffic to another property.
Working with Claude, she tested 12 bot checks through Google Tag Manager (GTM). About 96% of sessions with an Asia time zone failed, against 2% for everyone else. The clearest sign was the graphics card, because bots on servers report SwiftShader, Google’s software renderer. Her GTM variable sorts visitors into physical_gpu, virtual_gpu, or gpu_not_found and uses the result to pick the GA4 measurement ID, so suspected bots land in a separate property. You can use the same value to keep bots out of Google Ads and Meta conversions. Expect reported traffic to drop, so test in a spare GA4 property first and keep gpu_not_found visitors in production until you’ve reviewed them.
Read the full report from kpplaybook.com ➞
➞ From Google
Everything from Google search this week —
→ Google’s Q3 2026 Earnings Call: Search & AI are merging into one unified product. (mariehaynes.com) – Marie Haynes reads the call as a push toward agentic search, quoting Sundar Pichai on building one experience across AI Overviews, AI Mode, and traditional Search.
→ Google: AI Mode’s lower monetization slowed Search ad growth (searchengineland.com) – Search revenue grew 12% to $63.35 billion in Q3, down from 17% in Q2, and Google named lower AI Mode monetization as one reason.
→ Ads in AI Mode expand to new markets. (blog.google) – Google’s own post adds multiturn ads on US follow-up queries and says sponsored and organic product listings in AI Mode already get similar click-through rates.
→ Google AI Mode Tests Blending All Products With Sponsored Products (seroundtable.com) – Google may be mixing sponsored product ads into the “All products” results in AI Mode, and screenshots make it hard to tell which listings are paid.
→ Google AI Mode sends 57% of product links to Google Shopping: Study (searchengineland.com) – Findrix studied 55,198 AI answers and found 57% of AI Mode product-answer links went to Google Shopping, with 46.5% of US product answers linking only to Google pages.
→ Google AI Overviews & AI Mode 0.9% Click-Through Rate, Opt Out Study Says (seroundtable.com) – A small opt-out study put the click-through rate from AI Overviews and AI Mode at about 0.9%, and opting out also cut clicks to regular web results.
→ AI traffic drives 83% more revenue per visit for retailers: Adobe (searchengineland.com) – Adobe found AI-referred visits to US retail sites in September earned 83% more revenue per visit and converted 90% better than other traffic.
→ Google AI Overviews are pushing News Boxes out of search results: Study (searchengineland.com) – A new study finds AI Overviews are taking the place of Top Stories news boxes in more Google results.
→ Google AI Overviews With Query Refinement Bubbles (seroundtable.com) – Google is testing refinement bubbles above AI Overviews that open AI Mode for that query instead of changing the search.
→ Is Google Testing Dropping Citations Cards From AI Overviews? (seroundtable.com) – Google may be testing desktop AI Overviews without the citations card that lists clickable source links.
→ Google AI Overview Citation Cards With AI-Generated Summary Snippets (seroundtable.com) – Google appears to be testing AI-written summaries inside the citation cards beside AI Overviews.
→ Google Says No AI Click Reporting News Yet, Ahead Of UK Deadline (searchenginejournal.com) – The UK Competition and Markets Authority lists clicks and click-through rate among the AI search metrics Google must provide by December 3, while the current AI report counts impressions only.
→ Massive Click Spikes in Google Search Console (Early November 2026): What Is Going On and How to Filter Your Reports (mariehaynes.com) – Some sites saw a sudden jump in Search Console clicks around October 31 while impressions stayed flat, and many suspect bots or AI agents.
→ Google Updates Crawled – Currently Not Indexed Details (seroundtable.com) – Google rewrote its help text to say it does not index every page, that this status is common, and that it does not necessarily signal a problem.
→ Only Home Page Indexed? Might Be A Google Search Quality Issue (seroundtable.com) – John Mueller said a site that drops to only its home page indexed, after being fully indexed before, may have a general quality issue rather than a leftover hack problem.
→ Google Downplays The Importance Of Fresh Index Reporting (seroundtable.com) – After the page indexing report slowed down, Mueller said site owners should not spend much energy on how fresh that report is.
→ Google Says Technical SEO Remains Key For AI SEO (searchenginejournal.com) – Mueller warned against short-term AI SEO tricks and said AI answers rely on classic SEO factors, so technical SEO stays important.
→ Google’s John Mueller: Eventually Systems Will Catch The AI Spam & Tricks (seroundtable.com) – In an interview with David Iwanow, Mueller said Google’s systems will eventually catch cheap AI tactics such as self-serving listicles.
→ Google On How To Block Popups Using Robot.txt In Search (seroundtable.com) – Mueller gave two options for keeping a legal popup out of Search: block the embedded site’s content, or block the JavaScript file that loads it, as long as pages still load without it.
→ Supporting a shared AI training control for websites (developers.google.com) – Google plans to support one shared robots.txt token in 2027 for stating AI training preferences across providers, read as a Google-Extended disallow when no Google-Extended rule exists.
→ Google’s Gary Illyes: AI Made Bot Impersonation Worse (seroundtable.com) – On the Odys podcast, Illyes said AI has made it easier for rogue bots to pose as legitimate crawlers such as Googlebot.
→ Googlebot Must Follow Redirects From Homepage To Favicon, Updated Doc Say (seroundtable.com) – Google’s favicon guidance now says Googlebot must be able to follow redirects from the homepage to the favicon page, and none of those URLs may be blocked.
→ Google Explains Why 404 Errors Aren’t Scary 👻 (seroundtable.com) – A new Halloween-themed Non-Canonical Cabinet video explains why 404 pages are not a problem for Search.
→ Google Does Not Show All Spammy Links In Search Console (seroundtable.com) – The links report shows only a sample of links, spammy ones included, so missing spam links there is expected.
→ Prevent user-generated spam on your site and platform (developers.google.com) – Google’s guide tells platforms to publish an anti-spam policy, verify signups, moderate suspicious activity, and noindex content from users without a reputation.
→ Google Lighthouse Update Changes How PageSpeed Insights Simulates Loading (searchenginejournal.com) – PageSpeed Insights now runs Lighthouse 13.6.0, which simulates loading differently, so lab scores may rise on unchanged pages, especially JavaScript-heavy ones.
→ Rank Trackers Find Google Results About One Word Of A Query (searchenginejournal.com) – SE Ranking found 84% of the unrelated results it flagged followed just one word of the query, mostly at position 9 or lower.
→ Unconfirmed Google Search Ranking Update: Volatility & Chatter Spike (seroundtable.com) – Tracking tools and SEO chatter showed a bigger volatility spike following an unconfirmed update on October 24.
→ Google Gemini Stopped Attaching UTM Parameters To Cited URLs (seroundtable.com) – Gemini added utm_source=gemini to cited links in early September but stopped around September 28, though it still passes a referrer.
→ Now Google says it might use your Gmail and Google Photos data to target ads (theverge.com) – Google’s privacy policy, effective October 16, lets Gmail, Drive, and Photos content shape ads when users connect those apps to Search or Gemini through Personal Intelligence.
→ Connected Apps Platform (developers.google.com) – Google opened a preview that lets outside developers build their apps into Google Search and the Gemini app, with access rolling out in phases.
➞ AI + Social
Find out what’s happening in the social media and artificial intelligence world —
→ See the agents, set the terms: new visibility and control for personal agents (cloudflare.com) – Cloudflare added a Personal Assistants category to its analytics so site owners can see which personal AI agents visit, and set an allow, block, or ad-gated policy for each one.
→ Microsoft Clarity With New Agentic Analytics & Ask Clarity (seroundtable.com) – Clarity added agentic analytics and Ask Clarity, and Microsoft said AI agents visiting sites nearly doubled in three weeks to almost 2 million.
→ Your content, your choice: clearer preferences for training and grounding (bing.com) – Bing will let sites allow or block AI training and AI grounding separately, honoring the new preferences from February 1, 2027.
→ ChatGPT ads generate 64% less revenue per visit than Google Search: Study (searchengineland.com) – Researchers studying 1,898 online shops found a paid ChatGPT visit earned $0.89 in US revenue against $2.45 for a paid Google search visit.
→ ChatGPT Ads Turns On Automatic Ad Changes By Default (searchenginejournal.com) – Since October 30, ChatGPT Ads can rewrite headlines, translate text, resize images, and change destination URLs unless advertisers switch it off in Ads Manager.
→ ChatGPT Ads With Learn More Arrow Link; Some Served Through Amazon DSP (seroundtable.com) – A new ChatGPT ad format shows an underlined “Learn more” link with an arrow, and some of these ads appear to run through Amazon DSP.
→ Looking Things Up Now Rivals Writing In ChatGPT, OpenAI Data Shows (searchenginejournal.com) – OpenAI’s data shows information lookups passed writing in August at 20.7% of consumer messages, before writing took the lead again in September.
→ As search traffic dries up, Reddit announces it’s shifting to a logged-in experience (techcrunch.com) – Reddit plans to require logins, starting with tests outside the US, after Google’s AI search features cut its search referrals by more than 1 million a week.
→ Reddit tests ads in search results (socialmediatoday.com) – Reddit is testing ads in its in-app search results with select advertisers, matched partly by relevance to what people search.
→ Reddit introduces For You feed (socialmediatoday.com) – Reddit is rolling out an algorithm-driven For You feed globally as its main discovery tab.
→ Reddit pitches its 1,300 gaming-related communities to brands (socialmediatoday.com) – Citing TransUnion research, Reddit says its gaming communities returned an average $8.58 per ad dollar for gaming advertisers.
→ X launches updated Lead Gen Ads globally (socialmediatoday.com) – X’s revamped Lead Gen Ads, first launched in August, are now open to advertisers worldwide with a prominent call-to-action button.
→ X provides holiday campaign tips (socialmediatoday.com) – X advises keeping holiday ads to about 50 to 100 characters, writing conversationally, and adding a clear call to action.
→ Meta says its expanded processing capacity is driving ad results (socialmediatoday.com) – Meta credits better AI ad ranking for 8.3% more ad clicks and a 15.7% lift in Facebook conversions.
→ Meta adds individual performance data for ad creative (socialmediatoday.com) – A new Breakdown tool in Ads Manager shows how each creative element performs, though carousels and collections are excluded.
→ Meta Drops Training From Its Crawl Clause And Keeps The Crawl (searchenginejournal.com) – Meta’s new Commercial Terms remove AI training from the crawling clause but keep permission to crawl sites tied to business accounts.
→ Instagram rolls out limited Features test in the US (socialmediatoday.com) – Select US creators can post “Features,” videos up to 90 minutes long, and Instagram is expanding Clips that link up to eight Reels.
→ YouTube subscriptions are rising as more users turn to CTV (socialmediatoday.com) – YouTube Music and Premium reached 150 million subscribers, and connected TV now brings in over a third of YouTube’s ad revenue.
→ LinkedIn increases user time, members and creator ads (socialmediatoday.com) – Time spent on LinkedIn rose 15% year over year, membership passed 1.4 billion, and ads featuring creators grew 63%.
→ TikTok playbook outlines the value of its high-end ad options (socialmediatoday.com) – Citing Stackmatix, the report says TikTok TopView ads start at $50,000 for a single day.
→ Pinterest reports solid user growth in Q3 (socialmediatoday.com) – Pinterest added about 21 million monthly users in Q3 to reach 661 million, and revenue rose 12% to $1.17 billion.
→ Pinterest introduces new programmatic buying solution (socialmediatoday.com) – Pinterest’s Programmatic Deals, run through Magnite, open its display and video inventory to Yahoo DSP, Amazon Ads DVA+, and Google Display & Video 360.
→ More than half of US adults use social media to get their news (socialmediatoday.com) – In a June Pew survey of over 3,500 US adults, 58% called social media an important news source, up from 43% in 2022.
→ Shopify Retires Knowledge Base App, Moves AI Agent Facts To Admin (searchenginejournal.com) – Shopify moved AI agent facts into the Merchant Admin, and merchants with over 1,000 facts must re-enter the ones they want before January 15, 2027.
→ Firefox’s Smart Window adds AI form filling and webpage monitoring (theverge.com) – Firefox can now fill web forms using text from your open tabs and watch pages for changes such as sales or new job listings.
➞ Worth Reading
These are the articles that will help you refine your marketing knowledge —
→ Can AI crawlers read your site? (vanessafox.com) – Vanessa Fox’s free checker compares what your robots.txt says with how your server responds to each AI crawler, alongside a spreadsheet of every documented AI and search crawler token.
→ The Form Before the Answer: How ChatGPT Questions Buyers Before Recommending Anything (rankscale.ai) – Across 7,313 answers to buying prompts, 79.4% included a follow-up form, budget was asked in 87.1% of them, and 86.5% arrived already filled in.
→ AI Overviews Appear on 82.5% of US Brand Searches. Here Are the Brand Pages They Cite the Most (seranking.com) – Checking 1,000 US brand keywords, SE Visible found AI Overviews in 82.5% of checks, with homepages making up 36.8% of brand-site citations.
→ Dear SEOs, we must talk about marketing (iloveseo.net) – Gianluca Fiorelli agrees with Marie Haynes that Google still drives most conversions but argues AI visibility can build brand memory that referral data never shows.
→ SEO Beyond Traffic: Influence, Trust, and Business Results | Abby Gleason (advancedwebranking.com) – Compass SEO lead Abby Gleason explains how to measure AI search impact with surveys, branded search, and revenue, and why dropping informational content is a mistake.
→ The New Rules Of Journalist Outreach, Based On Millions Of Emails (searchenginejournal.com) – BuzzStream’s Vince Nero found fully authenticated domains got 30.1% open rates against 7.8% for unauthenticated ones, and subject lines stating a finding beat teasers.
→ Technical SEO Audit Reports: How to Turn Them into Client Retention Tools (sitebulb.com) – Mike Hakob shows agencies how to turn monthly audits into progress reports, since clients mostly want to know whether things are getting better or worse.
→ GEO Guide 2027: How to Improve Visibility in ChatGPT, AI Mode, and AI Search According to 200+ Top Experts and Resources (gddash.com) – GDdash collects advice from more than 200 experts and resources on improving visibility in ChatGPT, AI Mode, and other AI search tools.
➞ What This All Means for Your Strategy
Search engines and AI assistants can only rank or cite pages they can reach and read, and you can only judge the results if your analytics count real people. Most of this week’s fixes are small.
Pick one before next week. Request a log sample from your host and look for 403s and 499s, clean up your robots.txt, or test the GPU check in a spare GA4 property. Then reply and tell me which one you chose.
Until next week,
— Nishat from HighDegree* Marketing.
P.S. Have a question about implementing these strategies? Hit reply – we read every email and often feature reader questions in future issues.
➞ Who is Nishat Shahriyar?
I’ve been a Digital Marketing Strategist since 2007. Today I’m the Product Marketing Lead at Fluent Forms (a WordPress form and lead-generation plugin used by 700K people), FluentPlayer (a WordPress Video Player Plugin) and Ninja Tables (a WordPress table builder plugin used by 80K people). Before that, I led marketing at Fluent Support (a WordPress helpdesk plugin used by 10K people).
Connect with me on LinkedIn or follow me on X/Twitter at @rednishat for SEO & Marketing updates.