CloudFlare Opens Pay Per Crawl to Fight Against AI Crawlers

Key changes, Tech Insiders!

Cloudflare deadbolts its domains against freeloading AI crawlers, the Senate tossed the master key of AI regulation back to the states, and Grammarly secured Superhuman to dominate the AI-powered inbox.

Let's hear who drops the keys first.

Here's what you need to know today:

Cloudflare Blocks Leeching AI Bots, Adds Pay Per Crawl

The internet's all-you-can-eat data buffet for AI models may be closing soon.

Over the past year, Cloudflare—which routes and protects about 20% of the web—armed site owners with a one-click "block all AI bots" switch, a dashboard that traces every crawler hit, and a labyrinth to bewilder misbehaving spiders.

CEO Matthew Prince framed those tools as scaffolding for something bigger—and here it is.

Alongside a new default block on unverified bots for new websites, Cloudflare is launching Pay Per Crawl, a marketplace where site owners set a price each time a legit AI company wants to ingest their content.

Publishers can still allow friendly crawlers (or keep them out entirely), but every sanctioned scrape now carries a price tag. Early adopters span the Associated Press, Fortune, Reddit, Stack Overflow, and at least 40 more, all eager to replace evaporating search referrals with fresh AI revenue.

To ensure no one can dine and dash, Cloudflare plans to repurpose its heavyweight DDoS mitigation systems to spot and block scrapers that try to sneak around the paywall. Granular settings let academics and nonprofits stay nourished while commercial models pick up the tab.

Why it matters: The tollbooth is down—AI models must feed the meter or get booted off the ramp. Even Creative Commons is piloting CC Signals, a machine-readable tag that allows creators to demand credit or compensation when their work is used to train AI—proof that the push to monetize data is accelerating. And it could redraw the economics of how knowledge flows online.

Share this newsletter!

Pulse Check

Would you open your site's gates to AI crawlers?

Results from Yesterday's Pulse Check

Would you trust an AI doctor over a human?

Insider Intel

Plot Twist: Senate Axes 10-Year AI Rule Freeze

Even Capitol Hill couldn't ghostwrite this plot twist.

During a bleary-eyed 4 a.m. vote-a-rama on Tuesday, the US Senate torched a GOP plan to bar states from regulating AI for a decade.

The 99-1 smackdown—sparked by Tennessee Republican Marsha Blackburn—yanked the moratorium from President Donald Trump's megabill (which passed the Senate 51-50 later that day).

Big Tech backers, including OpenAI and Google, had pushed for a single federal framework to avoid "patchwork" rules, but governors from both parties cried foul over state rights. A compromise shortening the ban to five years fizzled, and even AI-friendly Senator Ted Cruz pivoted to strike the language.

With the provision gone, states remain free to pass deepfake, privacy, and child-safety laws while Congress continues its slow crawl toward national AI legislation.

Flash Bytes

Security Alerts

Kelly Benefits Breach Widens to 553K Users

Health and payroll vendor Kelly Benefits has upped its December 2024 breach tally from 32,000 to a whopping 553,000 individuals.

Attackers lingered for five days, snatching names, Social Security numbers, medical details, and financial info from 46 client organizations, including UnitedHealthcare and Aetna Life Insurance.

A year of free credit monitoring is offered, but phishing risks loom.

Note to HR: Holiday gifts shouldn't include your SSN.

Apple Alleges Vision Pro Secrets Walked to Snap

Apple accuses former senior product-design engineer Di Liu of swiping a "massive volume" of Vision Pro design, cost, and supply-chain files three days before his October 2024 exit. The company says Liu concealed a job offer from Snap—now developing its own AR headset—and told colleagues he was quitting for family time.

Apple's lawsuit in Santa Clara County seeks damages, a forensic audit of Liu's personal cloud storage and devices, and the return of all material.

Snap, which isn't named as a defendant, says it has found no evidence that the data is influencing Liu's current work.

Industry Shakeups

Grammarly Snaps Up Superhuman for AI Email Edge

Fresh off a $1 billion funding round, Grammarly is acquiring elite email app Superhuman to anchor an AI productivity platform.

Superhuman's 100-plus staff, led by founder Rahul Vohra, will join Grammarly, integrating lightning-fast inbox workflows with Grammarly's cross-app writing agents.

Grammarly CEO Shishir Mehrotra says email is the "perfect staging ground" for multiple AI agents—imagine bots triaging threads, drafting replies in your tone, and pulling docs from Coda (another recent Grammarly buy).

Superhuman's $825 million 2021 valuation and reported $35 million annual recurring revenue highlight the appetite for premium email experiences amid Gmail and Outlook's own AI upgrades.

The deal signals Grammarly's pivot from grammar cop to full-stack workplace assistant, setting up a turf war with Google Workspace, Microsoft 365, and a swarm of SaaS upstarts.

When your spell-checker starts scheduling meetings, you know the robots are unionizing.

Curious about where AI is really headed?

The Neuron cuts through the noise to bring you smart, hype-free takes on the latest AI trends, tools, and breakthroughs. Join 500,000+ professionals from top companies like Microsoft, Apple, Salesforce and more.

Advertise in Daily Tech Insider!

Daily Tech Insider is a TechnologyAdvice business

© 2025 TechnologyAdvice, LLC. All rights reserved.

TechnologyAdvice, 3343 Perimeter Hill Dr., Suite 100, Nashville, TN 37211, USA.

Want to change how you receive these emails? Update your preferences or unsubscribe here.