Bots and Fraud
Table of Contents
Bots in Your Analytics Data: The IAB List Is Just an Umbrella
Bot traffic is soaking your analytics data. Most teams check the IAB/ABC Spiders & Bots List box and call it handled, but that umbrella catches a fraction of a percent of bots on most sites. You need a roof.
- A data foundation, waiting for a house.
- Brick by brick, the house goes up: data governance, schema, data definitions, SDR or tracking plan.
- It gets furnished with good business rules.
- And a kickass TV. Someone sits down to watch. The house still has no roof.
- Then it starts to rain. Bot traffic pours in and pools inside.
- The rain turns blood red and the person on the sofa turns into a demon: a psycho letting the bot traffic in.
- They open an umbrella: the IAB list. It keeps the person holding it dry while the house keeps flooding.
- Then the roof goes on, and the house stays dry.
- Declared bots
- Undeclared bots
The house
Most of your time goes into schema, data definitions and your SDR. Like 85-90%.
This feels like a tired metaphor, but building an indestructible data architecture is a lot like building a solid house. There are a million articles talking about this ad nauseum – I might have written one of two of them.
Start with a solid foundation, carefully piece together each brick. I mean, it’s why I always recommend spending the overwhelming majority of your time on your Schema, Data definitions, SDR, and/or Tracking Plan. Like… 85-90% of your time on that.
Every good decision you make there reinforces the data structure you’re developing. Blah blah blah – we love good data like we love having the confidence that everything won’t suddenly collapse.
We also love a kickass TV. Our data can now live in harmony – you know, until it rains.
The rain
Bot traffic is the rain. Water damage ruins literally everything.
If you’ve ever owned a house, you know that water damage ruins literally everything. Good bones in your house? Not after significant water damage. The worst part is that so much of it happens without you even knowing it. By the time you start noticing water spots, you have to do more than just stopping it at the source.
When it comes to your dataset, bot traffic is the rain. Unless you’re some kind of psycho, you don’t want to let that rain in.
The umbrella
The IAB list is an umbrella. It catches a fraction of a % of bots.
If we’re being honest, most people don’t know how to solve for the rain. In fact, a lot of them have convinced themselves that an umbrella is enough to keep the rain out. That umbrella is the IAB block list. People check that box and can report back to leadership that bots have been addressed. Unfortunately, that only covers, like… a tiny percentage of bots.
For the uninitiated, the IAB list is a maintained list of user agents and IP Addresses observed crawling sites. To make the list, you have to be observed across enough properties to justify the organization saying “let’s call this out for literally everyone.” Additionally, it compares traffic against parameters that are notoriously easy to spoof. All-in-all, it’s nice for what it is and catches a couple bots here and there – but it is NOT the same as bot protection (as-in, for most digital properties it catches a fraction of a % of all bots).
The roof
For everyone else, you need a roof.
If you’re fine with just an umbrella, I’m proud of you. Gross data negligence means nothing to your organization. It’s never been so easy to collect a paycheck. For everyone else, you need a roof. It’s why I joined CHEQ and it’s why I won’t shut up about this topic.