Rise of the Machines: Bots Taking Over the Web


December is a difficult time to stay on top of everything, with work requiring you to meet tight deadlines ahead of the holidays and keep your clients on track. As a result, you have to sacrifice the luxuries in your everyday routine to prioritise the most important work. In my case, this has been reviewing news websites and performing research.

Discovery at TechCrunch

In the wake of Christmas, I have been scanning through my favourite websites in an effort to catch up on all that I have missed. One of the most interesting stories I found, coming from TechCrunch, reported that bots accounted for 61.5% of all web traffic in 2013. TechCrunch acquired that figure from Incapsula, which is a cloud-based network that improves the online security of websites.

10.5% Increase in Bots on 2012

The figures released by Incapsula reported that human users accounted for the remaining 38.5% of website visitors in 2013. In contrast to 2013, bot and human visitors accounted for shares of 51% and 49% of website visits, respectively. The 10.5% increase is certainly a dramatic rise in the space of just a year, so it’s important to take a look at the different types of bots.

Breakdown of Bot Visitors

When reading this article, it is easy to get carried away and label all bots as malicious, but the truth of the matter is that they aren’t. As you will be able to see in the figures below, only some of the bots in 2013 were serving malicious agendas:

  • Search engines and good bots31%
  • Scrapers 5%
  • Hacking tools4.5%
  • Spammers 0.5%
  • Other impersonators 20.5%

Comment Spamming Nearly Extinct

Focusing on the positives, Google Penguin has managed to help decrease visits from spammers from 2% in 2012 to 0.5% in 2013. Penguin has made comments irrelevant as an SEO technique, which explains the sharp decline. Also, the 31% of search engines merely indicates an increased level of activity to help improve the accuracy of indexing.

Malicious Bots Must be Stopped

In spite of the good news about the decline of comment spamming, there is still one concern from the figures above. Scrapers, hacking tools, spammers and other impersonators account for approximately half of all web visits by bots. The presence of bots has risen significantly in just one year, but key web players like Google need to take measures to ensure that beneficial bots outweigh their malicious relatives. (Photo credit to John Greenaway)

By Mike Porter