Crawling December: CDNs and crawling

Content delivery networks (CDNs) are particularly well suited for decreasing latency of your
website and in general keeping web traffic-related headaches away. This is their primary purpose
after all: speedy delivery of your content even if your site is getting loads of traffic. The “D”
in CDN is for delivering or distributing the content across the world, so transfer times to your
users is also lower than just hosting in one data center somewhere. In this post we’re going to
explore how to make use of CDNs in a way that improves crawling and users’ experience on your
site, and we also look at some nuances of crawling CDN-backed sites.