Understanding rate-limited websites

Sometimes when Silktide tests a website, the website becomes inaccessible to Silktide. This is usually due to one of the following:

• The website cannot handle many visitors
• The website has protection designed to prevent attacks by bots
• The website was experiencing temporary downtime

If this happens, Silktide may report that links were broken which appear to work for you, or that a page was an error message that you can’t see.

The website cannot handle many visitors

This is more common on smaller websites. If a website is large and well funded, it is more likely to be the website has protection against attacks by bots.

By default, Silktide will attempt to download up to six webpages from a website at once, which is equivalent to six highly active visitors browsing the site. For some websites, this is more than they can handle, and the website slows down or may fail to respond entirely.

You have two options to fix this:

• Limit how fast Silktide tests the website
• Tell your web developers to fix any scaling issues

For the latter, we recommend checking your speed screen to see if specific pages are slow – e.g. if a badly programmed piece of code is overwhelmed – and if so, focus on fixing those. If a website is using very low grade hosting without caching, that is likely the cause.

The website has protection designed to prevent attacks by bots

A common attack on websites is a Distributed Denial of Service (DDoS) attack, where a large number of computers automatically attempt to access a website at once. The idea is to overwhelm the website with so much traffic that it cannot be accessed by other people.

Some websites have systems designed to protect against this. They detect what they believe too many requests, and block those requests.

If this happens to Silktide, then Silktide will be unable to see the website. There are two things you can try:

• Limit how fast Silktide tests the website
• Tell your web developers to permit more connections – see what IP addresses and user agent does Silktide uses

The website was experiencing temporary downtime

For example, sometimes a website may be unavailable while a new version is made live, or while it is backed up.

This is very rare, but easy to test. If the website is working for you now, re-test it in Silktide. If you still have problems, this is not the cause.