Akamai blocks unordered HTTP request headers

Akamai blocks unordered HTTP request headers(gwillem.gitlab.io)

43 points by gwillem 9 years ago | 11 comments

idbehold 9 years ago |

In the second example the author does the following:

  $ ACCEPT="text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
  $ curl -v -H "$UA" -H "$ACCEPT" $URL |& grep '< HTTP'

The author fails to prefix $ACCEPT with the actual header key. It should be:

  $ curl -v -H "$UA" -H "Accept: $ACCEPT" $URL |& grep '< HTTP'

jstanley 9 years ago | |

From personal experience I'd be willing to give him the benefit of the doubt (i.e. he did it right, but wrote it up wrong). Good spot though.

gwillem 9 years ago | |

Thanks! Indeed a copy paste error, I updated the article.

jnfurst 9 years ago |

This is just the configuration for a single site. The author did not even try it against www.akamai.com:

$ URL=http://www.akamai.com

$ UA="User-Agent: Mozilla/5.0 My API Client"

$ ACCEPT="Accept: /"

$ curl -v -H "$UA" -H "$ACCEPT" $URL

< HTTP/1.1 301 Moved Permanently

< Content-Length: 0

< Location: https://www.akamai.com

< Date: Tue, 02 May 2017 14:46:59 GMT

< Connection: keep-alive

squeed 9 years ago |

Is it Akamai? Or is it a single site on Akamai? CDN customers can configure their sites in a million ways.

My guess is a single site that was getting DDoS'd added this as an attack signature and forgot about it.

tyingq 9 years ago | |

My money, in this case, is something like Akamai Kona or Shape Security, that does bot blocking. Comparing user-agent against known header order for that specific user-agent sounds like something they would do.

jnfurst 9 years ago | |

This is just a single sites configuration.

bluesmoon 9 years ago | | |

Well, everybody knows that in statistics, a sample size of 1 gives you a 0 margin of error ;)

michaelmior 9 years ago |

> most libraries use random order

Most libraries use an undefined order. This is not the same as random.

AznHisoka 9 years ago |

Did Akamai recently just make this change?

I'm asking because I've been running a web crawler for years now, and in the past week, I have noticed that the crawler is being rejected in more websites then usual.

gumby 9 years ago |

I disagree with the author's title (and I see it was submitted with a different title).

This is actually a report of two bugs:

1- the standard doesn't require an order

2 - the IETF's admonition that you be liberal in what you accept.