0

I recently came through the concept of ETag HTTP header. (this) But I still have a problem that for a particular HTTP resource who is responsible to generate ETags?

In other words, it is actual application, container (Ex:Tomcat), Web Server/Load balancer (Ex: Apache/Nginx)?

Can anyone please help?

2 Answers 2

3

Overview of typical algorithms used in webservers. Consider we have a file with

  • Size 1047 i.e. 417 in hex.
  • MTime i.e. last modification on Mon, 06 Jan 2020 12:54:56 GMT which is 1578315296 seconds in unix time or 1578315296666771000 nanoseconds.
  • Inode which is a physical file number 66 i.e. 42 in hex

Different webservers returns ETag like:

  • Nginx: "5e132e20-417" i.e. "hex(MTime)-hex(Size)". Not configurable.
  • BusyBox httpd the same as Nginx
  • monkey httpd the same as Nginx
  • Apache/2.2: "42-417-59b782a99f493" i.e. "hex(INode)-hex(Size)-hex(MTime in nanoseconds)". Can be configured but MTime anyway will be in nanos
  • Apache/2.4: "417-59b782a99f493" i.e. "hex(Size)-hex(MTime in nanoseconds)" i.e. without INode which is friendly for load balancing when identical file have different INode on different servers.
  • OpenWrt uhttpd: "42-417-5e132e20" i.e. "hex(INode)-hex(Size)-hex(MTime)". Not configurable.
  • Tomcat 9: W/"1047-1578315296666" i.e. Weak"Size-MTime in milliseconds". This is incorrect ETag because it should be strong as for a static file i.e. octal compatibility.
  • LightHTTPD: "hashcode(42-1047-1578315296666771000)" i.e. INode-Size-MTime but then reduced to a simple integer by hashcode (dekhash). Can be configured but you can only disable one part (etag.use-inode = "disabled")
  • MS IIS: it have a form Filetimestamp:ChangeNumber e.g. "53dbd5819f62d61:0". Not documented, not configurable but can be disabled.
  • Jetty: based on last mod, size and hashed. See Resource.getWeakETag()
  • Kitura (Swift): "W/hex(Size)-hex(MTime)" StaticFileServer.calculateETag
  • JS lib jshttp/etag "hex(Size)-hex(MTime)" stattag
  • H2O (C) "hex(MTime)-hex(Size)" h2o_filecache_get_etag

Few thoughts:

  • Hex numbers are used here so often because it's cheap to convert a decimal number to a shorter hex string.
  • Inode while adding more guarantees makes load balancing not possible and very fragile if you simply copied the file during application redeploy. MTime in nanoseconds is not available on all platforms and such granularity not needed.
  • Apache have a bug about this like https://bz.apache.org/bugzilla/show_bug.cgi?id=55573
  • The order MTime-Size or Size-MTime is also matters because MTime is more likely changed so comparing ETag string may be faster for a dozen CPU cycles.
  • Even if this is not a full checksum hash but definitely not a weak ETag. This is enough to show that we expect octal compatibility for Range requests.
  • Apache and Nginx shares almost all traffic in Internet but most static files are shared via Nginx and it is not configurable.

It looks like Nginx uses the most reasonable schema so if you implementing try to make it the same. The whole ETag generated in C with one line:

printf("\"%" PRIx64 "-%" PRIx64 "\"", last_mod, file_size)

My proposition is to take Nginx schema and make it as a recommended ETag algorithm by W3C.

2
  • 1
    Various parts of this late post (posted to a 3+ year-old topic with an already-accepted answer) have been challenged at redmine.lighttpd.net/issues/3055
    – gstrauss
    Jan 13, 2021 at 1:09
  • downvoted due to a) not answering OP's question, b) uncorrected technical inaccuracies, c) apparent misunderstanding of the word "opaque" in tools.ietf.org/html/rfc7232#section-2.3 "An entity-tag consists of an opaque quoted string"
    – gstrauss
    Aug 12, 2022 at 2:46
1

As with most aspects of the HTTP specification, the responsibility ultimately lies with whoever is providing the resource.

Of course, it's often that case that we use tools—servers, load balancers, application frameworks, etc.—that help us fulfill those responsibilities. But there isn't any specification defining what a "web server", as opposed to the application, is expected to provide, it's just a practical question of what features are available in the tools you're using.

Now, looking at ETags in particular, a common situation is that the framework or web server can be configured to automatically hash the response (either the body or something else) and put the result in the ETag. Then, on a conditional request, it will generate a response and hash it to see if it has changed, and automatically send the conditional response if it hasn't.

To take two examples that I'm familiar with, nginx can do this with static files at web server level, and Django can do this with dynamic responses at the application level.

That approach is common, easy to configure, and works pretty well. In some situations, though, it might not be the best fit for your use case. For example:

  • To compute a hash to compare to the incoming ETag you first have to have a response. So although the conditional response can save you the overhead of transmitting the response, it can't save you the cost of generating the response. So if generating your response is expensive, and you have an alternative source of ETags (for example, version numbers stored in the database), you can use that to achieve better performance.
  • If you're planning to use the ETags to prevent accidental overwrites with state-changing methods, you will probably need to add your own application code to make your compare-and-set logic atomic.

So in some situations you might want to create your ETags at the application level. To take Django as an example again, it provides an easy way for you to provide your own function to compute ETags.

In sum, it's ultimately your responsibility to provide the ETags for the resources you control, but you may well be able to take advantage of the tools in your software stack to do it for you.

5
  • your answer is quite good. Let me verify these things. So as you said we can configure (usually) a web server to do etag business (3rd para) and also an application framework (4th para). As per your understanding, will there be any potential difference of performance in two approaches? Or is it quite negligible? Please add this to the answer :) ) Nov 28, 2017 at 6:07
  • 1
    @SupunWijerathne: I've added some specific reasons why the default hash-based approach might not always be a good fit. It's impossible to generalize about which is better, but I would usually recommend doing it the default hash-based way unless you've come up a compelling reason to do it yourself. Nov 28, 2017 at 7:00
  • just another side question, there are situations we have to calculate last modified time of a particular resource manually and it might not be a good fit, right? Nov 28, 2017 at 7:11
  • @SupunWijerathne: I'm afraid I don't understand what you're asking. Using the Last-Modified header is a different situation, so you might be better off posting a separate question. Nov 28, 2017 at 13:23
  • please look at this question. stackoverflow.com/questions/47544930/… Nov 29, 2017 at 4:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.