6

I am interested in using Varnish to cache/throttle/etc responses to a RESTful API I am creating. I may be using the term/acronym "HMAC" too loosely, but what I mean is that each request to my API should include a header that includes a hash that was calculated by the client by hashing parts of the request (including a timestamp) with a shared secret. The server then calculates this same hash with the same ingredients from the request, and determines if the request is valid and should be responded to.

This works well enough, but now I would like to use Varnish to cache my API responses. The nature of HMAC requires that each request calculates the hash to verify the user is who they are, but the actual response that is returned is the same - so the meat of the API call is very much cacheable.

What I'd like (and I'm assuming this can be achieved, I just don't know HOW) is to pass the authentication task to the backend, somehow tell Varnish "yes, go ahead and respond to this request" or "no, don't respond to this request" and then from there let Varnish determine if the request can be served from cache or not.

Even more ideally, would be to do something slightly fancier, and allow Varnish to handle the authentication itself, or pass the HMAC processing onto something faster then the backend. For example, the API might store the client secret/public key in a redis cache, then Varnish might actually calculate the hash itself using the values from Redis.

5
  • You can either pass the authentication task to the backend splitting the request in two internally or implement HMAC inside Varnish using inline C.
    – NITEMAN
    Commented Feb 20, 2014 at 8:32
  • Thanks for the reply @NITEMAN - I'm new to Varnish, does this seem like the sort of thing that would be "OK" from a performance/scalability standpoint? And I mentioned using something like Redis to share secret key/token info between the application and varnish, does this seem like a good approach if I were to implement HMAC inside of Varnish? Thanks again! Commented Feb 22, 2014 at 4:56
  • It's difficult for me to say (no experience inlining significant C code in big sistems), but in theory it should work and scale well. If you're concerned about performance, maybe writing a varnish module is the way to go.
    – NITEMAN
    Commented Feb 22, 2014 at 9:44
  • Well thank you @NITEMAN. To be honest I'm not really sure what would be involved in writing such a varnish module, but I'm going to investigate. Commented Feb 25, 2014 at 5:30
  • @KevinMitchell: How did this work out for you? Any experiences would be much appreciated, of course only if you are able to, for example your employer allows it and other such considerations ;) Commented May 26, 2016 at 11:21

2 Answers 2

3

You should be able to implement the fancier solution in Varnish VCL code (Varnish Configuration Language) by using two Varnish Modules:

Both modules are used in production, as listed in the modules directory.

If Varnish handles the authentication in VCL, you can let Varnish cache your API backend response and deliver it only for authenticated requests.

If the HMAC implementation requires the request body:

As Gridfire points out in his/her answer, Varnish cannot access the request body. And we can/should not send the full request body in a HTTP header from the backend/application.

But, we can send a hash/digest of the full request body in a HTTP header. Calculation of the hash on the backend should be negligible compared to generating the output(markup|data|whatever). AFAICT there should be no cryptological/practical downsides to this method as long as the hash/digest and HMAC is robust, and the digest is lengthy (256bits or more). Performance testing is adviced as usual.

1
  • I'm going to check both of these out. Thank you! Commented Oct 2, 2015 at 23:43
0

Varnish can easily do HMAC using the VMOD's in Geir Bostad answer, unless your HMAC implementation uses the request body as part of the hash. Varnish does not give you access to the request body, libvmod-bodyaccess provides some functions but I have found no way of actually getting the request body.

You could theoretically add a header containing the request body, but this is pretty bad practice and will either bloat your HTTP requests with redundant data, or break HTTP request standards if you choose to only put the data in the header. Simply put not recommended.

An alternate solution would be to use Nginx, which can also act as an SSL terminator if you want to use HTTPS (Varnish doesn't do SSL). Nginx has a module to run Lua scripts (Ubuntu/Debian package nginx-extras provides it without requiring you to compile it yourself), and the module brings the handy access_by_lua_file directive to allow or block access based on the result of the script. There's a HMAC script for Nginx here.

1
  • I can think of one possible solution to "If the HMAC implementation uses the request body as part of the hash": Let the application calculate a hash of the body and return the hash in a http header (instead of the full body). I don't know if there are any major cryptological/practical downsides to that method. Choosing a lengthy hash (maybe 256bits or more) should be ok. Long hashes might have performance issues, so testing and benchmarking is required. Commented Mar 29, 2016 at 14:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.