1

Our UI server needs to make certain downloadable files available to users. The files live on a specialized storage server. The UI server enforces some permissions and uses HTTP Basic Auth user/pwd to get the files off the storage server, but the user should never know these storage credentials so the files are streamed through our UI server to users.

We need to stream these through our UI server running Rails 4.2.11 and Ruby 2.4.9. Up until now we've been assigning an enumerator to the response body like Ruby on Rails 3: Streaming data through Rails to client , but it has been quite unreliable and we've had many cut off or incomplete downloads.

It appeared that the rack-hijack method ( https://blog.chumakoff.com/en/posts/rails_sse_rack_hijacking_api )method was newer and we hoped more reliable, but it also cuts off downloads when slower clients connect to download and it is a large file.

It appears that IO buffers are filling up and creating EOF or other errors since the download is streamed faster from the storage server than it is usually streamed through the UI by users.

  • I'm getting the rack-hijack IO stream (PhusionPassenger::Utils::UnseekableSocket) and writing the results of the chunks from the GET response to it.

  • I've tried many different http client gems and they all have similar problems, I think because it is filling the output buffer when slow clients download, not because they have problems, though the errors look that way.

  • When http clients have problems in this scenario they variously complain of either EOF or error reading from socket: Could not parse data entirely (1 != 0).

  • I've tried different versions of Passenger and tried various general solutions.

  • I've looked at a similar question at Rails: How to send file from S3 to remote server . In the answer, he suggests using <obj>.stat.size to check on the buffer and not have it overflow and have it match the user's download speed more. However, the stream I'm given by rack-hijack doesn't seem to have any way to check the size or buffer that I can find (PhusionPassenger::Utils::UnseekableSocket). If it exposed a way to do this, I could likely do this.

  • I've looked through the Passenger documentation for buffering information and changed some buffering settings but it didn't help.

  • I've looked into the methods on the stream I get from Passenger and also on the .to_io settings but I'm having a hard time finding tools I think I need and having trouble finding much documentation.

  • If I match downloads speeds from both ends then no problems. Stuff like wget --limit-rate=2m http://localhost:3000/download_test/stream1 allows me to throttle the speed of the client and I usually get an error around 180MB into the download.

  • It's running on Amazon Linux but runs either much better or without many problems on a local Mac development machine. It's hard to tell exactly if it's still problematic or just not as much since the problem can be a little dependent on size and other factors.

  # a simplified version of the code, using a public URL
  def stream1
    url = 'https://www.spacetelescope.org/static/archives/images/publicationtiff40k/heic1502a.tif'
    response.headers['Content-Type'] = 'image/tiff'
    response.headers['Content-Disposition'] = 'attachment; filename="funn.tif"'
    response.headers["X-Accel-Buffering"] = 'no'
    response.headers["Cache-Control"] = 'no-cache'
    response.headers["Last-Modified"] = Time.zone.now.ctime.to_s

    response.headers["rack.hijack"] = proc do |stream|
      Thread.new do
        begin
          response = HTTP.get(url)
          response.body.each do |chunk|
            stream.write(chunk)
          end
        rescue HTTP::Error => ex
          logger.error("while streaming: #{ex}")
          logger.error("while streaming: #{ex.backtrace.join("\n")}")
        ensure
          stream.close
        end
      end
    end
    head :ok
  end

The results I'm looking for are to make downloads reliable no matter what (reasonable) speed of the client downloading from the UI.

Thanks for any help you can give since I'm feeling stuck.

3
  • FYI: the rack-hijack IO solution is likely to break on HTTP/2 enabled application servers (and many of them will implement HTTP/2 at some point)... HTTP/2 requires packet framing and multiplexing that the hijack IO approach doesn't really play along with all that well. Consider something else.
    – Myst
    Oct 18, 2019 at 23:51
  • 1
    One possible approach - download the file in the background to a temporary file on the UI server, poll to test if the download is available, download when file becomes available. P.S... the blog post on hijacking is a poor approach for SSE... better use iodine or agoo for that.
    – Myst
    Oct 19, 2019 at 0:03
  • Seems like problems were caused by buffering issues with Passenger, (though even reducing the buffer sizes didn't solve it) and some timeout or disconnect issues on the http client or remote servers when the buffer creates long pauses. I eventually created a solution that worked by downloading chunks to a local temp file in one thread and reading from that same file and serving it in another thread at the same time. Of course, building in checks to not read past what has been written, etc. It works for what I need, though it's more complicated. I'd be happy to share if anyone is interested.
    – skunkshow
    Oct 20, 2019 at 0:20

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.