0

Let me start off by saying I understand performance is hard, and even harder through a forum, but I hope someone can point me towards the next step of figuring out this problem we're having in production on a Ruby on Rails environment. I've been seeing this for a while, but now with our usage down due to Covid, it's become more clear that it's not a user-load issue, but something in the infrastructure.

NewRelic Transaction Time

The day starts off fine, but midday, things slow more and more... until a passenger restart clears everything up for 24/48/96 hours.

Checking the obvious items:

  • Server memory usage is in check. Memory use increases during this time from 20%-30%. So some increase, but definitely not swapping (I've seen that happen before)
  • CPU usage is similar. It increases from 1% at 6am to 18% at worst case, but still well within the capabilities of the server.
  • Passenger usage. Looking at passenger-status, there's never a backlog of requests. Because of covid, our user-base is down, and even at its slowest, I'm seeing 2-3 passenger processes serving content (out of ~40 max), and no long-running requests.
  • Delayed job workers (3) are running, but no jobs at the time. They don't seem to be using a significant amount of cpu or mem
  • Watching the apache/rails logs, nothing looks awry. No DoSes, no unexpected load.
  • Looking at long-running transactions in NewRelic, there are some that are taking 20-30-50 sec, but after the passenger restart, they are back to 1-3 sec like usual. All the time is in Ruby.

So this is where I'm stuck. I can see where the problem is (185 sec): Example of one long-running transaction segment

but if I restart passenger and re-run, that same code will take less than a second. And that's just the example that I see from today's traces. Yesterday, it looked like it was a different controller having problems.

Any recommendations on my next steps to troubleshoot? I don't know if I should instrument a specific controller, because it's not just one method that's causing the problem (afaik). I think I'm seeing the symptoms, not the cause - and no idea how to see what's really going on.

Thanks in advance -Mike

Rails 4.2.11.1 Ruby 2.4.9 Passenger 6.0.5 on Apache 2.4.43

3
  • 2
    A service such as Scout APM can give you detailed time-sliced performance information with stack traces and database analysis.
    – Schwern
    Commented Jun 17, 2020 at 18:39
  • Most likely scenario is either you have some queries that are taking too long, check your table cache hit rate/index cache hit rate and see what they're at.
    – ricks
    Commented Jun 17, 2020 at 20:58
  • Thanks Schwern, we invested in NewRelic, and it's been great - but I'm admittedly still learning how to dive into some of the details. Ricks - will take a look, I've wondered if the db is blocking somehow... will do more research and post back.
    – Mike
    Commented Jun 18, 2020 at 13:19

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.