I have been looking for an explanation for why twitter had to migrate part of its middle ware from Rails to Scala. What prevented them from scaling the way facebook has, by adding servers as its user base expanded. More specifically what about the Ruby/Rails technology prevented the twitter team from taking this approach?
-
5As far as I know, Twitter did not replace RoR with Scala. They replaced some components with Scala, but the front end is still Rails. I think. And I saw a presentation last year by someone from Twitter talking about a lot of the work they have been doing in making Ruby run faster.– Daniel C. SobralCommented Mar 22, 2012 at 1:36
-
@luchosrock That link is broken, is it up anywhere else?– AddisonCommented Mar 29, 2017 at 20:34
-
@Addison web.archive.org/web/20120102102852/http://canrailsscale.com ahahahahah– luchosrockCommented Mar 31, 2017 at 20:02
7 Answers
It's not that Rails doesn't scale, but rather, requests for "live" data in Ruby (or any interpreted language) do not scale, as they are comparatively far more expensive both in terms of CPU & memory utilization than their compiled language counterparts.
Now, were Twitter a different type of service, one that had the same enormous user base, but served data that changed less frequently, Rails could be a viable option via caching; i.e. avoiding live requests to the Rails stack entirely and offloading to front end server and/or in-memory DB cache. An excellent article on this topic:
How Basecamp Next got to be so damn fast
However, Twitter did not ditch Rails for scaling issues alone, they made the switch because Scala, as a language, provides certain built-in guarantees about the state of your application that interpreted languages cannot provide: if it compiles, time wasting bugs such as fat-fingered typos, incorrect method calls, incorrect type declarations, etc. simply cannot exist.
For Twitter TDD was not enough. A quote from Dijkstra in Programming in Scala illustrates this point: "testing can only prove the presence of errors, never their absence". As their application grew, they ran into more and more hard to track down bugs. The magical mystery tour was becoming a hindrance beyond performance, so they made the switch. By all accounts an overwhelming success, Twitter is to Scala what Facebook is to PHP (although Facebook uses their own ultra fast C++ preprocessor so cheating a bit ;-))
To sum up, Twitter made the switch for both performance and reliability. Of course, Rails tends to be on the innovation forefront, so the 99% non-Twitter level trafficked applications of the world can get by just fine with an interpreted language (although, I'm now solidly on the compiled language side of the fence, Scala is just too good!)
-
2Let me start off by saying that I really like Scala, even though I'm still more of a Ruby guy. Regarding Dijkstra and the perceived advantages of static typing, I have to paraphrase Rich Hickey, who said that what all bugs in productive software have in common, is that they got past the type checker and the unit tests. BTW: Facebook's HipHop is written in C++, not C. Commented Mar 17, 2012 at 13:39
-
1Thanks, updated Facebook reference to C++ from C. re: Hickey and origin of bugs, sure, totally agree, and with interpreted languages there is no type checker, so a class of errors that cannot exist in a compiled language must be caught at unit test level in Ruby, PHP et al. Moving forward it will be interesting to see whether interpreted or compiled languages are the wave of the future (I suspect the latter), will be fun to find out regardless ;-) Commented Mar 17, 2012 at 14:12
-
Of course there's type checking in interpreted languages, but at runtime. Tbh though, I barely ever end up with a type error in Ruby, except for the cases where something is
nil
that shouldn't be. Here Haskell'sMaybe
monad or Scala'sOption
type are a big benefit, whereas Ruby solutions like theandand
gem feel really kludgy. I don't think that either paradigm will be wave of the future, both have been around for a while and are probably here to stay. Commented Mar 17, 2012 at 17:19 -
Runtime type checking provides zero guarantees about the state of the application. Errors are caught at runtime (by the user if your unit test don't catch them) which in your case may be a rarity, but for other, less skilled Ruby coders, these errors likely do sneak into production. Interpreted languages will let you do virtually anything and only tell you about it at runtime. In Scala your "barely ever" becomes "never", as human error (you are human, right? ;-)) is taken out of the equation. Of course the Scala compiler can be a pain in the ass as well, tradeoffs everywhere Commented Mar 17, 2012 at 18:18
-
2@MichaelKohl Well, Rich Hickey conveniently did not mention that all programs with bugs had one thing in common: they were not proven correct. Of course, since his castle is based on the premise that all programs have bugs, it would not fit well in his presentation, now would it? Commented Mar 22, 2012 at 1:30
No platform can infinitely scale out whilst still dealing with complex sets of data that change moment to moment. Language and infrastructure matters, but how you build your site and the data access patterns matter more.
If you've ever played games like Transport Tycoon or Settlers where you have to transport resources around, you'll know how you need to stay on top of upgrading infrastructure as usage increases.
Scaling platforms like Facebook and Twitter is a never-ending task. You have an ever increasing number of users, and you're being pushed to add more features and functionality. It's a continual process of upgrading one bit, which causes more stress on another bit.
Throwing servers at the problem isn't always the answer, and sometimes can cause more problems.
http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster links to a set of posts about the changes, including a decent history of the steps taken over time.
The short version is that Ruby and Rails didn't deliver the performance and reliability they required for the service. Given the scale, this isn't surprising; most COTS solutions are not satisfactory at the super-large end of scale.
High Scalability covers a lot of questions about architecture at that top end, for other sites, so helps answer broader questions in the area too.
They could have thrown more hardware at the problem, but it is a good deal more expensive then simply writing more efficient code. Like many high-level frameworks, Ruby on Rails is great at many things, but high-performance isn't one of them. Compiled languages will always be faster than interpreted languages.
-
1'faster' - in terms of how many CPU ticks it takes to execute something, sure. In busy platforms it's IO that's generally more of a killer. Whether that's because you need to hit a disk, or wait on a remote server to ship you some set of data. Language is a factor, but excepting certain limited scenarios, probably not a deciding factor.– user111013Commented Mar 17, 2012 at 7:23
-
2In this particular case, if you read the interviews, it's pretty clearly stated that it was a language performance issue.– sosbornCommented Mar 17, 2012 at 10:16
-
4If you compare the costs of rewriting a somewhat substantial app in a new language and the ridiculously low hardware prices of today, I'd be careful in assuming "simply" writing more efficient code is always cheaper. Commented Mar 17, 2012 at 13:34
-
@Michael true that hardware is cheap, but the more users you have, the more important code efficiency becomes, because hardware cost is proportional to user base, but code complexity is weakly correlated if at all. E.g. if you have 10m users and need to double capacity via hardware, it will cost you 10x as much as it would for site with 1m users, but a re-write would cost the same. Twitter's certainly at the high-userbase end of this equation. Commented Mar 17, 2012 at 21:50
-
@MichaelKohl, of course you are correct, but I am talking about this particular situation, where they could rewrite sections on the backend (not the whole app) and they were facing growth not seen by the vast majority of sites.– sosbornCommented Mar 17, 2012 at 23:45
Facebook (and Google) scale by adding more servers, but at the same time they break their application out into various services. Those services communicate via an agreed upon interface and type, and they are now free to build these services out in any technology they see fit. Just because you read that facebook uses php doesn't mean that all their backend services are being served by php (and it doesn't make sense either since in SOA you can choose any tech stack).
I think this video is the best answer to your question:
"From Ruby to the JVM" https://www.youtube.com/watch?v=ohHdZXnsNi8
I think one important bit missing here is the platform. Yes we had the compiled vs interpreted argument and a couple of others. But one very important aspect was indeed the platform. There are different Ruby VMs but none did please twitter, although they tuned it quite a bit. But scala runs on the JVM and twitter engineers hat pretty goog experience with that. Why they they didnt try/choose JRuby? Well I guess the reasons mentioned above come here into play.
-
IIRC, Rails didn't run on JRuby then. And, still IIRC, getting Rails to run on JRuby had a substantial help from Twitter. Commented Mar 22, 2012 at 1:34
-
Ok! My main point here was that it was about the language but ALSO because of the JVM Platform twitter made the decision Commented Mar 22, 2012 at 7:22
-
Sorry the down vote was a miss-click on mobile. It'll take ~53 minutes for the system to let me change the vote unless you edit your answer.– butchCommented Feb 25, 2014 at 18:10
-
Looks like only an edit will let me fix the accidental down vote.– butchCommented Feb 26, 2014 at 1:01
Linear gains with parallelism (which is what multiple servers is) is exceedingly rare, and very application dependent. Yes, it exists -- that's how GPU do most of their work. If you are serving static pages, with no session state, that would also be the case.
For the most part, however, adding servers do not increase performance linearly (ie, 10 servers are not 10 times faster than 1 server), and that means that any gains you can make on a single server will have much more impact than just adding servers. It's not like Twitter doesn't have a bunch of servers, now is it?