I've worked on both mostly-server-rendering and mostly-client-rendering applications. Each type has its own advantages and disadvantages. The idea that you have to choose between one or the other is a false dichotomy however. If you have the resources you can combine both to get the best of both worlds.
I see 4 main challenges with purely client-side frameworks:
- SEO and Analytics
- Caching
- Memory
- Latency
SEO
Because you are using Node.JS, the SEO problem can be mitigated by simply using the client-side framework on the server to output static pages for googlebot and company. Recently Google has made a nice Analytics API for single page applications, but that will be a little more work than simply adding a couple of lines to the end of your master template.
Caching
Caching is really important way to speed up any web application. For small amounts of data it can be faster to cache data on the client in memory or in localStorage, but storage space on is very limited (currently about 5MB). Plus cache invalidation is pretty hard to do in localStorage.
Memory
Memory is something I've paid dearly for overlooking. Before I knew it I had accidentally made an Application that takes up more than 200MB of RAM. I might be able to bring that down to half with optimizations, but I doubt it would've taken more than 20 MB if I rendered it all on the server.
Latency
Latency is also easy to miss. Drupal for example runs about 50 to 100 SQL queries for each page. When the Database server is right next to the Application server, you don't have to worry about latency and all those queries can be executed in less than a couple hundred milliseconds. Your client side application will usually take a hundred milliseconds to make a single AJAX request. This means you need to spend a lot of time designing your server side API to minimize these round trips, but at that point the server already has all the data it needs to just generate the HTML too. Having a client-side application that talks to a properly RESTful interface can turn out to be glacially slow if you are not careful.
37 Signals recently blogged about the hybrid client/server architecture they implemented for the new version of Basecamp. This hybrid approach uses the server to render HTML but leverages something like PJAX on the client to get rid of full page refreshes. The effect is really fast and is what I recommend.