I will answer my own question as I managed to solve it.
It seems each time you run your tests in Visual Studio, it stores the result of your tests in TestResults folder in the project folder and it was about 800MB in size in my computer.
I just deleted it and clean the project bin and obj folders as well (which I did for several times before and didn't really work). But removing TestResults folder did the magic.
My guess is that because of the humongous size of the TestResults folder, Visual Studio 2013 cannot load all the results etc. quickly and that's why it was taking years to show me Code Lens feature. So delete that folder time to time for better performance.
Here are some results about TestResults folder after I googled it:
- How to: Deploy Files for Tests: http://msdn.microsoft.com/en-us/library/ms182475.aspx
- How to preserve the test results directory?: http://blogs.msdn.com/b/aseemb/archive/2013/02/23/how-to-preserve-the-test-results-directory.aspx
- How to delete Test Results folder:http://blogs.msdn.com/b/ploeh/archive/2006/07/13/cleaningawaythetestresultsfolder.aspx
UPDATE
Even if those methods above may work for you, I realized that they actually fixed this issue since Visual Studio 2013 Update 2. So instead, please update and install it.
Tools -> Options -> Text Editor -> All Languages -> CodeLens
? – Scott Solmer Dec 2 '13 at 22:28