226

I need to present a large number of rows of data (ie. millions of rows) to the user in a grid using JavaScript.

The user shouldn't see pages or view only finite amounts of data at a time.

Rather, it should appear that all of the data are available.

Instead of downloading the data all at once, small chunks are downloaded as the user comes to them (ie. by scrolling through the grid).

The rows will not be edited through this front end, so read-only grids are acceptable.

What data grids, written in JavaScript, exist for this kind of seamless paging?

19
  • 1
    I unaccepted the jqgrid answer, since it seems to fail for large data sets... Any other suggestions? What about ext-livegrid.com?
    – Rudiger
    Mar 10, 2010 at 16:06
  • 7
    Write your own. I am sure that the other ones are choking up because they just keep appending to the DOM. I think you will need a solution that removes rows as they scroll off the screen. That is the only way. You simply can't have a million table rows in the DOM and expect every browser to to display and scroll seamlessly in every environment. Be reasonable. Mar 18, 2010 at 0:35
  • 2
    @Rudiger: SlickGrid now supports unlimited numbers of rows natively. See github.com/mleibman/SlickGrid/tree/unlimited-rows . Once this gets tested thoroughly it will be merged into the main branch.
    – Tin
    Jun 28, 2010 at 6:06
  • 11
    And I feel sorry which ever firm you are working for. For your information, a 1920x1080 screen with only 1 million rows displayed, will jump 20 row for every one pixel of movement on the scroll bar. Go do some usability testing instead of wasting your time. Mar 18, 2011 at 6:34
  • 3
    This question and its top two answers (at least) are exceedingly useful. It might have attracted some low-quality answers, but by no means should this question be Closed. Using SlickGrid to solve this problem can save people many many hours of trouble and difficult coding, if they attempt to reimplement this for themselves. Apr 11, 2016 at 0:58

19 Answers 19

192

(Disclaimer: I am the author of SlickGrid)

UPDATE This has now been implemented in SlickGrid.

Please see http://github.com/mleibman/SlickGrid/issues#issue/22 for an ongoing discussion on making SlickGrid work with larger numbers of rows.

The problem is that SlickGrid does not virtualize the scrollbar itself - the scrollable area's height is set to the total height of all the rows. The rows are still being added and removed as the user is scrolling, but the scrolling itself is done by the browser. That allows it to be very fast yet smooth (onscroll events are notoriously slow). The caveat is that there are bugs/limits in the browsers' CSS engines that limit the potential height of an element. For IE, that happens to be 0x123456 or 1193046 pixels. For other browsers it is higher.

There is an experimental workaround in the "largenum-fix" branch that raises that limit significantly by populating the scrollable area with "pages" set to 1M pixels height and then using relative positioning within those pages. Since the height limit in the CSS engine seems to be different and significantly lower than in the actual layout engine, this gives us a much higher upper limit.

I am still looking for a way to get to unlimited number of rows without giving up the performance edge that SlickGrid currently holds over other implementations.

Rudiger, can you elaborate on how you solved this?

6
  • 1
    I have found SlickGrid to be the most appealing - especially if one works with jQuery. Congrats! (esp. for the great attitude and persistence.) :-) Jul 26, 2010 at 10:41
  • I am trying to use slickgrid to show excel headers, and I am seeing that when having too many columns slickgrid only optimizes the scrolling of rows and not of columns. I have also noticed that when having more than 120 columns or so - slickgrid puts the new rows in a new line. Can the maximum number of rows be set somewhere in the files?
    – oneiros
    Feb 25, 2012 at 2:19
  • 1
    SlickGrid v2.1 has uses virtual scrolling for columns as well as rows. Also, the overflowing columns issue has been resolved.
    – Tin
    Sep 27, 2012 at 20:25
  • @Tin - This is similar to my approach; I was years ahead of my time! "A lazy block layout primitive for building infinite scroll into web applications." docs.google.com/document/d/…
    – Rudiger
    Sep 19, 2013 at 17:42
  • @Rudiger Yes, I've seen this on Blink group about a month ago, but I'm not quite sure how this fit into the picture. The lazy layout operates on elements actually present in the DOM, which we can't really do. Please elaborate :)
    – Tin
    Sep 19, 2013 at 18:33
85
+250

https://github.com/mleibman/SlickGrid/wiki

"SlickGrid utilizes virtual rendering to enable you to easily work with hundreds of thousands of items without any drop in performance. In fact, there is no difference in performance between working with a grid with 10 rows versus a 100’000 rows."

Some highlights:

  • Adaptive virtual scrolling (handle hundreds of thousands of rows)
  • Extremely fast rendering speed
  • Background post-rendering for richer cells
  • Configurable & customizable
  • Full keyboard navigation
  • Column resize/reorder/show/hide
  • Column autosizing & force-fit
  • Pluggable cell formatters & editors
  • Support for editing and creating new rows." by mleibman

It's free (MIT license). It uses jQuery.

5
  • It works fine until precisely 131,001 rows... That is, there's a line of code like this: data.length = Math.min(131000, parseInt(resp.total)); ... And, of course, that hard-coded for a reason :(
    – Rudiger
    Mar 18, 2010 at 0:00
  • 6
    It took a little work, but I made a few changes to make grid independent of the length of the data array. It's a kludge, but I have the responses populating a bigdata array, and the smaller data pulls from the bigdata array. The rest of the program uses the smaller data array, except for the scroll-bar measurement and a few other places that are now unbounded for a large number of rows. All in all, was much easier than writing my own.
    – Rudiger
    Mar 18, 2010 at 14:41
  • 9
    @Rudiger: SlickGrid now supports unlimited numbers of rows natively. See github.com/mleibman/SlickGrid/tree/unlimited-rows . Once this gets tested thoroughly it will be merged into the main branch.
    – Tin
    Jun 28, 2010 at 6:07
  • I am trying to use slickgrid to show excel headers, and I am seeing that when having too many columns slickgrid only optimizes the scrolling of rows and not of columns. I have also noticed that when having more than 120 columns or so - slickgrid puts the new rows in a new line. Can the maximum number of rows be set somewhere in the files?
    – oneiros
    Feb 25, 2012 at 2:20
  • if you want something really fast, don't rely on anything that uses jquery to do things in the core, and rather use innerHTML than DOM append. Javascript scrollbars can be noticeable slower than browser scrollbars on slow computers, avoid complex css rules, and you should spend time simplifying the layout of a single row. Micro-optimizations could be significative in this case. This is just general pratices to improve performance. jsPerf.com is your friend.
    – Vitim.us
    Nov 10, 2012 at 2:22
38

The best Grids in my opinion are below:

My best 3 options are jqGrid, jqxGrid and DataTables. They can work with thousands of rows and support virtualization.

6
24

I don't mean to start a flame war, but assuming your researchers are human, you don't know them as well as you think. Just because they have petabytes of data doesn't make them capable of viewing even millions of records in any meaningful way. They might say they want to see millions of records, but that's just silly. Have your smartest researchers do some basic math: Assume they spend 1 second viewing each record. At that rate, it will take 1000000 seconds, which works out to more than six weeks (of 40 hour work-weeks with no breaks for food or lavatory).

Do they (or you) seriously think one person (the one looking at the grid) can muster that kind of concentration? Are they really getting much done in that 1 second, or are they (more likely) filtering out the stuff the don't want? I suspect that after viewing a "reasonably-sized" subset, they could describe a filter to you that would automatically filter out those records.

As paxdiablo and Sleeper Smith and Lasse V Karlsen also implied, you (and they) have not thought through the requirements. On the up side, now that you've found SlickGrid, I'm sure the need for those filters became immediately obvious.

3
  • 2
    Having the need for millions of rows isn't always about viewing them. Sometimes clients want a partial dump of records for running in their own data analysis systems.
    – cbmeeks
    Oct 30, 2012 at 22:05
  • 10
    If its a data dump for their own analysis, then it wouldn't be displayed in a grid on a webpage, would it? Feb 4, 2013 at 20:34
  • 6
    i don't have to see them all at once. That's what column sorting and Ctrl+F are for. The alternative (paging, web-site searching) is much worse. Just look at StackOverflow when trying to scroll through questions or answers, Reddit for scrolling through a user's comment history. Sorting and instant searching provide a power that Windows Explorer has, but web-sites lack.
    – Ian Boyd
    Jul 10, 2014 at 17:57
14

I can say with pretty good certainty that you seriously do not need to show millions of rows of data to the user.

There is no user in the world that will be able to comprehend or manage that data set so even if you technically manage to pull it off, you won't solve any known problem for that user.

Instead I would focus on why the user wants to see the data. The user does not want to see the data just to see the data, there is usually a question being asked. If you focus on answering those questions instead, then you would be much closer to something that solves an actual problem.

1
  • 17
    My users are researchers who are used to petabytes of data. I think I know my users a little bit better than you do, though you're certainly right in the general case. As for the why, this datagrid is simply part of a set of tools for managing big data.
    – Rudiger
    Nov 5, 2010 at 15:12
7

I recommend the Ext JS Grid with the Buffered View feature.

http://www.extjs.com/deploy/dev/examples/grid/buffer.html

3
  • ExtJs, indeed. It's basically built especially for data presentation
    – KdgDev
    Mar 13, 2010 at 23:08
  • 1
    ExtJs is so good I want to cry that its not built on top of jQuery Apr 6, 2010 at 22:59
  • Now you can load only the grid-related portions of ExtJS, so that adding an ExtJS grid to your application won't be too heavy. However, you still have to consider the differences in appearance and use the ExtJS way of theming for just that component.
    – JD Smith
    Jul 11, 2012 at 19:19
7

(Disclaimer: I am the author of w2ui)

I have recently written an article on how to implement JavaScript grid with 1 million records (http://w2ui.com/web/blog/7/JavaScript-Grid-with-One-Million-Records). I discovered that ultimately there are 3 restrictions that prevent from taking it highter:

  1. Height of the div has a limit (can be overcome by virtual scrolling)
  2. Operations such as sort and search start being slow after 1 million records or so
  3. RAM is limited because data is stored in JavaScript array

I have tested the grid with 1 million records (except IE) and it performs well. See article for demos and examples.

1
  • With this one million record your html page is of size 3MB, But when I load my data the page is of 15MB, does w2ui can handle that? I need all data for some calculations. Dec 15, 2016 at 12:02
6

dojox.grid.DataGrid offers a JS abstraction for data so you can hook it up to various backends with provided dojo.data stores or write your own. You'll obviously need one that supports random access for this many records. DataGrid also provides full accessibility.

Edit so here's a link to Matthew Russell's article that should provide the example you need, viewing millions of records with dojox.grid. Note that it uses the old version of the grid, but the concepts are the same, there were just some incompatible API improvements.

Oh, and it's totally free open source.

4

I used jQuery Grid Plugin, it was nice.

Demos

2
4

Here are a couple of optimizations you can apply you speed up things. Just thinking out loud.

Since the number of rows can be in the millions, you will want a caching system just for the JSON data from the server. I can't imagine anybody wanting to download all X million items, but if they did, it would be a problem. This little test on Chrome for an array on 20M+ integers crashes on my machine constantly.

var data = [];
for(var i = 0; i < 20000000; i++) {
    data.push(i);
}
console.log(data.length);​

You could use LRU or some other caching algorithm and have an upper bound on how much data you're willing to cache.

For the table cells themselves, I think constructing/destroying DOM nodes can be expensive. Instead, you could just pre-define X number of cells, and whenever the user scrolls to a new position, inject the JSON data into these cells. The scrollbar would virtually have no direct relationship to how much space (height) is required to represent the entire dataset. You could arbitrarily set the table container's height, say 5000px, and map that to the total number of rows. For example, if the containers height is 5000px and there are a total of 10M rows, then the starting row ≈ (scroll.top/5000) * 10M where scroll.top represents the scroll distance from the top of the container. Small demo here.

To detect when to request more data, ideally an object should act as a mediator that listens to scroll events. This object keeps track of how fast the user is scrolling, and when it looks like the user is slowing down or has completely stopped, makes a data request for the corresponding rows. Retrieving data in this fashion means your data is going to be fragmented, so the cache should be designed with that in mind.

Also the browser limits on maximum outgoing connections can play an important part. A user may scroll to a certain position which will fire an AJAX request, but before that finishes the user can scroll to some other portion. If the server is not responsive enough the requests would get queued up and the application will look unresponsive. You could use a request manager through which all requests are routed, and it can cancel pending requests to make space.

4

I know it's an old question but still.. There is also dhtmlxGrid that can handle millions of rows. There is a demo with 50,000 rows but the number of rows that can be loaded/processed in grid is unlimited.

Disclaimer: I'm from DHTMLX team.

1
  • I want to show 10 MB Json data and want to use it in calculation, can DHTMLX do that thing, With that data and html tags my browser's page is of around 15 MB. Do it worth using DHTMLX? Dec 15, 2016 at 11:58
3

I suggest you read this

http://www.sitepen.com/blog/2008/11/21/effective-use-of-jsonreststore-referencing-lazy-loading-and-more/

3

Disclaimer: i heavily use YUI DataTable without no headache for a long time. It is powerful and stable. For your needs, you can use a ScrollingDataTable wich suports

  • x-scrolling
  • y-scrolling
  • xy-scrolling
  • A powerful Event mechanism

For what you need, i think you want is a tableScrollEvent. Its API says

Fired when a fixed scrolling DataTable has a scroll.

As each DataTable uses a DataSource, you can monitoring its data through tableScrollEvent along with render loop size in order to populate your ScrollingDataTable according to your needs.

Render loop size says

In cases where your DataTable needs to display the entirety of a very large set of data, the renderLoopSize config can help manage browser DOM rendering so that the UI thread does not get locked up on very large tables. Any value greater than 0 will cause the DOM rendering to be executed in setTimeout() chains that render the specified number of rows in each loop. The ideal value should be determined per implementation since there are no hard and fast rules, only general guidelines:

  • By default renderLoopSize is 0, so all rows are rendered in a single loop. A renderLoopSize > 0 adds overhead so use thoughtfully.
  • If your set of data is large enough (number of rows X number of Columns X formatting complexity) that users experience latency in the visual rendering and/or it causes the script to hang, consider setting a renderLoopSize.
  • A renderLoopSize under 50 probably isn't worth it. A renderLoopSize > 100 is probably better.
  • A data set is probably not considered large enough unless it has hundreds and hundreds of rows.
  • Having a renderLoopSize > 0 and < total rows does cause the table to be rendered in one loop (same as renderLoopSize = 0) but it also triggers functionality such as post-render row striping to be handled from a separate setTimeout thread.

For instance

// Render 100 rows per loop
 var dt = new YAHOO.widget.DataTable(<WHICH_DIV_WILL_STORE_YOUR_DATATABLE>, <HOW YOUR_TABLE_IS STRUCTURED>, <WHERE_DOES_THE_DATA_COME_FROM>, {
     renderLoopSize:100
 });

<WHERE_DOES_THE_DATA_COME_FROM> is just a single DataSource. It can be a JSON, JSFunction, XML and even a single HTML element

Here you can see a Simple tutorial, provided by me. Be aware no other DATA_TABLE pluglin supports single and dual click at the same time. YUI DataTable allows you. And more, you can use it even with JQuery without no headache

Some examples, you can see

Feel free to question about anything else you want about YUI DataTable.

regards,

0
3

I kind of fail to see the point, for jqGrid you can use the virtual scrolling functionality:

http://www.trirand.net/aspnetmvc/grid/performancevirtualscrolling

but then again, millions of rows with filtering can be done:

http://www.trirand.net/aspnetmvc/grid/performancelinq

I really fail to see the point of "as if there are no pages" though, I mean... there is no way to display 1,000,000 rows at once in the browser - this is 10MB of HTML raw, I kind of fail to see why users would not want to see the pages.

Anyway...

2

best approach i could think of is by loading the chunk of data in json format for every scroll or some limit before the scrolling ends. json can be easily converted to objects and hence table rows can be constructed easily unobtrusively

4
  • That's how I have it. A request is made for a set of rows sent back in JSON... I'm looking for a javascript client side renderer that supports this!
    – Rudiger
    Mar 10, 2010 at 16:07
  • What??? What the heck is "client site renderer"? Any javascript will still need to make an ajax call - so you will still need to settle on some transport format. You can't escape doing some work. No-one will do this for you my friend. Mar 16, 2010 at 15:19
  • 1
    I know that an AJAX call has to be made; this part is simple. The client requests something like "start=20&limit=20" and retrieves rows 20-39 from the server (XML or JSON format). The "client-side renderer" (my terminology!) makes these requests intelligently (eg. when the user scrolls down), and renders the results seamlessly in a pretty grid. Contrary to what you say, someone else has done this work for me. That's what all the other answers to this question are.
    – Rudiger
    Mar 16, 2010 at 17:13
  • Well, it seems no one "else" has done it for you:) Mar 18, 2010 at 1:24
1

I would highly recommend Open rico. It is difficult to implement in the the beginning, but once you grab it you will never look back.

0
1

I know this question is a few years old, but jqgrid now supports virtual scrolling:

http://www.trirand.com/blog/phpjqgrid/examples/paging/scrollbar/default.php

but with pagination disabled

0

I suggest sigma grid, sigma grid has embed paging features which could support millions of rows. And also, you may need a remote paging to do it. see the demo http://www.sigmawidgets.com/products/sigma_grid2/demos/example_master_details.html

0

Take a look at dGrid:

https://dgrid.io/

I agree that users will NEVER, EVER need to view millions of rows of data all at once, but dGrid can display them quickly (a screenful at a time).

Don't boil the ocean to make a cup of tea.

2
  • your cup of tea (link) is not found. :)
    – Akshay
    Aug 28, 2019 at 7:14
  • It has its own site now :)
    – ColemanTO
    Aug 28, 2019 at 17:52

Not the answer you're looking for? Browse other questions tagged or ask your own question.