1

I'm tryin to achieve some pagination with random ordering of the items. I've got a solution and i want an expert advice if its ok or not.

My table loos like:

id | category | product_name | product_description | product_image | filter1 | filter2 | filter 3 .. bla bla bla

And i want to paginate search results based on some filters and randomize the results.

I thinked that i should make another table where i keep only the id of the product, the category and filters with indexeses for all posibilities of filtering. Then, when a search occurs, i search trough the smaller table based on some specific index and keep the ids in an array and randomize them in PHP.

EDIT: My tables has like 500k - 1 mil entries.

It will be slow?

6
  • Have you heard of pagination using LIMIT and OFFSET? Aug 27, 2017 at 9:45
  • Of course, but how to randomize the results :) ? Aug 27, 2017 at 9:49
  • 1
    ORDER BY RAND() , and out of that use LIMIT and OFFSET
    – sagi
    Aug 27, 2017 at 9:49
  • Edited main post: My tables has like 500k - 1 mil entries. Aug 27, 2017 at 9:51
  • After all filtering, how large do you expect the resultsets to be? Couple of records? Hundreds? Thousands?
    – Shadow
    Aug 27, 2017 at 10:35

2 Answers 2

0
  1. Filter 1M rows to find 100K rows -- a full table scan of 1M rows
  2. Pick, say, 10 random rows from the 100K -- a scan of 100K rows, unless you use some technique from http://mysql.rjweb.org/doc.php/random .
  3. Leave behind something that lets you "paginate". Do you care if it accidentally repeats rows? Is this web-based, meaning that "leaving something behind" can be messy -- such as listing all the items already seen when in the url of the [Next] button.
  4. Pagination via LIMIT and OFFSET is Order(N*N) to walk through all N pages. 100K rows, 10 rows at a time, would be something like 500M. Even worse than the original 1M.

Now, let me invent a solution that solves the cost of both "random" and "pagination". See http://mysql.rjweb.org/doc.php/random#case_extra_float_column_for_randomizing

  1. Prep: Add an extra column with some random number in it. Index it. This will become both the randomizer and the "where you left off" to avoid using the costly OFFSET.
  2. See the link for one approach to getting the first 10 random rows.
  3. See http://mysql.rjweb.org/doc.php/pagination for more on "remembering where you left off".

Do all those, and the cost becomes mostly the original Step 1, namely whatever it takes to filter the 1M rows. Hopefully, it won't even be that costly -- this depends on whether a suitable index can be had that gets all the way to the LIMIT 10. (My gut says that won't be practical.)

Summing up:

SELECT ...
    FROM ...
    WHERE filters...
       AND rnd > $left_off
    ORDER BY rnd
    LIMIT 10;

If you start $left_off at 0, you get a repeatable, but seemingly random, sequence. And the [Next] button only needs to pass the last value of rnd that was fetched from the current page.

Now to make it not repeatable? Re-populate rnd every night. (That would mess with anyone in the process of paginating.)

0

Similar questions was asked before. Here is one quick selection of a random row from a large table in mysql

I thinked that i should make another table where i keep only the id of the product, the category and filters with indexeses for all posibilities of filtering. Then, when a search occurs, i search trough the smaller table based on some specific index and keep the ids in an array and randomize them in PHP.

You are not able to do it in real world project. Example: You have 10-15 filters with 5-15 options for each filter.

Let's do some math.

For minimum version you have 31*31 combinations. Database is able to handle it. For maximum in this example you have 32767*1023=33520641 combination. This is only the number of combinations without products matched those filters.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.