1

I have the following MySQL table:

CREATE TABLE IF NOT EXISTS `pics` (
  `id` mediumint(8) unsigned NOT NULL auto_increment,
  `bnb_id` mediumint(7) unsigned NOT NULL,
  `img_path` varchar(128) NOT NULL,
  `img_path_gallery` varchar(128) NOT NULL,
  `img_path_thumb_small` varchar(128) NOT NULL,
  `img_path_thumb_large` varchar(128) NOT NULL,
  `img_path_thumb_grid` varchar(128) NOT NULL,
  `title` varchar(80) NOT NULL,
  `order` tinyint(2) NOT NULL,
  `upload_date` datetime NOT NULL,
  `state` enum('LOCAL','S3') NOT NULL default 'LOCAL',
  `is_cover` tinyint(1) unsigned default NULL,
  PRIMARY KEY  (`id`),
  UNIQUE KEY `bnb_id_2` (`bnb_id`,`is_cover`),
  KEY `bnb_id` (`bnb_id`),
  KEY `is_cover` (`is_cover`)
) ENGINE=InnoDB  DEFAULT CHARSET=utf8 AUTO_INCREMENT=30371 ;

is_cover is a field I created to choose only one picture for each bnb_id: it is set to 1 when the picture has been choosen as a cover and to NULL otherwise. I need to LEFT JOIN the table to another one, let's call it bnb; there may be multiple rows in the pics table for each bnb entry (there is a referential integrity bound on bnb_id), but in this case I must extract only one row from the pics table, hence the need for the is_cover coulmn and all the indexing (every other solution I tried produced queries lasting anywhere from 10 to 50 seconds).

Even in this case, though, queries are very slow and each takes anywhere from 5 to 8 seconds of execution on a data pool of about 10000 elements in the bnb table and 30000 in the pics table. Selecting from the table where is_cover = 1 is pretty fast and straightforward, but when put in a larger query everything breaks up.

SELECT subbnb.*, 
            3956 * 2 * ASIN(
                SQRT(
                    POWER(
                        SIN((_LAT - abs(lat)) * pi()/180 / 2), 
                    2) +
                    COS(_LAT * pi()/180 ) * 
                    COS(abs(lat) * pi()/180) * 
                    POWER(
                        SIN((_LNG - abs(lng)) * pi()/180 / 2), 
                    2) 
                )
            ) AS distance,
            prices.price,
            pics.img_path_thumb_grid,
            reviews.count reviewsCount,
            likes.count likesCount
        FROM 
            (SELECT
                bnb.*,
                bnbdata_a.*,
                pos.lat,
                pos.lng

                FROM bnb

                JOIN bnbdata 
                    ON (bnb.id = bnbdata.bnb_id)

                JOIN positions pos
                    ON (bnb.id = pos.bnb_id) 
            ) subbnb

            LEFT JOIN (
                    SELECT *
                    FROM pics 
                    WHERE is_cover = 1
                ) pics
                ON (subbnb.id = pics.bnb_id)


            LEFT JOIN (SELECT price, bnb_id FROM prices WHERE category = "DAILY") prices
                ON (subbnb.id = prices.bnb_id)

            LEFT JOIN (SELECT COUNT(*) AS count, bnb_id FROM reviews GROUP BY bnb_id) reviews
                ON (subbnb.id = reviews.bnb_id)

            LEFT JOIN (SELECT COUNT(*) AS count, bnb_id FROM likes GROUP BY bnb_id) likes
                ON (subbnb.id = likes.bnb_id)
        WHERE
            lng BETWEEN _LNGA AND  _LNGB
            AND lat BETWEEN _LATA AND  _LATB
        HAVING distance < 10
        ORDER BY distance
        LIMIT 0, 25

(the strings you see with _ prepended are actual numerical values)

EXPLAINing the query produces the following result:

id  select_type table   type    possible_keys   key key_len ref rows    Extra
1   PRIMARY <derived5>  system  NULL    NULL    NULL    NULL    0   const row not found
1   PRIMARY <derived6>  system  NULL    NULL    NULL    NULL    0   const row not found
1   PRIMARY <derived2>  ALL NULL    NULL    NULL    NULL    10522   Using where; Using temporary; Using filesort
1   PRIMARY <derived3>  ALL NULL    NULL    NULL    NULL    7040    
1   PRIMARY <derived4>  ALL NULL    NULL    NULL    NULL    1   
6   DERIVED likes   index   NULL    PRIMARY 6   NULL    1   Using index
5   DERIVED reviews index   NULL    bnb_id  5   NULL    1   Using index
4   DERIVED prices  ALL NULL    NULL    NULL    NULL    1   Using where
3   DERIVED pics    ref is_cover    is_cover    2       11760   Using where
2   DERIVED pos ALL PRIMARY NULL    NULL    NULL    10543   
2   DERIVED bnbdata eq_ref  PRIMARY PRIMARY 3   db.pos.bnb_id   1   
2   DERIVED bnb eq_ref  PRIMARY PRIMARY 3   db.pos.bnb_id   1   

It looks like the is_cover index is being ignored by MySQL (Using where, id 4) but the same happens when I run the small select against the pics table and everything happens very quickly. I can't find the bottleneck in this query, removing the JOIN to pics makes everything way faster but the JOINed subquery itself is quite fast and so is the rest of the big query - even with the math-computing code at the beginning it never goes much past 2 seconds of execution.

Does anybody know where the bottleneck is, and how to get around that?

1 Answer 1

1

You could try rebuilding your query using joins like this one (sorry if incorrect but you only described one table):

SELECT
  bnb.*, bnbdata_a.*, 
  pos.lat, pos.lng
  3956 * 2 * ASIN(
    SQRT(
      POWER(
        SIN((_LAT - abs(lat)) * pi()/180 / 2), 
      2) +
      COS(_LAT * pi()/180 ) * 
      COS(abs(lat) * pi()/180) * 
      POWER(
        SIN((_LNG - abs(lng)) * pi()/180 / 2), 
      2) 
    )
  ) AS distance,
  prices.price,
  pics.img_path_thumb_grid,
  reviews.count reviewsCount,
  likes.count likesCount
FROM bnb
JOIN bnbdata 
  ON bnb.id = bnbdata.bnb_id
JOIN positions pos 
  ON bnb.id = pos.bnb_id
LEFT JOIN pics 
  ON bnb.id = pics.bnb_id AND pics.is_cover = 1
LEFT JOIN prices 
  ON bnb.id = prices.bnb_id 
LEFT JOIN (SELECT COUNT(*) AS count, bnb_id FROM reviews GROUP BY bnb_id) reviews
  ON bnb.id = reviews.bnb_id
LEFT JOIN (SELECT COUNT(*) AS count, bnb_id FROM likes GROUP BY bnb_id) likes
  ON bnb.id = likes.bnb_id
WHERE
  lng BETWEEN _LNGA AND _LNGB AND lat BETWEEN _LATA AND _LATB AND distance < 10
ORDER BY distance
LIMIT 0, 25

Or rebuild like that:

SELECT tmp_bnb.*,
  pics.img_path_thumb_grid,
  reviews.count reviewsCount,
  likes.count likesCount 
FROM     
  (
    SELECT
      bnb.*, bnbdata_a.*, 
      pos.lat, pos.lng
      3956 * 2 * ASIN(
      SQRT(
        POWER(
          SIN((_LAT - abs(lat)) * pi()/180 / 2), 
        2) +
        COS(_LAT * pi()/180 ) * 
        COS(abs(lat) * pi()/180) * 
        POWER(
          SIN((_LNG - abs(lng)) * pi()/180 / 2), 
        2) 
      )
      ) AS distance,
      prices.price
    FROM bnb
    JOIN bnbdata 
      ON bnb.id = bnbdata.bnb_id
    JOIN positions pos 
      ON bnb.id = pos.bnb_id
    WHERE
      lng BETWEEN _LNGA AND _LNGB AND lat BETWEEN _LATA AND _LATB AND distance < 10
    ORDER BY distance
    LIMIT 0, 25
  ) as tmp_bnb
LEFT JOIN pics 
  ON tmp_bnb.id = pics.bnb_id AND pics.is_cover = 1
LEFT JOIN prices 
  ON tmp_bnb.id = prices.bnb_id 
LEFT JOIN (SELECT COUNT(*) AS count, bnb_id FROM reviews GROUP BY bnb_id) reviews
  ON tmp_bnb.id = reviews.bnb_id
LEFT JOIN (SELECT COUNT(*) AS count, bnb_id FROM likes GROUP BY bnb_id) likes
  ON tmp_bnb.id = likes.bnb_id

Or you could split your query in two and in first query you get basic info and in second you get additional info like rewiews count and likes count.

I also think that a good idea would be adding reviews_counter and likes_counter to bnb table and not count it every time but once per some time (hour maby) or increment it using insert trigger. Also sonsider adding new column cover_pic_id that would hold id of cover picture in bnb table

Let me know how is performance.

4
  • I didn't know you could put AND clauses in JOINs' conditions! Haven't tried yet though: meanwhile I solved the problem by creating a view of the cover pictures, execution times dropped from 8s to 0.2s. I guess the bulk of execution time is now taken by the math calculations but haven't found a way around that yet. Now I'm going to try your solution and I'll let you know :)
    – veeenu
    Dec 3, 2012 at 9:01
  • UPDATE: your first solution is marginally faster, consistently drops execution time from an average of 0.28s to an average of 0.23s, will check the second one asap and let you know!
    – veeenu
    Dec 3, 2012 at 10:48
  • I think that even faster would be adding cover_pic_id to bnb table and joining by that field. You also could do those calculations when inserting new rows not every time you select data. And yes, you can use conditions in JOIN clausule.
    – piotrekkr
    Dec 3, 2012 at 13:38
  • Problem with those calculations is that they're user-input dependent (namely, this SELECT finds the entries nearest the supplied latitude-longitude coordinates) so I can't run them at INSERT time. Adding cover_pic_id to bnb could be an idea!
    – veeenu
    Dec 3, 2012 at 14:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.