0

We have a data set that is fairly static in a MySQL database, but the read times are terrible (even with indexes on the columns being queried). The theory is that since rows are stored randomly (or sometimes in order of insertion), the disk head has to scan around to find different rows, even if it knows where they are due to the index, instead of just reading them sequentially.

Is it possible to change the order data is stored in on disk so that it can be read sequentially? Unfortunately, we can't add a ton more RAM at the moment to have all the queries cached. If it's possible to change the order, can we define an order within an order? As in, sort by a certain column, then sort by another column if the first column is equal.

Could this have something to do with the indices?

Additional details: non-relational single-table database with 16 million rows, 1 GB of data total, 512 mb RAM, MariaDB 5.5.30 on Ubuntu 12.04 with a standard hard drive. Also this is a virtualized machine using OpenVZ, 2 dedicated core E5-2620 2Ghz CPU

Create syntax:

CREATE TABLE `Events` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `provider` varchar(10) DEFAULT NULL,
  `location` varchar(5) DEFAULT NULL,
  `start_time` datetime DEFAULT NULL,
  `end_time` datetime DEFAULT NULL,
  `cost` int(11) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `provider` (`provider`),
  KEY `location` (`location`),
  KEY `start_time` (`start_time`),
  KEY `end_time` (`end_time`),
  KEY `cost` (`cost`)
) ENGINE=InnoDB AUTO_INCREMENT=16321002 DEFAULT CHARSET=utf8;

Select statement that takes a long time:

SELECT *
FROM `Events` 
WHERE `Events`.start_time >= '2013-05-03 23:00:00' AND `Events`.start_time <= '2013-06-04 22:00:00' AND `FlightRoutes`.location = 'Chicago'

Explain select:

1   SIMPLE  Events  ref location,start_time location    18  const   3684    Using index condition; Using where
3
  • Yes, it's possible that random disk I/O is your problem; and yes, there are things that you can do to restructure the data on disk. However, before going that way, let's check that your queries are using suitable indexes; please show the table schema, the query and its execution plan: SHOW CREATE TABLE foo, SELECT ... and EXPLAIN SELECT ....
    – eggyal
    May 15, 2013 at 5:16
  • Thank you so much for answering. Syntax and more info added above.
    – AC360
    May 15, 2013 at 5:20
  • Your data isn't stored "randomly", MySQL orders the data according to your primary key ("clustered index"). It might wind up randomly on disk due to fragmentation of the file where the data is stored in. From MySQL's point of view the data is ordered by id May 15, 2013 at 7:00

1 Answer 1

0

MySQL can only select one index upon which to filter (which makes sense, because having restricted the results using an index it cannot then determine how such restriction has affected other indices). Therefore, it tracks the cardinality of each index and chooses the one that is likely to be the most selective (i.e. has the highest cardinality): in this case, it has chosen the location index, but that will typically leave 3,684 records that must be fetched and then filtered Using where to find those that match the desired range of start_time.

You should try creating a composite index over (location, start_time):

ALTER TABLE Events ADD INDEX (location, start_time)
3
  • Thank you so much! It works a TON faster!! Now I'm wondering if there's any other query optimization we can do? What stuff should I look into to maximize disk I/O? Thanks again!
    – AC360
    May 15, 2013 at 6:44
  • @AC360: There are tradeoffs to everything one does, so be careful only to achieve satisfactory performance; remember Knuth's maxim "premature optimisation is the root of all evil". If, however, performance is still unsatisfactory, you should first profile your query to discover exactly where time is being spent; if disk I/O really is presenting a material problem for you, you might consider partitioning your data - but not before tweaking various server- and engine-specific settings; in many cases, however, hardware upgrade may be preferred.
    – eggyal
    May 15, 2013 at 6:55
  • @eggyval: Just a side note: the "which makes sense" is not entirely true. There are several DBMS that can efficiently use more than one index for one table in a query. May 15, 2013 at 6:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.