1

Ok, I have a complicated query to get all article details, each article has many versions. I have a need to get details of article with latest version only (ie take the max version). Here is the table:

+------------+-----------+----------+
| ArticleID  |  Detail   |  Version |
+------------+-----------+----------+
| 1          |  detail1  |  1       |
| 1          |  detail2  |  1       |
| 1          |  detail3  |  2       |
| 1          |  detail4  |  2       |
| 3          |  detail3  |  2       |
| 3          |  detail6  |  2       |
| 3          |  detail4  |  3       |
+------------+-----------+----------+

Now user just provides a detail & the query will take all details of all articles with version=max(version)

Suppose that if we don't care about max version, then a simple query could be

Select * from articleTb where Detail like '%3'

It will print out:

+------------+-----------+----------+
| ArticleID  |  Detail   |  Version |
+------------+-----------+----------+
| 1          |  detail3  |  2       |
| 3          |  detail3  |  2       |
+------------+-----------+----------+

But this doesn't meet the requirement cos the result should not have this record 3 - detail3 - 2 cos it doesn't contain the max version of articleID=3.

Let say user search for Detail like '%4', then a correct query should be:

ArticleID - Detail - Version
+----+-----------+----+
| 1  |  detail4  |  2 |
+----+-----------+----+
| 3  |  detail4  |  3 |
+----+-----------+----+

The 2 records appear cos they belongs to the article with max version. Explain, 2 is the maxversion of articleID=1 so it matches the condition, & 3 is the max version of articleID=3 so it also matches the condition.

So here is what i did,

select * from (Select * from articleTb where Detail like '%3') tb1
Join (select articleID, max(version) maxversion from articleTb where 
Detail like '%3' group by articleID) tb2
on tb1.articleID=tb2.articleID and tb1.version=tb2.maxversion

However, for the above query the system have to duplicate the task where Detail like '%3' which is not good. Besides, my real world query1 is much more complicated than where Detail like '%3', then if i do like the above then the query will implement the same job TWICE? & that is very inefficient.

So how to deal this problem?

2
  • ok, it will return "1 - detail4 - 2" & "3 - detail4 - 3" right? & 2 is the maxversion of ArticleID=1 so it matches the condition, & 3 is the max version of articleID=3 so it also matches the condition
    – Tum
    Mar 5, 2014 at 1:32
  • This question appears to be off-topic because it appears to lack research - this being the single most frequently asked question on SO... hang on - is this about MySQL?
    – Strawberry
    Mar 5, 2014 at 1:38

4 Answers 4

2

To improve performance, remove the unnecessary inline view, e.g.

SELECT tb1.*
  FROM articleTb tb1
  JOIN ( SELECT b.articleID
              , MAX(b.version) AS maxversion
           FROM articleTb b 
          WHERE b.Detail LIKE '%3'
          GROUP BY b.articleID
       ) tb2
    ON tb1.articleID = tb2.articleID 
   AND tb1.version = tb2.maxversion
 WHERE tb1.Detail LIKE '%3'

and...

make sure you have appropriate indexes. A covering index with a leading column of article may enable MySQL to use the index to optimize the GROUP BY (avoiding a "Using filesort" operation.)

... ON articleTb (articleID, version, detail)

MySQL may also be able to use that index for the join to tb1; the derived table (inline view) won't have an index.

You can confirm the execution plan with an EXPLAIN.

6
  • ur solution sounds interesting but what is inline view?
    – Kiti
    Mar 5, 2014 at 1:51
  • An "inline view" is a query referenced as a rowsource by an outer query. In the OP query, both tb1 and tb2 are inline views. You may be more familiar with the name MySQL uses for it; MySQL refers to an inline view as a "derived table". (That name makes sense we understand the operations that MySQL performs: materializing the resultset from the query as a temporary MyISAM table, and then running the outer query against the temporary "derived" MyISAM table. (This is quite different from what other RDBMS do with inline views.) Mar 5, 2014 at 2:00
  • Hi Spencer, what if to get the Max version we need to join with other table, then how your solution can deal in this case, pls look at the Extra info I updated in my question. In this case, should we use some sort of Session Varible???
    – Kiti
    Mar 5, 2014 at 2:20
  • This is doing exactly what my suggested CTE would do. You'd need to test to see which was more efficient, that really depends on the optimiser. I prefer a CTE to a subquery, just because I find it more readable, but if this solution is faster, you should use it.
    – cgraus
    Mar 5, 2014 at 2:31
  • ok, i can do CTE, but need to manage it & that could be headchade, maybe just calculate max for all articles
    – Tum
    Mar 5, 2014 at 2:37
1

I would use a CTE to create a table that contains the article id and the version id, then use that in my main query to filter down to the most recent version.

with latest as
(
  select articleId, max(version) as version from articleTb
)

select ....
from articleTb a
inner join latestl on a.articleid = l.articleid and l.version = a.version
4
  • what if we have a lot of article then you have to let the system calculate the max version for all articleID in CTE table?
    – Tum
    Mar 5, 2014 at 1:35
  • Yes, that's true, it's going to create a table which looks up ALL the articles. So, filtering based on other criteria MAY be faster, but if it involves a LIKE statement, I would not be surprised if it wasn't. I'd do some testing timing different statements with your data set to get some idea of what works best. The only other thing you could do is keep a table with just the latest document, or have a bit for 'latest version', to make it easier to search just the latest version.
    – cgraus
    Mar 5, 2014 at 1:38
  • there is no other solution? is it take a lot of time for mysql to calculate the max value?
    – Tum
    Mar 5, 2014 at 1:40
  • Someone else may offer one, but ultimately, you either design a database to optimise what you want to do ( look only in the latest version ), or you slow things down because you have to filter on the latest version. A CTE is the cleanest way to do that, in my opinion, and if that's not fast enough ( have you tried anything to see if speed is even an issue ? ), then you may need to refactor your database so it's better able to do what you want. Is the version number indexed ? That will help a lot, too.
    – cgraus
    Mar 5, 2014 at 1:42
1

Use of aggregate table will helpful. Let me describe a scenario first. Day 1, you get a flat file first time ever.

1. Load that in a staging table.
2. Find ArticleID, MAx (Version) for each Article ID, and store in the aggregate table.
3. Left outer join the stage table with the aggregate table joining on article ID. Pick the higher version. This will lead to your result.
4. Truncate the staging table.

Next day when a new feed arrives, the file will again be loaded into the truncated table, and left joined.

You can add a few audit fields in aggregate table such as date when that file arrived, maybe file name too. I had used this method in one of the projects in a insurance companies that resulted into several fold performance gain.

4
  • seem like i have to go through many steps, i may forget , also the artile is dynamically added into the DB minutely
    – Kiti
    Mar 5, 2014 at 1:55
  • This is just a pattern. You will need to improvise it to suit your needs. For example, you can think that the table where the records are being added is the staging table. Add an extra field, IsProcessed. Once you have done processed the newly arrived records, marked them as 1. This way you will be processing only on the increments. If the same article ID arrives again with a higher version, update the aggregate table, else insert. Mar 5, 2014 at 1:59
  • You can automate the process, this is giving specifics on what I said - if you want it to be more efficient, you need to improve your database.
    – cgraus
    Mar 5, 2014 at 2:30
  • "Once you have done processed the newly arrived records, marked them as 1." is an interesting idea
    – Kiti
    Mar 5, 2014 at 2:31
1

This is your query:

select *
from (Select * from articleTb where Detail like '%3'
     ) tb1 Join
     (select articleID, max(version) maxversion
      from articleTb
      where Detail like '%3'
      group by articleID
     ) tb2
     on tb1.articleID=tb2.articleID and tb1.version=tb2.maxversion;

You are trying to get the last version of a particular type of article. Another approach is to use not exists:

select *
from articleTb t
where Detail like '%3' and
      not exists (select 1
                  from articleTb t2
                  where t2.articleID = t1.articleID and
                        t2.Detail like '%3'
                        t2.version > t.version
                 );

This is saying: "Get me all the rows from articleTb where Detail ends in 3 and there isn't another version that is higher".

To improve performance, create an index on: articleTb(articleID, Detail, version). The one question is whether t2.Detail like '%3' is needed for the subquery -- does that condition filter articles or versions within an article? If it is not needed, then remove the index and change the condition to articleTb(articleID, version).

4
  • what if we get max by jOin with other table ex: "select b1.articleID, max(b1.version) maxversion from articleTb b1 Join otherTb b2 on b1.articleID=b2.id where b1.Detail like '%3' group by b1.articleID
    – Tum
    Mar 5, 2014 at 2:56
  • @Tum . . . That requires an aggregation. It is perfectly valid and it may meet your performance needs. This is an alternative that. If you don't need the detail condition in the subquery, it can be done by a simple index lookup which is likely to perform better. Mar 5, 2014 at 3:01
  • if u say like that then MYSQL has to calculate the JOIN 2 times, One is outer where & 1 is inside Not Exists??
    – Tum
    Mar 5, 2014 at 3:08
  • @Tum . . . There is only one relation in this question. The not exists is a type of join, but there is only one of them. Mar 5, 2014 at 3:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.