9

According to another SO post (SQL: How to keep rows order with DISTINCT?), distinct has pretty undefined behavior as far as sorting.

I have a query:

select col_1 from table order by col_2

This can return values like

3
5
3
2

I need to then select a distinct on these that preserves ordering, meaning I want

select distinct(col_1) from table order by col_2 

to return

3
5
2

but not

5
3
2

Here is what I am actually trying to do. Col_1 is a user id, and col_2 is a log in timestamp event by that user. So the same user (col_1) can have many login times. I am trying to build a historical list of users in which they were seen in the system. I would like to be able to say "our first user ever was, our second user ever was", and so on.

That post seems to suggest to use a group by, but group by is not meant to return an ordering of rows, so I do not see how or why this would be applicable here, since it does not appear group by will preserve any ordering. In fact, another SO post gives an example where group by will destroy the ordering I am looking for: see "Peter" in what is the difference between GROUP BY and ORDER BY in sql. Is there anyway to guarantee the latter result? The strange thing is, if I were implementing the DISTINCT clause, I would surely do the order by first, then take the results and do a linear scan of the list and preserve the ordering naturally, so I am not sure why the behavior is so undefined.

EDIT:

Thank you all! I have accepted IMSoP answer because not only was there an interative example that I could play around with (thanks for turning me on to SQL Fiddle), but they also explained why several things worked the way they worked, instead of simply "do this". Specifically, it was unclear that GROUP BY does not destroy (rather, keeps them in some sort of internal list) values in the other columns outside of the group by, and these values can still be examined in an ORDER BY clause.

23
  • I believe you will need to do that type of data processing upon retrieval of the data. That is functionality that I have never heard of being implemented.
    – Joe
    Oct 16, 2013 at 21:46
  • 3
    If you don't provide an ORDER BY clause there is never a guarantee on the order of the data. If you want it in a specific order you must use order by.
    – Taryn
    Oct 16, 2013 at 21:50
  • 1
    Well I'm using mysql 5.6 but I was thinking that DISTINCT is probably defined somewhere in the SQL standard.
    – Tommy
    Oct 16, 2013 at 22:00
  • 1
    There is no undefined behaviour. It's pretty much defined: Queries without ORDER BY can return their result sets in any order. If you want a specific order, use ORDER BY. If you don't have ORDER BY, you have a table as a result. And order does not matter for tables. Oct 16, 2013 at 22:30
  • 1
    @Tommy: there is no first, unless it was written down in the record (for example, by means of a timestamp or a sequence number). Without that, all red balls are red, and none of them can be picked as the first. Oct 17, 2013 at 0:08

4 Answers 4

13

This all has to do with the "logical ordering" of SQL statements. Although a DBMS might actually retrieve the data according to all sorts of clever strategies, it has to behave according to some predictable logic. As such, the different parts of an SQL query can be considered to be processed "before" or "after" one another in terms of how that logic behaves.

As it happens, the ORDER BY clause is the very last step in that logical sequence, so it can't change the behaviour of "earlier" steps.

If you use a GROUP BY, the rows have been bundled up into their groups by the time the SELECT clause is run, let alone the ORDER BY, so you can only look at columns which have been grouped by, or "aggregate" values calculated across all the values in a group. (MySQL implements a controversial extension to GROUP BY where you can mention a column in the SELECT that can't logically be there, and it will pick one from an arbitrary row in that group).

If you use a DISTINCT, it is logically processed after the SELECT, but the ORDER BY still comes afterwards. So only once the DISTINCT has thrown away the duplicates will the remaining results be put into a particular order - but the rows that have been thrown away can't be used to determine that order.


As for how to get the result you need, the key is to find a value to sort by which is valid after the GROUP BY/DISTINCT has (logically) been run. Remember that if you use a GROUP BY, any aggregated values are still valid - an aggregate function can look at all the values in a group. This includes MIN() and MAX(), which are ideal for ordering by, because "the lowest number" (MIN) is the same thing as "the first number if I sort them in ascending order", and vice versa for MAX.

So to order a set of distinct foo_number values based on the lowest applicable bar_number for each, you could use this:

SELECT foo_number
FROM some_table
GROUP BY foo_number
ORDER BY MIN(bar_number) ASC

Here's a live demo with some arbitrary data.


EDIT: In the comments, it was discussed why, if an ordering is applied before the grouping / de-duplication takes place, that order is not applied to the groups. If that were the case, you would still need a strategy for which row was kept in each group: the first, or the last.

As an analogy, picture the original set of rows as a set of playing cards picked from a deck, and then sorted by their face value, low to high. Now go through the sorted deck and deal them into a separate pile for each suit. Which card should "represent" each pile?

If you deal the cards face up, the cards showing at the end will be the ones with the highest face value (a "keep last" strategy); if you deal them face down and then flip each pile, you will reveal the lowest face value (a "keep first" strategy). Both are obeying the original order of the cards, and the instruction to "deal the cards based on suit" doesn't automatically tell the dealer (who represents the DBMS) which strategy was intended.

If the final piles of cards are the groups from a GROUP BY, then MIN() and MAX() represent picking up each pile and looking for the lowest or highest value, regardless of the order they are in. But because you can look inside the groups, you can do other things too, like adding up the total value of each pile (SUM) or how many cards there are (COUNT) etc, making GROUP BY much more powerful than an "ordered DISTINCT" could be.

15
  • @Tommy The example in your column has no ORDER BY in the outer query, so the DBMS can return the results in whatever order it likes - basically, whichever comes to hand first. Similarly, if you specify anything in the ORDER BY clause which could not exist in the SELECT clause, the DBMS is free to either guess, or tell you there is a logical error in your query.
    – IMSoP
    Oct 16, 2013 at 22:31
  • 1
    As for ordering of operations being ridiculous, I took great pains to repeat that this is a logical ordering: however the DBMS actually implements the query, it must behave as though this order of events took place. If it was free to interpret your SQL in a completely different logical order, you wouldn't be able to write a query and know how it would behave.
    – IMSoP
    Oct 16, 2013 at 22:33
  • Actually, re-reading, I think you meant that it's ridiculous that you can't, as a user, change that logical order manually. But that wouldn't help you, because as you say, a DISTINCT that ran after an ORDER BY would still need to know which of each set of duplicates to keep - the first or the last? Once you had syntax to choose that, you might as well use ORDER BY MIN as in the working example provided.
    – IMSoP
    Oct 16, 2013 at 22:42
  • 2
    @Tommy there are various ways to do this in SQL. Some DBMS have more ways to do it, some less. Postgres has something that is more close to what you describe, it's called DISTINCT ON Oct 16, 2013 at 22:53
  • 1
    @wildplasser I suspect you and ypercube are just using different values of "this" (insert JS joke here...)
    – IMSoP
    Oct 16, 2013 at 23:58
2

I would go for something like

select col1
from (
select col1,
       rank () over(order by col2) pos
from table
)
group by col1
order by min(pos)

In the subquery I calculate the position, then in the main query I do a group by on col1, using the smallest position to order.

Here the demo in SQLFiddle (this was Oracle, the MySql info was added later.

Edit for MySql:

select col1
from (
select col1 col1,
       @curRank := @curRank + 1 AS pos
from table1, (select @curRank := 0) p
) sub
group by col1
order by min(pos)

And here the demo for MySql.

4
  • You don't need variables for this query. Oct 16, 2013 at 22:25
  • This is massively overkill for what the OP needs, which is just a better understanding of GROUP BY and aggregates. If you're sorting by an ungrouped column, you generally want min() or max() of that column.
    – IMSoP
    Oct 16, 2013 at 22:26
  • I know guys, but the solution was so trivial that I thought Tommy already tried it :) Anyway in case it's not possible to use min/max, here is another solution :) Anyway there is another way to have rank with MySql? (in case you need it?)
    – mucio
    Oct 16, 2013 at 22:31
  • 1
    @mucio Welcome to StackOverflow; it's best not to assume anything about the people asking questions (positive or negative). If the solution seems too obvious, ask why it didn't work out before jumping in with something more advanced which may not be applicable to their situation.
    – IMSoP
    Oct 16, 2013 at 22:36
1

The GROUP BY in the referenced answer isn't attempting to perform an ordering... it is simply picking a single associated value for the column that we want to be distinct.

Like @bluefeet states, if you want a guaranteed ordering, you must use ORDER BY.

Why can't we specify a value in the ORDER BY that isn't included in the SELECT DISTINCT?

Consider the following values for col1 and col2:

create table yourTable (
  col_1 int,
  col_2 int
);

insert into yourTable (col_1, col_2) values (1, 1);
insert into yourTable (col_1, col_2) values (1, 3);
insert into yourTable (col_1, col_2) values (2, 2);
insert into yourTable (col_1, col_2) values (2, 4);

With this data, what should SELECT DISTINCT col_1 FROM yourTable ORDER BY col_2 return?

That's why you need the GROUP BY and the aggregate function, to decide which of the multiple values for col_2 you should order by... could be MIN(), could be MAX(), maybe even some other function such as AVG() would make sense in some cases; it all depends on the specific scenario, which is why you need to be explicit:

select col_1
from yourTable
group by col_1
order by min(col_2)

SQL Fiddle Here

6
  • if we do the ordering first, we get (1,1) (2,2),(1,3),(2,4). It should then return (1,1) and (2,2), and scrap (1,3) and (2,4) which is exactly what I want. Where is the ambiguity?>
    – Tommy
    Oct 16, 2013 at 22:36
  • 2
    You're assuming that the distinct should only return the first value it encounters... why not the last / most recent value it encounters? That seems just as valid, and could give completely different results. Oct 16, 2013 at 22:39
  • that is exactly my point! As to whether its invalid, if you run DISTINCT On (3,5,3), the result (3,5) can be argued to be more "naturally temporally" than (5,3). The first one just seems the most logical to me given that I have already specified (3,5,3) is an ordered list ordered by some other clause.
    – Tommy
    Oct 16, 2013 at 22:44
  • 2
    Ah, so the argument is that the distinct should infer a MIN() aggregation since the order by was ASC, and temporally MIN() and ASC go together. That's too subjective and specific to your intentions in this specific scenario to be applied universally. I can just as easily imagine a scenario where I want a MAX() aggregation applied to my distinct, but then I want to order the results ascending instead of descending. It's a lot simpler if SQL just forces the desired behavior to be explicitly stated instead of attempting to infer the intention based on other parts of the query. Oct 16, 2013 at 22:55
  • @MichaelFredrickson Incidentally, that's why I find Postgres's DISTINCT ON really hard to follow: the behaviour is well-defined, but you have to jump between the SELECT and ORDER BY clauses to see what data will actually be returned.
    – IMSoP
    Oct 16, 2013 at 23:20
1

For MySQL only, when you select columns that are not in the GROUP BY it will return columns from the first record in the group. You can use this behavior to select which record is returned from each group like this:

SELECT foo_number, bar_number
FROM 
(
  SELECT foo_number, bar_number 
  FROM some_table 
  ORDER BY bar_number
) AS t
GROUP BY foo_number
ORDER BY bar_number DESC;

This is more flexible because it allows you to order the records within each group using expressions that are not possible with aggregates - in my case I wanted to return the one with the shortest string in another column.

For completeness, my query looks like this:

SELECT
  s.NamespaceId,
  s.Symbol,
  s.EntityName
FROM 
(
  SELECT 
    m.NamespaceId,
    i.Symbol, 
    i.EntityName
  FROM ImportedSymbols i
  JOIN ExchangeMappings m ON i.ExchangeMappingId = m.ExchangeMappingId
  WHERE
    i.Symbol NOT IN 
    (
      SELECT Symbol 
      FROM tmp_EntityNames
      WHERE NamespaceId = m.NamespaceId
    )
      AND
    i.EntityName IS NOT NULL
  ORDER BY LENGTH(i.RawSymbol), i.RawSymbol
) AS s
GROUP BY s.NamespaceId, s.Symbol;

What this does is return a distinct list of symbols in each namespace, and for duplicated symbols returns the one with the shortest RawSymbol. When the RawSymbol lengths are the same, it returns the one who's RawSymbol comes first alphabetically.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.