0

I'm new to Postgres, but not to relational databases, and wrote some queries with some odd results tonight. Can someone help explain what is going on with the counts here?

First query, gives 400,000 (rows) as result:

select * from mytable;

Second query, groups on the two compound primary key columns (results are good — returns 400,000 records):

select PK1, PK2, count(*) cnt
  from mytable
 group by PK1, PK2

Third query, gives me over 1.4 million records erroneously:

select PK1, PK2
  from mytable
 group by PK1, PK2

The only difference between the second and third queries is the removal of the count(*) field. The output doesn't look like distinct records - rows are duplicated.

Anybody know why removing this 'count' field, in a Postgres query, would skew the output in this way?

11
  • 1
    This doesn't make sense. It is highly suspicious. I suspect something else is going on. Jun 3, 2016 at 2:51
  • So have you checked what are those extra rows?
    – zerkms
    Jun 3, 2016 at 2:55
  • I can't reproduce this (with a very minimal table). Could you provide a minimal example that show this problem, by providing a (small) dataset? In fact, can you reproduce this problem with a much smaller dataset?
    – user707650
    Jun 3, 2016 at 3:05
  • 1
    Please list the Postgres version and the platform you're using.
    – user707650
    Jun 3, 2016 at 3:06
  • 1
    Could you explain how have you counted rows from these queries ? I suppose it hasn't been done by hand (1,4 mln). Export to a file (excel or something else)?
    – krokodilko
    Jun 3, 2016 at 4:16

2 Answers 2

1

That doesn't look right to me. PostgreSQL groups at the same time it does the aggregate. So something else must be going on. Here are some things to rule out:

  1. Double check the number of rows accurately. Do this by putting your main query in a cte and selecting count(*) from it. An example is below.

  2. Check query plans. Is there a join projection issue going on that you are not taking into account? explain is your friend here.

For the example:

   with query (
   select PK1, PK2, count(*) cnt
     from mytable
    group by PK1, PK2
   )
   select count(*) from query;
1

These are some actions you can take to troubleshoot your enigma:

First ensure you have an accurate number of rows in your table:

select count(*) from mytable

Then check how many distinct PK1 and PK2 values you have:

select count(*) from (select distinct PK1 from mytable) a

and

select count(*) from (select distinct PK2 from mytable) a

Both values should be either equal (you tested for a unique key), or significantly smaller (none of them are a unique key), than the total count.

Then test counting for your keys combined:

select count(*) from (select distinct PK1, PK2 from mytable) a

The resulting value should be either equal ((PK1, PK2) are a unique key), or significantly smaller ((PK1, PK2) is not a unique key), than the total count.

The following query should give you the exact same number than the previous one:

select count(*) from (
  select PK1, PK2
  from mytable
  group by PK1, PK2) a

and again the same value from the following

select count(*) from (
  select PK1, PK2, count(*)
  from mytable
  group by PK1, PK2) a

Finally, the following should give you the exact same number as the total count:

select sum(cnt) from (
  select PK1, PK2, count(*) as cnt
  from mytable
  group by PK1, PK2) a

If at any point, any of these tests do not give you the expected results, then your data is seriously corrupted.

2
  • I have run these queries as you suggested and still get the same erroneous results. I thought wrapping a "select count(*) " around my group by statement would clear things up, but I still get the larger erroneous count. Hmm. Jun 5, 2016 at 23:54
  • @user1399233 at this point, the only way you'll get any support for your mysterious case will be by updating your question with the DDL of your table, and the exact results you get (including explain analyze would be even better) for each one of these queries. Jun 6, 2016 at 3:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.