0

Ok, so I've been stuck on this for 2 days! I've solved it from a semantic point of view but the query can take up to 10 minutes to execute. My database of choice for this is SQLite (for reasons I do not want to elaborate here), but I have tried running the same thing on a SQL Server 2012, it didn't make much of a difference in performance.

So, the problem is that I have 2 tables

  • prices (product_id INT, for_date DATE, value INT)
  • events (starts_on DATE, ends_on DATE NULLABLE)

I have approximately 500K rows in the prices table and around 100 rows in the events table.

Now I need to write a query to do the following.

Pseudo code is:

  • For each event:
    • IF the event has an ends_on value THEN fetch all product_id(s) that have a matching for_date, For products that DO NOT MATCH then fetch the last for_date which is less than the ends_on value but greater than starts_on for that event.
    • ELSE IF the ends_on date of the event is NULL, THEN fetch all product_id(s) that have a for_date that matches to starts_on, For products that DO NOT MATCH fetch the last for_date which is less than the starts_on value.

The query I have written in SQL Server 2012 is

SELECT    
    sp.for_date, sp.value
FROM 
    prices sp
INNER JOIN 
    events ev ON (((ev.ends_on IS NOT NULL AND
                   (sp.for_date = (SELECT for_date
                                   FROM prices
                                   WHERE for_date <= ev.ends_on 
                                      AND for_date > ev.starts_on
                                   ORDER BY for_date DESC
                                      OFFSET 0 ROWS
                                      FETCH NEXT 1 ROWS ONLY))))
          OR
          ((ev.ends_on is null 
          and
             (sp.for_date = (SELECT for_date
                                FROM prices
                                WHERE 
                                    for_date <= ev.starts_on_j
                                    AND for_date > dateadd(day, -14,   ev.starts_on)
                                order by for_date desc
                                offset 0 rows
                                    fetch next 1 row only))))

                                    );

Btw I have also tried to create temp tables with partial data and done the same op on them. It just gets stuck.

The strange thing is if I run the 2 "OR" conditions separately, the response time is perfect !

Update

Sample Dataset and Expected Result

Price Entries

Product ID, ForDt, Value
1, 25-01-2010, 123
1, 26-01-2010, 112
1, 29-01-2010, 334
1, 02-02-2010, 512
1, 03-02-2010, 765
1, 04-02-2010, 632
1, 05-02-2010, 311
1, 06-02-2010, 555
2, 03-02-2010, 854
2, 04-02-2010, 625
2, 05-02-2010, 919
3, 20-01-2010, 777
3, 06-02-2010, 877
3, 10-03-2010, 444
3, 11-03-2010, 888

Event Entries (To make it more understandable, Im adding an event id also)

Event ID, StartsOn, EndsOn
22, 27-01-2010, NULL
33, 02-02-2010, 06-02-2010
44, 01-03-2010, 13-03-2010

Expected Result Set

Event ID, Product ID, ForDt, Value
22, 1, 26-01-2010, 112
33, 1, 06-02-2010, 311
44, 1, 06-02-2010, 311

33, 2, 05-02-2010, 919
44, 2, 05-02-2010, 919

22, 3, 20-01-2010, 777
33, 3, 06-02-2010, 877
44, 3, 11-03-2010, 888
6
  • If you can show the query and output,it will be helpfull to understand what query is doing Aug 10, 2016 at 16:32
  • also show the indexes and how are those above two tables related with some sampe data Aug 10, 2016 at 16:32
  • 1
    And the output of EXPLAIN QUERY PLAN.
    – CL.
    Aug 10, 2016 at 16:32
  • The answer you want should be for SQLite or SQL Server?
    – Lamak
    Aug 10, 2016 at 16:39
  • I must admit I have difficulties understanding the task. You are selecting records from table prices, but only show the date and price. As you combine the records with many events, you finally get a list of many days and many prices, many of them probably duplicates, and without the information what product and event they refer to. That doesn't seem to make sense. Aug 10, 2016 at 18:28

1 Answer 1

1

Okay, now that you have shown the expected results being a list of events and associated products the question makes sense. Your query only selecting dates and values didn't.

You are looking for the best product price record per event. This would be easily done with analytic functions, but SQLite doesn't support them. So we must write a more complicated query.

Let's look at events with ends_on null first. Here is how to find the best product prices (i.e. last before starts_on):

select e.event_id, p.product_id, max(for_date) as best_for_date
from events e
join prices p on p.for_date < e.starts_on
where e.ends_on is null
group by e.event_id, p.product_id;

We extend this query to also find the best product prices for events with an ends_on and then access the products table again so we get the full records with the values:

select ep.event_id, p.product_id, p.for_date, p.value
from
(
  select e.event_id, p.product_id, max(for_date) as best_for_date
  from events e
  join prices p on (e.ends_on is null and p.for_date < e.starts_on)
                or (e.ends_on is not null and p.for_date between e.starts_on and e.ends_on)
  group by e.event_id, p.product_id
) ep
join prices p on p.product_id = ep.product_id and p.for_date = ep.best_for_date;

(By the way: You are describing a very special case here. The databases I have seen so far would treat an ends_on null as unlimited or "still active". Thus the price to retrieve for such an event would not be the last before starts_on, but the most current one at or after starts_on.)

2
  • Ok so I think this query principally works, will try it out on different combinations of my main production dataset. For now I've substituted p.for_date < e.starts_on with p.for_date <= e.starts_on Will mark the answer correct once tested. Thanks :)
    – Storm
    Aug 13, 2016 at 13:59
  • Thanks Thorsten. It was really dumb of me to not use the max in the select clause :o)
    – Storm
    Aug 15, 2016 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.