2

I'm building a Human Resources OLAP schema, and I'm having lots of trouble calculating headcount. It sounds simple enough, but it's actually quite tricky following OLAPs fact table design and handling distinct employees. Essentially I'm following the following model laid out by Ralph Kimball. I have an Employee table that represents the transactions performed on an employee, and then I have a Employment table that is the fact table.

In Ralph's example he only calculates the fact table on a per month basis (ie month_key), but in my table I can calculate either by month, quarter, year, etc. At the month level everything works fine because there are no duplicate entries for a single employee. But, move up the hierarchy to quarter or year and a single employee gets double counted or more. For example for 1 year if an employee is employed all 12 months if you sum up his records he'll be counted 12 times!

The problem is that you can't aggregate the numbers in the table because of these duplicate entries. I've tried several other methods, but none of them really work reliably. But I thought about it and figured I could create aggregate tables for each level in the Employment Date. One table that used for year, one table that used for quarters, etc. That way my loading procedures could decide how to count employees at each level, and make sure only one employee is rolled up. And my logical structure of the data remains intact. From a query perspective I can still think of my data in years, quarters, months, etc.

Is this an appropriate use of aggregate tables? I've never heard of anyone using them for this purpose. I'm not using it for performance boost, but I'm using it to normalize the data and make sure everything is loaded in a way that can be aggregated without concern about duplicates. My queries won't change will they? I still be able to do something like:

select [Work Location] on ROWS, [Measures].[headcount] on COLUMNS from [Employment] where [EmploymentDate].[2014]

And

select [Work Location] on ROWS, [Measures].[headcount] on COLUMNS from [Employment] where [EmploymentDate].[2014].[5]

And Mondrian will use the appropriate table to pull the data from without me having to specify it in the query.

1 Answer 1

4

Distinct count is a world of problems. Avoid it at all costs.

Problem 1: doing a "select count(distinct some_column) from some_table" is very slow;

Problem 2: distinct counts do not aggregate, which means you cannot take totals at some detail level and combine them to get the totals at another level. As such, you cannot "pick the nearest aggregation table", you need to pick "the exact aggregation table". As such, two things happen:

  • You need 1 aggregation table at every combination of levels of every hiearchy
  • Missing one or more agg tables means Mondrian needs to pick up data directly from the fact table.

In your case, that means that if you have a Company-Department-Team and a Dates-Year-Quarter-Month hierarchies you will need:

  • Agg table at Company/All and Dates/Year level
  • Agg table at Company/All and Dates/Quarter level
  • Agg table at Company/All and Dates/Month level
  • 3 more, now at the Department level
  • 3 more at the Team level

With more dimensions things only get worse, you may end up with hundreds of agg tables.

To make matters worse, there's no way to force Mondrian to pick agg table A or B. It will pick one (or not) in a semi-obscure way with little or no control by the user.

Plus, having to do distinct counts to build the aggregation tables means your aggregation script will be quite slow.

Some alternatives:

  1. Do not use aggregations for that measure. Face the fact that those distinct counts will need to come from the fact table anyway;
  2. Build snapshot fact tables, with unique counts for month to date, quarter to date and year to date as 3 separate columns. Have your "snapshot" date as a hierarchy with hasAll=false and skip all levels, so that you either pick a date or you see nothing (as you cannot aggregate across snapshots);
  3. Try to somehow avoid doing a distinct count in some way. One possible way is keeping track of a "number of days since this person was last seen by the ETL", which you can achieve with a lookup. Your distinct count of people in, lets say, a month is the sum of (people seen on day 1 of the month) + (people seen on day 2 with "days since last seen" >= 2) + (people seen on day 3 with "days since last seen" >= 3) + ... + (people seen on day 30 with "days since last seen" >= 30).

Method 1 has the clear advantage of being simpler, but the disadvantage of deferring all complexity to the DB itself; Method 2 has the advantage of showing you quickly all the values you need, but at the expense of further ETL work and it's limited to the measures you added, not allowing any flexibility; Method 3 is the most flexible, but at the cost of both a significant increase in ETL work and more complex queries.

Which one is the correct approach? Quite frankly, none of them. It's a very tough problem to tackle with a star schema and Mondrian.

5
  • In my case I always make the user filter by a date. In the employment fact table I have a single date (employmentDate). If I do that then I could get away with just 4 agg tables for each level (year, month, quarter, fiscal year). Am I thinking correctly about that? I think I'm ok with slow aggregation scripts or at least I can tolerate it. Commented Aug 8, 2014 at 15:09
  • If the user MUST select a day, how can he get a distinct count per month? Or are we talking about two different date dimensions?
    – nsousa
    Commented Aug 11, 2014 at 9:13
  • If the user selects a month (lowest level is a month) then you don't have a double count issues. But anything above a month (multiple months, quarter, year, multi-year) double counting will happen. But if I have aggregation tables I think I can fix (quarter, year) issues, but I can't fix multi-month or multi-year if it's not on the row or columns axis. Commented Aug 11, 2014 at 18:34
  • In such a scenario yes, you need a different agg table for each level.
    – nsousa
    Commented Aug 12, 2014 at 18:29
  • Thanks. Your answer was a great explanation of why this is so frustrating, but also gave me perspective on what is doable and what isn't. This is such a common problem for HR analytics and it's really painful seems like this would be something that should be more well known or discussed. Commented Aug 12, 2014 at 20:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.