0

In my MySQL DB (v5.7.31-34, using InnoDB), I'm trying to streamline the performance of the following query:

SELECT DISTINCT SQL_CALC_FOUND_ROWS `story_allocation_days`.* 
FROM `story_allocation_days` 
INNER JOIN `workspaces` 
ON `workspaces`.`id` = `story_allocation_days`.`workspace_id` 
INNER JOIN `participations` 
ON `participations`.`workspace_id` = `workspaces`.`id` 
INNER JOIN `assignments` 
ON `assignments`.`id` = `story_allocation_days`.`assignment_id` 
WHERE (story_allocation_days.deleted_at is null) 
AND `workspaces`.`is_budgeted` = TRUE 
AND (participations.account_id = 5071) 
AND (participations.access_integer >= 30) 
AND (participations.type = 'MavenParticipation' OR workspaces.account_id = 5071) 
AND (assignments.assignee_id IS NOT NULL);
99822 rows in set (5.35 sec)

The EXPLAIN on the above:

| id | select_type | table                 | partitions | type   | possible_keys                                                                                                                                                                                                       | key                                    | key_len | ref                                                   | rows   | filtered | Extra                              |
|  1 | SIMPLE      | story_allocation_days | NULL       | ALL    | index_story_allocation_days_on_workspace_id,index_story_allocation_days_on_assignment_id_and_date,index_story_allocation_days_on_assignment_id                                                                      | NULL                                   | NULL    | NULL                                                  | 430531 |    10.00 | Using where; Using temporary       |
|  1 | SIMPLE      | workspaces            | NULL       | eq_ref | PRIMARY,index_workspaces_on_account_id                                                                                                                                                                              | PRIMARY                                | 4       | bm_rpm.story_allocation_days.workspace_id       |      1 |    10.00 | Using where; Distinct              |
|  1 | SIMPLE      | assignments           | NULL       | eq_ref | PRIMARY,index_assignments_on_assignee_id                                                                                                                                                                            | PRIMARY                                | 4       | bm_rpm.story_allocation_days.assignment_id      |      1 |    50.00 | Using where; Distinct              |
|  1 | SIMPLE      | participations        | NULL       | ref    | index_participations_on_account_id_and_workspace_id,index_participations_on_workspace_id_and_user_id,index_participations_for_sad_api_index,index_participations_on_account_id,index_participations_on_workspace_id | index_participations_for_sad_api_index | 10      | bm_rpm.story_allocation_days.workspace_id,const |      7 |    33.33 | Using where; Using index; Distinct |

Here's the FORMAT=JSON version of the above:

| {
  "query_block": {
    "select_id": 1,
    "cost_info": {
      "query_cost": "153230.79"
    },
    "duplicates_removal": {
      "using_temporary_table": true,
      "using_filesort": false,
      "nested_loop": [
        {
          "table": {
            "table_name": "story_allocation_days",
            "access_type": "ALL",
            "possible_keys": [
              "index_story_allocation_days_on_workspace_id",
              "index_story_allocation_days_on_assignment_id_and_date",
              "index_story_allocation_days_on_assignment_id"
            ],
            "rows_examined_per_scan": 430268,
            "rows_produced_per_join": 43026,
            "filtered": "10.00",
            "cost_info": {
              "read_cost": "80548.24",
              "eval_cost": "8605.36",
              "prefix_cost": "89153.60",
              "data_read_per_join": "3M"
            },
            "used_columns": [
              "id",
              "assignment_id",
              "story_id",
              "workspace_id",
              "account_id",
              "current",
              "date",
              "minutes",
              "created_at",
              "updated_at",
              "deleted_at",
              "cost_amount_in_cents",
              "bill_amount_in_cents",
              "cost_rate_in_cents",
              "bill_rate_in_cents"
            ],
            "attached_condition": "isnull(`bm_rpm`.`story_allocation_days`.`deleted_at`)"
          }
        },
        {
          "table": {
            "table_name": "workspaces",
            "access_type": "eq_ref",
            "possible_keys": [
              "PRIMARY",
              "index_workspaces_on_account_id"
            ],
            "key": "PRIMARY",
            "used_key_parts": [
              "id"
            ],
            "key_length": "4",
            "ref": [
              "bm_rpm.story_allocation_days.workspace_id"
            ],
            "rows_examined_per_scan": 1,
            "rows_produced_per_join": 4302,
            "filtered": "10.00",
            "distinct": true,
            "cost_info": {
              "read_cost": "43026.80",
              "eval_cost": "860.54",
              "prefix_cost": "140785.76",
              "data_read_per_join": "21M"
            },
            "used_columns": [
              "id",
              "is_budgeted",
              "account_id"
            ],
            "attached_condition": "(`bm_rpm`.`workspaces`.`is_budgeted` = TRUE)"
          }
        },
        {
          "table": {
            "table_name": "assignments",
            "access_type": "eq_ref",
            "possible_keys": [
              "PRIMARY",
              "index_assignments_on_assignee_id"
            ],
            "key": "PRIMARY",
            "used_key_parts": [
              "id"
            ],
            "key_length": "4",
            "ref": [
              "bm_rpm.story_allocation_days.assignment_id"
            ],
            "rows_examined_per_scan": 1,
            "rows_produced_per_join": 2151,
            "filtered": "50.00",
            "distinct": true,
            "cost_info": {
              "read_cost": "4302.68",
              "eval_cost": "430.27",
              "prefix_cost": "145948.98",
              "data_read_per_join": "134K"
            },
            "used_columns": [
              "id",
              "assignee_id"
            ],
            "attached_condition": "(`bm_rpm`.`assignments`.`assignee_id` is not null)"
          }
        },
        {
          "table": {
            "table_name": "participations",
            "access_type": "ref",
            "possible_keys": [
              "index_participations_on_account_id_and_workspace_id",
              "index_participations_on_workspace_id_and_user_id",
              "index_participations_for_sad_api_index",
              "index_participations_on_account_id",
              "index_participations_on_workspace_id"
            ],
            "key": "index_participations_for_sad_api_index",
            "used_key_parts": [
              "workspace_id",
              "account_id"
            ],
            "key_length": "10",
            "ref": [
              "bm_rpm.story_allocation_days.workspace_id",
              "const"
            ],
            "rows_examined_per_scan": 7,
            "rows_produced_per_join": 5537,
            "filtered": "33.33",
            "using_index": true,
            "distinct": true,
            "cost_info": {
              "read_cost": "3959.11",
              "eval_cost": "1107.46",
              "prefix_cost": "153230.79",
              "data_read_per_join": "11M"
            },
            "used_columns": [
              "id",
              "workspace_id",
              "type",
              "account_id",
              "access_integer"
            ],
            "attached_condition": "((`bm_rpm`.`participations`.`access_integer` >= 30) and ((`bm_rpm`.`participations`.`type` = 'MavenParticipation') or (`bm_rpm`.`workspaces`.`account_id` = 5071)))"
          }
        }
      ]
    }
  }
} |

I tried finding an index which would change this from a full table scan to a more efficient type of search. Here are the indexes:

mysql> show indexes in story_allocation_days;
| Table                 | Non_unique | Key_name                                              | Seq_in_index | Column_name   | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
| story_allocation_days |          0 | PRIMARY                                               |            1 | id            | A         |      418853 |     NULL | NULL   |      | BTREE      |         |               |
| story_allocation_days |          1 | index_story_allocation_days_on_workspace_id           |            1 | workspace_id  | A         |         295 |     NULL | NULL   |      | BTREE      |         |               |
| story_allocation_days |          1 | index_story_allocation_days_on_assignment_id_and_date |            1 | assignment_id | A         |       42533 |     NULL | NULL   |      | BTREE      |         |               |
| story_allocation_days |          1 | index_story_allocation_days_on_assignment_id_and_date |            2 | date          | A         |      418853 |     NULL | NULL   |      | BTREE      |         |               |
| story_allocation_days |          1 | index_story_allocation_days_on_account_id             |            1 | account_id    | A         |          41 |     NULL | NULL   |      | BTREE      |         |               |
| story_allocation_days |          1 | index_story_allocation_days_on_story_id_and_date      |            1 | story_id      | A         |        2519 |     NULL | NULL   |      | BTREE      |         |               |
| story_allocation_days |          1 | index_story_allocation_days_on_story_id_and_date      |            2 | date          | A         |       83246 |     NULL | NULL   |      | BTREE      |         |               |
| story_allocation_days |          1 | index_story_allocation_days_on_date                   |            1 | date          | A         |         740 |     NULL | NULL   |      | BTREE      |         |               |
| story_allocation_days |          1 | index_story_allocation_days_on_created_at             |            1 | created_at    | A         |        6977 |     NULL | NULL   |      | BTREE      |         |               |
| story_allocation_days |          1 | index_story_allocation_days_on_updated_at             |            1 | updated_at    | A         |        7681 |     NULL | NULL   |      | BTREE      |         |               |
| story_allocation_days |          1 | index_story_allocation_days_on_assignment_id          |            1 | assignment_id | A         |       43891 |     NULL | NULL   |      | BTREE      |         |               |
| story_allocation_days |          1 | index_story_allocation_days_on_story_id               |            1 | story_id      | A         |        2494 |     NULL | NULL   |      | BTREE      |         |               |
| story_allocation_days |          1 | sad_deleted_at                                        |            1 | deleted_at    | A         |         225 |     NULL | NULL   | YES  | BTREE      |         |               |

I tried USE INDEX(index_story_allocation_days_on_assignment_id) on story_allocation_days, however this is actually marginally slower than the original query:

99822 rows in set (5.99 sec)

On a hunch, I thought maybe WHERE (story_allocation_days.deleted_at is null) was tripping up the optimizer, so I tried creating the sad_deleted_at index and updating all rows with NULL for their deleted_at column to 1970-01-01 00:00:00, on the chance that the index performance was suffering due to the presence of NULL values. However, this resulted in performance which was also worse than the original:

SELECT DISTINCT SQL_CALC_FOUND_ROWS `story_allocation_days`.* 
FROM `story_allocation_days` 
INNER JOIN `workspaces` 
ON `workspaces`.`id` = `story_allocation_days`.`workspace_id` 
INNER JOIN `participations` 
ON `participations`.`workspace_id` = `workspaces`.`id` 
INNER JOIN `assignments` 
ON `assignments`.`id` = `story_allocation_days`.`assignment_id` 
WHERE (story_allocation_days.deleted_at = '1970-01-01 00:00:00') 
AND `workspaces`.`is_budgeted` = TRUE 
AND (participations.account_id = 5071) 
AND (participations.access_integer >= 30) 
AND (participations.type = 'MavenParticipation' OR workspaces.account_id = 5071) 
AND (assignments.assignee_id IS NOT NULL);
99822 rows in set (6.12 sec)

Here's the corresponding EXPLAIN statement:

|  1 | SIMPLE      | story_allocation_days | NULL       | ref    | index_story_allocation_days_on_workspace_id,index_story_allocation_days_on_assignment_id_and_date,index_story_allocation_days_on_assignment_id,sad_deleted_at                                                       | sad_deleted_at                         | 6       | const                                                 | 210391 |   100.00 | Using temporary                    |
|  1 | SIMPLE      | workspaces            | NULL       | eq_ref | PRIMARY,index_workspaces_on_account_id                                                                                                                                                                              | PRIMARY                                | 4       | bm_rpm.story_allocation_days.workspace_id       |      1 |    10.00 | Using where; Distinct              |
|  1 | SIMPLE      | assignments           | NULL       | eq_ref | PRIMARY,index_assignments_on_assignee_id                                                                                                                                                                            | PRIMARY                                | 4       | bm_rpm.story_allocation_days.assignment_id      |      1 |    50.00 | Using where; Distinct              |
|  1 | SIMPLE      | participations        | NULL       | ref    | index_participations_on_account_id_and_workspace_id,index_participations_on_workspace_id_and_user_id,index_participations_for_sad_api_index,index_participations_on_account_id,index_participations_on_workspace_id | index_participations_for_sad_api_index | 10      | bm_rpm.story_allocation_days.workspace_id,const |      7 |    33.33 | Using where; Using index; Distinct |

And again, the FORMAT=JSON version of the above:

# NEW:
| {
  "query_block": {
    "select_id": 1,
    "cost_info": {
      "query_cost": "366895.66"
    },
    "duplicates_removal": {
      "using_temporary_table": true,
      "using_filesort": false,
      "nested_loop": [
        {
          "table": {
            "table_name": "story_allocation_days",
            "access_type": "ref",
            "possible_keys": [
              "index_story_allocation_days_on_workspace_id",
              "index_story_allocation_days_on_assignment_id_and_date",
              "index_story_allocation_days_on_assignment_id",
              "sad_deleted_at"
            ],
            "key": "sad_deleted_at",
            "used_key_parts": [
              "deleted_at"
            ],
            "key_length": "6",
            "ref": [
              "const"
            ],
            "rows_examined_per_scan": 210552,
            "rows_produced_per_join": 210552,
            "filtered": "100.00",
            "cost_info": {
              "read_cost": "11223.00",
              "eval_cost": "42110.40",
              "prefix_cost": "53333.40",
              "data_read_per_join": "16M"
            },
            "used_columns": [
              "id",
              "assignment_id",
              "story_id",
              "workspace_id",
              "account_id",
              "current",
              "date",
              "minutes",
              "created_at",
              "updated_at",
              "deleted_at",
              "cost_amount_in_cents",
              "bill_amount_in_cents",
              "cost_rate_in_cents",
              "bill_rate_in_cents"
            ]
          }
        },
        {
          "table": {
            "table_name": "workspaces",
            "access_type": "eq_ref",
            "possible_keys": [
              "PRIMARY",
              "index_workspaces_on_account_id"
            ],
            "key": "PRIMARY",
            "used_key_parts": [
              "id"
            ],
            "key_length": "4",
            "ref": [
              "bm_rpm.story_allocation_days.workspace_id"
            ],
            "rows_examined_per_scan": 1,
            "rows_produced_per_join": 21055,
            "filtered": "10.00",
            "distinct": true,
            "cost_info": {
              "read_cost": "210552.00",
              "eval_cost": "4211.04",
              "prefix_cost": "305995.80",
              "data_read_per_join": "105M"
            },
            "used_columns": [
              "id",
              "is_budgeted",
              "account_id"
            ],
            "attached_condition": "(`bm_rpm`.`workspaces`.`is_budgeted` = TRUE)"
          }
        },
        {
          "table": {
            "table_name": "assignments",
            "access_type": "eq_ref",
            "possible_keys": [
              "PRIMARY",
              "index_assignments_on_assignee_id"
            ],
            "key": "PRIMARY",
            "used_key_parts": [
              "id"
            ],
            "key_length": "4",
            "ref": [
              "bm_rpm.story_allocation_days.assignment_id"
            ],
            "rows_examined_per_scan": 1,
            "rows_produced_per_join": 10527,
            "filtered": "50.00",
            "distinct": true,
            "cost_info": {
              "read_cost": "21055.20",
              "eval_cost": "2105.52",
              "prefix_cost": "331262.04",
              "data_read_per_join": "657K"
            },
            "used_columns": [
              "id",
              "assignee_id"
            ],
            "attached_condition": "(`bm_rpm`.`assignments`.`assignee_id` is not null)"
          }
        },
        {
          "table": {
            "table_name": "participations",
            "access_type": "ref",
            "possible_keys": [
              "index_participations_on_account_id_and_workspace_id",
              "index_participations_on_workspace_id_and_user_id",
              "index_participations_for_sad_api_index",
              "index_participations_on_account_id",
              "index_participations_on_workspace_id"
            ],
            "key": "index_participations_for_sad_api_index",
            "used_key_parts": [
              "workspace_id",
              "account_id"
            ],
            "key_length": "10",
            "ref": [
              "bm_rpm.story_allocation_days.workspace_id",
              "const"
            ],
            "rows_examined_per_scan": 7,
            "rows_produced_per_join": 27096,
            "filtered": "33.33",
            "using_index": true,
            "distinct": true,
            "cost_info": {
              "read_cost": "19373.95",
              "eval_cost": "5419.35",
              "prefix_cost": "366895.66",
              "data_read_per_join": "55M"
            },
            "used_columns": [
              "id",
              "workspace_id",
              "type",
              "account_id",
              "access_integer"
            ],
            "attached_condition": "((`bm_rpm`.`participations`.`access_integer` >= 30) and ((`bm_rpm`.`participations`.`type` = 'MavenParticipation') or (`bm_rpm`.`workspaces`.`account_id` = 5071)))"
          }
        }
      ]
    }
  }
} |

I'm confused why the query time doesn't improve (and in fact gets marginally worse), even though the filter column for story_allocation_days goes from 10.0 to 100.0 and the # of rows examined goes from 400,000+ to 200,000-ish.

Can anyone see why the index isn't improving the query time?

EDIT:

I tried removing SQL_CALC_FOUND_ROWS from the original query, but it still took 5.51 seconds for 99k rows. EXPLAIN below, in JSON format (since I'm running out of characters for this question):

| {
  "query_block": {
    "select_id": 1,
    "cost_info": {
      "query_cost": "153230.79"
    },
    "duplicates_removal": {
      "using_temporary_table": true,
      "using_filesort": false,
      "nested_loop": [
        {
          "table": {
            "table_name": "story_allocation_days",
            "access_type": "ALL",
            "possible_keys": [
              "index_story_allocation_days_on_workspace_id",
              "index_story_allocation_days_on_assignment_id_and_date",
              "index_story_allocation_days_on_assignment_id"
            ],
            "rows_examined_per_scan": 430268,
            "rows_produced_per_join": 43026,
            "filtered": "10.00",
            "cost_info": {
              "read_cost": "80548.24",
              "eval_cost": "8605.36",
              "prefix_cost": "89153.60",
              "data_read_per_join": "3M"
            },
            "used_columns": [
              "id",
              "assignment_id",
              "story_id",
              "workspace_id",
              "account_id",
              "current",
              "date",
              "minutes",
              "created_at",
              "updated_at",
              "deleted_at",
              "cost_amount_in_cents",
              "bill_amount_in_cents",
              "cost_rate_in_cents",
              "bill_rate_in_cents"
            ],
            "attached_condition": "isnull(`bm_rpm`.`story_allocation_days`.`deleted_at`)"
          }
        },
        {
          "table": {
            "table_name": "workspaces",
            "access_type": "eq_ref",
            "possible_keys": [
              "PRIMARY",
              "index_workspaces_on_account_id"
            ],
            "key": "PRIMARY",
            "used_key_parts": [
              "id"
            ],
            "key_length": "4",
            "ref": [
              "bm_rpm.story_allocation_days.workspace_id"
            ],
            "rows_examined_per_scan": 1,
            "rows_produced_per_join": 4302,
            "filtered": "10.00",
            "distinct": true,
            "cost_info": {
              "read_cost": "43026.80",
              "eval_cost": "860.54",
              "prefix_cost": "140785.76",
              "data_read_per_join": "21M"
            },
            "used_columns": [
              "id",
              "is_budgeted",
              "account_id"
            ],
            "attached_condition": "(`bm_rpm`.`workspaces`.`is_budgeted` = TRUE)"
          }
        },
        {
          "table": {
            "table_name": "assignments",
            "access_type": "eq_ref",
            "possible_keys": [
              "PRIMARY",
              "index_assignments_on_assignee_id"
            ],
            "key": "PRIMARY",
            "used_key_parts": [
              "id"
            ],
            "key_length": "4",
            "ref": [
              "bm_rpm.story_allocation_days.assignment_id"
            ],
            "rows_examined_per_scan": 1,
            "rows_produced_per_join": 2151,
            "filtered": "50.00",
            "distinct": true,
            "cost_info": {
              "read_cost": "4302.68",
              "eval_cost": "430.27",
              "prefix_cost": "145948.98",
              "data_read_per_join": "134K"
            },
            "used_columns": [
              "id",
              "assignee_id"
            ],
            "attached_condition": "(``bm_rpm`.`assignments`.`assignee_id` is not null)"
          }
        },
        {
          "table": {
            "table_name": "participations",
            "access_type": "ref",
            "possible_keys": [
              "index_participations_on_account_id_and_workspace_id",
              "index_participations_on_workspace_id_and_user_id",
              "index_participations_for_sad_api_index",
              "index_participations_on_account_id",
              "index_participations_on_workspace_id"
            ],
            "key": "index_participations_for_sad_api_index",
            "used_key_parts": [
              "workspace_id",
              "account_id"
            ],
            "key_length": "10",
            "ref": [
              "bm_rpm.story_allocation_days.workspace_id",
              "const"
            ],
            "rows_examined_per_scan": 7,
            "rows_produced_per_join": 5537,
            "filtered": "33.33",
            "using_index": true,
            "distinct": true,
            "cost_info": {
              "read_cost": "3959.11",
              "eval_cost": "1107.46",
              "prefix_cost": "153230.79",
              "data_read_per_join": "11M"
            },
            "used_columns": [
              "id",
              "workspace_id",
              "type",
              "account_id",
              "access_integer"
            ],
            "attached_condition": "((`bm_rpm`.`participations`.`access_integer` >= 30) and ((`bm_rpm`.`participations`.`type` = 'MavenParticipation') or (`bm_rpm`.`workspaces`.`account_id` = 5071)))"
          }
        }
      ]
    }
  }
} |

EDIT #2:

Here's the SHOW CREATE TABLE for story_allocation_days:

| story_allocation_days | CREATE TABLE `story_allocation_days` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `assignment_id` int(11) NOT NULL,
  `story_id` int(11) NOT NULL,
  `workspace_id` int(11) NOT NULL,
  `account_id` int(11) NOT NULL,
  `current` tinyint(1) NOT NULL,
  `date` date NOT NULL,
  `minutes` int(11) NOT NULL DEFAULT '0',
  `created_at` datetime NOT NULL,
  `updated_at` datetime NOT NULL,
  `deleted_at` datetime DEFAULT NULL,
  `cost_amount_in_cents` bigint(20) DEFAULT NULL,
  `bill_amount_in_cents` bigint(20) DEFAULT NULL,
  `cost_rate_in_cents` bigint(20) DEFAULT NULL,
  `bill_rate_in_cents` bigint(20) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `index_story_allocation_days_on_workspace_id` (`workspace_id`),
  KEY `index_story_allocation_days_on_assignment_id_and_date` (`assignment_id`,`date`),
  KEY `index_story_allocation_days_on_account_id` (`account_id`),
  KEY `index_story_allocation_days_on_story_id_and_date` (`story_id`,`date`),
  KEY `index_story_allocation_days_on_date` (`date`),
  KEY `index_story_allocation_days_on_created_at` (`created_at`),
  KEY `index_story_allocation_days_on_updated_at` (`updated_at`),
  KEY `index_story_allocation_days_on_assignment_id` (`assignment_id`),
  KEY `index_story_allocation_days_on_story_id` (`story_id`)
) ENGINE=InnoDB AUTO_INCREMENT=9646396 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci |
10
  • Take out SQL_CALC_FOUND_ROWS. This is known to have a high performance cost, and there's no reason you need it in the query you show. Sep 12, 2021 at 23:40
  • I'll update the question with further details, but TL;DR is that the original query still takes 5.51 seconds for 99822 rows, even without SQL_CALC_FOUND_ROWS. Sep 12, 2021 at 23:46
  • please edit your question to show output (as text, not images) of show create table yourtablename for all the tables in the query
    – ysth
    Sep 12, 2021 at 23:48
  • it seems likely to me you want to be reading from participations by account_id and then getting the workspaces/story_allocation_days for those participations
    – ysth
    Sep 12, 2021 at 23:51
  • @ysth I'm unable to add SHOW CREATE TABLE for all 4 tables due to StackOverflow's 30,000 char limit on a question body. However, I was able to add such a statement for the main table, story_allocation_days. Hopefully that's helpful. Sep 12, 2021 at 23:59

2 Answers 2

1

I would offer the following. Rewrite the query starting with the participants as you are starting with that granularity first vs ALL the "sad" entries. Adding the MySQL keyword "STRAIGHT_JOIN" tells the engine... dont think for me, do as I wrote it... since you are doing "Distinct", doing distinct as "*" might be assessing all columns for uniqueness. If you put in just the "sad.id" (assumption is each table has "Id" as its own primary key. I put below as [whateverItsUniqueKeyIdIs] but think it probably is just sad.id and to change accordingly below. if that is the unique value for the row, the engine does not need to go to the actual data row pages to qualify distinctness.

Next, the indexes. By having multi-column composite indexes on the tables to cover the joins and conditions of WHERE will also help the engine query based on the indexes directly instead of going to the raw data pages. Dont try to have the tables have indexes on individual columns because just one wont be as optimized as those covering the criteria you are looking for.

Table                  Index
Participants           ( account_id, access_integer, type, workspace_id)
Workspaces             ( id, is_budgeted, account_id )
Story_Allocation_Days  ( workspace_id, deleted_at, assignment_id, [whateverItsUniqueKeyIdIs]
Assignments            ( id, assignee_id )

Finally the actual query.

SELECT STRAIGHT_JOIN DISTINCT 
        SQL_CALC_FOUND_ROWS sad.[whateverItsUniqueKeyIdIs]
    FROM 
        participations p
            INNER JOIN workspaces ws
                ON p.workspace_id = ws.id
                AND ws.is_budgeted = TRUE 
            INNER JOIN story_allocation_days sad
                ON p.workspace_id = sad.workspace_id
                AND sad.deleted_at is null
                INNER JOIN assignments a
                    ON sad.assignment_id = a.id  
                    AND a.assignee_id IS NOT NULL
    where
            p.account_id = 5071
        AND p.access_integer >= 30
        AND ( p.type = 'MavenParticipation' 
            OR ws.account_id = 5071)

The only other possible killer drag is the "OR ws.Account_id = 5701", but should still be quick as the primary basis is on the participant-restricted criteria and not all accounts.

Response to Comments

With respect to intuition of querying. Databases can be full of bloat and hide in the weeds of what you are searching sometimes. As a person behind the database, you know better where the granularity is within the data.

I try to look at queries at the root "what do I want". In your case you wanted all participants of a specific account (with some minor outlying other criteria). By putting that in my head first, that becomes my first table. Now, look at the indexes that help me get only that one component FIRST, and only that. Then, I join outwards to the other pieces like your workspace and story_allocation_days and assignments. I apply their restrictive criteria directly at their JOIN condition with exception of the "OR" on the ws.account_id due to the context of its comparison, but the core was still always p.account_id = 5071 AND p.access_integer >= 30

Now, having proper / effective indexes is another thing, and having them in the correct ordinal order/position can be a huge impact, not just having multiple indexes with one field, but more efficient single index with multiple columns -- based on types of queries you run most frequently.

As I have described in other post answers, and with your scenario of data. You originally started with the story_allocation_days table. It had to blow throught everything, for every account, even before it got to the participants table, then on to all the workstation and assignments. The engine does not know what you want precisely, hence having effective indexes.

By starting with the key table in question Participants, and knowing you want a specific account_id, and access_integer value, that is your starting point. Think of a room of boxes and they hold all the participants data. On each box in the room has first-things-first, an "Account_ID" on the side of the box, and all the boxes are sequentially ordered. You could have 1,000 boxes and you can run directly to find the one account ID you want. You have already eliminated every other box in the rest of the room. So now, you open that box and within it, they are pre-sorted by the access_integer from 1-??. So now, you flip through and find 31 and you are done.

Only then do you have your short list of things to find the rest of the details and you have not even looked at the raw data pages. On the top of each page has a note for the type and workspace ID because that is how I suggested the index. So again, without having to go into the document to flip pages over to see where the type is and what workspace, those are written at the top of the page for quick reference. These are the fields that are used to join to the next level of tables. All done without getting to the raw data of the underlying records.

By using the straight_join clause, you have already eliminated all the heavy lifting of the query from the engine, and it is now going to just do simple joins with proper indexes to those secondary level tables. HTH.

3
  • Wow, so using your suggestion with sad.* brought the query down to 0.83 seconds, and using sad.id brought it down even further to 0.41 seconds! From the EXPLAIN statements, it looks like changing the order of the JOINs, along with using STRAIGHT_JOIN, moves the sad lookup to 3rd position. I noticed that the optimizer decided to use the index_story_allocation_days_on_workspace_id index in this case, as well. Is this change of position the decisive factor which allowed the optimizer to use the index, and is this index the reason for the improvement? Sep 13, 2021 at 2:04
  • Also @drapp, what's the intuition behind re-arranging the order of the column declarations so that participations comes before story_allocation_days? Is it that starting with sads means we'll have to iterate over an order of magnitude more rows up-front, instead of kicking the can down the road until we've narrowed down that list of rows to something more manageable? Sep 13, 2021 at 4:18
  • 1
    @RichieThomas, see revised answer for comment/feedback to your questions.
    – DRapp
    Sep 13, 2021 at 11:53
1

OR (usually) kills performance. UNION is a workaround:

( SELECT   sad.*
    FROM  `story_allocation_days` AS sad
    INNER JOIN  `workspaces` AS w  ON w.`id` = sad.`workspace_id`
    INNER JOIN  `participations` AS p  ON p.`workspace_id` = w.`id`
    INNER JOIN  `assignments` AS a  ON a.`id` = sad.`assignment_id`
    WHERE  sad.deleted_at = '1970-01-01'
      AND  w.`is_budgeted` = TRUE
      AND  p.account_id = 5071
      AND  p.access_integer >= 30
      AND  p.type = 'MavenParticipation'
      AND  a.assignee_id IS NOT NULL
)
UNION DISTINCT
( SELECT   sad.*
    FROM  `story_allocation_days` AS sad
    INNER JOIN  `workspaces` AS w  ON w.`id` = sad.`workspace_id`
    INNER JOIN  `participations` AS p  ON p.`workspace_id` = w.`id`
    INNER JOIN  `assignments` AS a  ON a.`id` = sad.`assignment_id`
    WHERE  sad.deleted_at = '1970-01-01'
      AND  w.`is_budgeted` = TRUE
      AND  w.account_id = 5071
      AND  p.account_id = 5071
      AND  p.access_integer >= 30
      AND  a.assignee_id IS NOT NULL
)

That will need these. The order of the columns in these indexes is important. (I am assuming that each table has PRIMARY KEY(id).)

w:  INDEX(account_id, is_budgeted)
p:  INDEX(account_id, type, access_integer, workspace_id)
p:  INDEX(workspace_id, account_id, access_integer)
sad:  INDEX(workspace_id, deleted_at)
sad:  INDEX(assignment_id, deleted_at)

There are some extra indexes because I can't tell what order the Optimizer will prefer to access the table. (It is data-dependent.)

I removed SQL_CALC_FOUND_ROWS since it is unnecessary without LIMIT.

Let me know if it is common for the same rows to show up in both SELECTs and there are lots of big columns (eg TEXT or BLOB) in story_allocation_days. In that case, I will want to modify the query to avoid hauling the bulk around so much:

SELECT sad2.*
    FROM ( ( SELECT sad.id
               FROM ... (1st 4-way join plus WHERE)
             UNION DISTINCT
           ( SELECT sad.id
               FROM ... (2nd 4-way join plus WHERE)
         ) )  AS u
    JOIN `story_allocation_days` AS sad2  USING(id);

This could be faster because of hauling around only id until finished with the UNION.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.