0

I have a large SQLite DB file ~ 3 GB. In it there are 5 tables (Table1-5) with the following definition:

CREATE TABLE IF NOT EXISTS [TableOne] (
    [EntryDate_Epoch_Utc] INTEGER DEFAULT (strftime('%s','now')),
    [Key] TEXT PRIMARY KEY,
    [Count] INTEGER NOT NULL
)

I have the below SELECT query that I need to run every 1 min:

using (Db)
{
    using (Db.OpenTransaction())
    {
        foreach (var table in _tables)
        {
            var query = StringExtensions.FormatWith("SELECT Key, Count FROM {0} ORDER BY Count DESC LIMIT 100", table);

            var result = Db.Select<Result>(query);
            // do something with the result
        }
    }
}

At the start of my program when the size of the DB is small the query runs quite fast ~ 400 ms however as the DB file gets larger (towards the end of the day) it takes around 30 seconds to run the above query.

Is there any way to improve the query? below is the connection string I use when opening the SQLite DB file:

var conStr = new SQLiteConnectionStringBuilder
{
    DataSource = dbFilePath,
    FailIfMissing = false,
    Pooling = true,
    DateTimeKind = DateTimeKind.Utc,
    DateTimeFormat = SQLiteDateFormats.UnixEpoch,
    JournalMode = SQLiteJournalModeEnum.Memory,
    SyncMode = SynchronizationModes.Off,
    UseUTF16Encoding = true,
    PageSize = 4096,
    CacheSize = 5000,
    Version = 3
}.ToString();
3
  • 1
    Do you have index on Count field?
    – PM 77-1
    Aug 3, 2014 at 23:50
  • @ PM 77-1, No I don't, the DDL is exactly as above, considering that [Count] cannot be UNIQUE are you suggesting I should create a NON UNIQUE index on [Count]?
    – MaYaN
    Aug 3, 2014 at 23:54
  • Yes, I think it will allow for figuring TOP 100 with just the index search. See Section 8 in The SQLite Query Planner. I would say it's worth a try.
    – PM 77-1
    Aug 4, 2014 at 0:00

1 Answer 1

3

To find the 100 rows with the largest Count values, the database has to go through all rows of the table.

When you create an index on this column:

CREATE INDEX whatever ON TableOne(Count);

the database can just take the last 100 entries from the index.

3
  • tnx for the explanation, would it help to add the ordering explicitly to the index? i.e. CREATE INDEX whatever ON TableOne(Count DESC)?
    – MaYaN
    Aug 4, 2014 at 11:54
  • I see, so adding DESC or ASC would be redundant or would it mean something completely different? I am trying to understand how these 2 work with INDEXes but cannot find any documentation on the subject.
    – MaYaN
    Aug 4, 2014 at 11:56
  • 1
    The index sort order is useful when using ORDER BY with a multi-column index.
    – CL.
    Aug 4, 2014 at 11:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.