ICU provides a way of cutting down the size of the .dat file. I'm almost certain I don't need most of the encodings that are default. If I want to build a CJK .dat file specifically for sqlite, which ones can I cut out.
I just need the tokenizer to work and possibly collation. Seems that all those character conversions may not really be necessary. At 17MB, it is too FAT! For all database, we use
PRAGMA encoding = UTF8;
Data Customizer Link: http://apps.icu-project.org/datacustom/
To put it another way, if I'm using UTF8 in SQLite to collate and index, what parts of the dat file do I really need? I bet the majority is never used. I suspect I don't need the Charset Mapping Tables, and maybe not some of the Misc data.
ICU. This tool will generate a data library that can only be used with the 4.8 series of ICU. The help page provides information on how to use this tool.
- Charset Mapping Tables (4585 KB) <-- axe?
- Break Iterator (1747 KB) <-- seems like i need this
- Collators (3362 KB) <-- seems like i need this for sorting (but maybe not)
- Rule Based Number Format (292 KB) <-- axe?
- Transliterators (555 KB) <-- axe?
- Formatting, Display Names and Other Localized Data (856 KB) <-- axe?
- Miscellaneous Data (5682 KB) <-- axe?
- Base Data (311 KB) <-- seems basic
Update. It seems that everything can be removed except for Base Data and Break Iterator. Regarding the Collators from http://userguide.icu-project.org/icudata:
The largest part of the data besides conversion tables is in collation for East Asian languages. You can remove the collation data for those languages by removing the CollationElements entries from those source/data/locales/*.txt files. When you do that, the collation for those languages will become the same as the Unicode Collation Algorithm.
This seems "good enough".
On Collation
Starting in release 1.8, the ICU Collation Service is updated to be fully compliant to the Unicode Collation Algorithm (UCA) (http://www.unicode.org/unicode/reports/tr10/ ) and conforms to ISO 14651. There are several benefits to using the collation algorithms defined in these standards. Some of the more significant benefits include:
Unicode contains a large set of characters. This can make it difficult for collation to be a fast operation or require collation to use significant memory or disk resources. The ICU collation implementation is designed to be fast, have a small memory footprint and be highly customizable.
The algorithms have been designed and reviewed by experts in multilingual collation, and therefore are robust and comprehensive.
Applications that share sorted data but do not agree on how the data should be ordered fail to perform correctly. By conforming to the UCA/14651 standard for collation, independently developed applications, such as those used for e-business, sort data identically and perform properly.
The ICU Collation Service also contains several enhancements that are not available in UCA. For example:
Additional case handling: ICU allows case differences to be ignored or flipped. Uppercase letters can be sorted before lowercase letters, or vice-versa.
Easy customization: Services can be easily tailored to address a wide range of collation requirements.
Flexibility: ICU offers both sort key generation and fast incremental string comparison. It also provides low-level access to collation data through the collation element iterator (§)
Update2. If Break Iterator is removed from the .dat, the following occurs:
sqlite> CREATE VIRTUAL TABLE test USING fts4(tokenize=icu);
sqlite> CREATE VIRTUAL TABLE testaux USING fts4aux(test);
sqlite> .import test.csv test
Error: SQL logic error or missing database