We have a database that we would like to index with Sphinx search engine. The problem is that Sphinx requires integer ID for each document but our database has GUIDs (actually, random GUID-like strings) as primary keys. I could generate synthetic IDs on the fly as described in this recipe, but it is good only for full indexes. What if I want to have incremental indexes or runtime index? Are there any best practices to deal with GUIDs using Sphinx that would work with incremental indexes and runtime indexes? The databases in question would be quite large so I wouldn't want to reindex them frequently.
1 Answer
Real-time indexes: you can convert your non-numeric GUID to a numeric value (you can use crc32 or md5 casted to a number) like this:
mysql> select conv(mid(md5('abc'), 1, 16), 16, 10);
+--------------------------------------+
| conv(mid(md5('abc'), 1, 16), 16, 10) |
+--------------------------------------+
| 10376663631224000432 |
+--------------------------------------+
1 row in set (0.00 sec)
test or google to figure out what algorithm will give less collisions.
Once you have numeric IDs you can just INSERT your docs to Sphinx RT index.
Incremental indexes: you can convert GUID to numbers using the same method described above, but you will have to remember somewhere where the main part of your index finishes to build the delta. You can do it easily if you have some "updated" or "inserted" field in your table. sql_query_range will be smth like "select ... where updated > (select last_updated from sphinx_helper)". You can't base on the ids in this case as they're not sequential.
I've tested the half of md5 algo on 11.8M different domains:
mysql> update domain_tmp set hash = conv(mid(md5(domain), 1, 16), 16, 10);
Query OK, 5901483 rows affected (1 min 59.03 sec)
Rows matched: 11800403 Changed: 5901483 Warnings: 0
mysql> select count(*) from domain_tmp;
+----------+
| count(*) |
+----------+
| 11800403 |
+----------+
1 row in set (16.30 sec)
mysql> select count(distinct hash) from domain_tmp;
+----------------------+
| count(distinct hash) |
+----------------------+
| 11800403 |
+----------------------+
1 row in set (1 min 5.51 sec)
i.e. no collisions occured. So if you don't have much more documents in your dataset it should be ok to use half of md5.
-
I can't really change all the table structure (the DB has 50+ tables with table structure generated by ORM-like system) to generate new IDs. I could use md5 method probably... But I'm worried about the collisions.– StasMFeb 10, 2011 at 20:29
-
1You don't need to change the table structure, you can generate new IDs on the fly when you make SELECT by Sphinx indexer. As for the collisions - half of md5 looks quite ok, I've tested it on 11.8M different domains - no one collision. Feb 12, 2011 at 11:59