The short answer is yes, there are ways to make this particular query fun faster, but with a corpus of 20 million rows, 5 seconds isn't bad. You'll need to seriously consider whether the below suggestions are optimal for your FT search workload and weigh the costs vs the benefits. If you blindly implement these, you're going to have a bad time.
General Suggestions for Improving Sql Server Full-text Search Performance
Reduce the size of the Full-Text index being searched The smaller the FT Index, the faster the query. There are a couple of ways to reduce the FT index size. The first two may or may not apply and the third would take considerable work to accomplish.
Add domain specific noise words Noise words are words that don't add value to full-text search queries, such as "the", "and", "in", etc. If there are terms related to the business that add no value being indexed, you may benefit from excluding them from the FT index. Consider a hypothetical full-text index on the MSDN library. Terms such as "Microsoft", "library", "include", "dll" and "reference" may not add value to search results. (Is there any real value in going to http://msdn.microsoft.com and searching for "microsoft"?) A FT index of legal opinions might exclude words such as "defendant", "prosecution" and "legal", etc.
Strip out extraneous data using iFilters Full-Text search using Windows iFilters to extract text from binary documents. This is the same technology that window search functionality uses to search pdf and powerpoint documents. The one case where this is particularly useful is when you have a description column that can contain HTML markup. By default, Sql Server full-text search will index everything, so you get terms such as "font-family", "Arial" and "href" as searchable terms. Using the HTML iFilter can strip out the markup.
The two requirements for using an iFilter in your FT index is that the indexed column is a VARBINARY and there is a "type" column that contains the file extension. Both these can be accomplished with computed columns.
CREATE TABLE t (
....
description varbinary(max),
FTS_description as (CAST(description as VARBINARY(MAX)),
FTS_filetype as ( N'.html' )
)
-- Then create the fulltext index on FTS_description specifying the filetype.
Index portions of the table and stitch together results There are several ways to accomplish this, but the overall idea is to split the table into smaller chunks, query the chunks individually and combine the results. For example, you could create two indexed views, one for the current year and one for historical years with full-text indexes on them. Your query to return 100 rows changes to look like this:
DECLARE @rows int
DECLARE @ids table (id int not null primary key)
INSERT INTO @ids (id)
SELECT TOP (100) id
FROM vw_2013_FTDocuments WHERE CONTAINS (....)
ORDER BY Id DESC
SET @rows = @@rowcount
IF @rows < 100
BEGIN
DECLARE @rowsLeft int
SET @rowsLeft = 100 - @rows
INSERT INTO @ids (id) SELECT TOP (@rowsLeft) ......
--Logic to incorporate the historic data
END
SELECT ... FROM t INNER JOIN @ids .....
This can result in a substantial reduction in query times at the cost of adding complexity to the search logic. This approach is also applicable when searches are typically limited to a subset of the data. For example, craigslist might have a FT index for Housing, one for "For Sale" and one for "Employment". Any searches done from the home page would be stitched together from the individual indexes while the common case of searches within a category are more efficient.
Unsupported technique that will probably break in a future version of Sql Server.
You'll need to test extensively with data of the same quantity and quality as production. If the behavior changes in future versions of Sql server, you will have no right to complain. This is based off of observations, not proof. Use at your own RISK!!
A bit of full-text history In Sql Server 2005, the full-text search functionality was in an external process from sqlservr.exe. The way FTS was incorporated into query plans was as a black-box. Sql server would pass FTS a query, FTS would return a stream of id's. This limited the plans to available to Sql Server to plans where the FTS operator could basically be treated as a table scan.
In Sql Server 2008, FTS was integrated into the engine which improved performance. It also gave the optimizer new options for FTS query plans. Specifically, it now has the option to probe into the FTS index inside a LOOP JOIN operator to check if individual rows match the FTS predicate.(see http://sqlblog.com/blogs/joe_chang/archive/2012/02/19/query-optimizer-gone-wild-full-text.aspx for an excellent discussion of this and ways things can go wrong .)
Requirements for our optimal FTS query plan There are two characteristics to strive for to get the optimal query plan.
- No Sort Operations. Sorting is slow, and we don't want to sort either 20 million rows or 350,000 rows.
- Don't return all 350k rows matching the FTS predicate. We need to avoid this if at all possible.
These two criteria eliminate any plan with a hash join, as a hash join requires consuming all of one input to build the hash table.
For plans with a loop join, there are two options. Scan the clustered index backwards, and for each row probe into the fulltext search engine to see if that particular row matches. In theory, this seems like a good solution, as once we match 100 rows, we're done. We may have to try 10,000 id's to find the 100 that match, but that may be better than reading all 350k. It could also be worse (see above link to Joe Chang's blog) if each probe is expensive, then our 10k probes could take substantially longer than just reading all 350k rows.
The other loop join option is to have the FTS portion on the outer side of the loop, and seek into the clustered index. Unfortunately, the FTS engine doesn't like to return results in reverse order, so we'd have to read all 350k, and then sort them to return the top 100.
The roadblock is getting the FTS engine to return rows in reverse order. If we can overcome this, then we can reduce the IO's to reading only the last 100 rows that match. Fortunately the FTS engine has a tendancy to return rows in order by the key of the unique index specified when the index was created. (This is a natural side-effect of the internal storage the FTS engine uses)
By adding a computed column that is the negative of the id, and specifying a unique index on that column when creating the FT index, then we're really close.
CREATE TABLE t (id int not null primary key, txt varchar(max), neg_id as (-id) persisted )
CREATE UNIQUE INDEX IX_t_neg_id on t (neg_id)
CREATE FULLTEXT INDEX on t ( txt ) KEY INDEX IX_t_neg_id
Now for our query, we'll use CONTAINSTABLE, and some LEFT-join trickery to ensure that the FTS predicate doesn't end up on the inside of a LOOP JOIN.
SELECT TOP (100) t.id, t.txt
FROM CONTAINSTABLE(t, txt, 'house') ft
LEFT JOIN t on tf.[Key] = t.neg_id ORDER BY tf.[key]
The resulting plan should be a loop join that reads only the last 100 rows from the FT index.
Small gusts of wind that could blow down this house of cards:
- Complex FTS queries (as in multiple terms or the use of NOT or OR operators can cause Sql 2008+ to get "Smart" and translate the logic into Multiple FTS queries that are joined in the query plan.
- Any Cumulative Update, Service Pack or Major version upgrade could render this approach useless.
- It may work in 95% of the cases and timeout in the remaining 5%.
- It may not work at all for you.
Good Luck!