27

This is the query:

  1. It looks like some NULL values are appearing in the list.
  2. Some NULL values are being filtered out by the query. I have checked.
  3. If I add AND AdditionalFields = '', both these results are still returned
  4. AdditionalFields is a varchar(max)
  5. The database is SQL Server 10 with Compatibility Level = Sql Server 2005 (90)
  6. I am using Management Studio 2008

I appear to have empty strings whose length is NULL, or NULL values that are equal to an empty string. Is this a new datatype?!

EDIT: New datatype - hereby to be referred to as a "Numpty"

EDIT 2 inserting the data into a temporary table turns Numpties into NULLS. (The result from this sql is 10)

CREATE TABLE #temp(ID uniqueidentifier , Value varchar(max))

INSERT INTO #temp 
SELECT top 10 g.ID, g.AdditionalFields
FROM grants g 
WHERE g.AdditionalFields IS NOT NULL AND LEN(g.AdditionalFields) IS NULL

SELECT COUNT(*) FROM #temp WHERE Value is null

DROP TABLE #temp

EDIT 3 And I can fix the data by running an update:

UPDATE Grants SET AdditionalFields = NULL
WHERE AdditionalFields IS NOT NULL AND LEN(AdditionalFields) IS NULL

So that makes me think the fields must contain something, rather than some problem with the schema definition. But what is it? And how do I stop it ever coming back?

EDIT 4 There are 2 other fields in my database, both varchar(max) that return rows when the field IS NOT NULL AND LEN(field) IS NULL. All these fields were once TEXT and were changed to VARCHAR(MAX). The database was also moved from Sql Server 2005 to 2008. It looks like we've got ANSI_PADDING etc OFF by default.

Another example: enter image description here

Converting to varbinary enter image description here

Execution plan: Execution plan EDIT 5: removed table definition - turned out to be not relevant in the end

EDIT 6 Scripts to generate scripts for altering TEXT to VARCHAR(MAX) then update values to prevent bug and enhance performance

--Generate scripts to alter TEXT to VARCHAR(MAX)
SELECT 'ALTER TABLE [' + tab.table_schema + '].[' + tab.table_name  + '] ALTER COLUMN [' + col.column_name + '] VARCHAR(MAX)' + CASE WHEN col.IS_NULLABLE = 'YES' THEN ' NULL' ELSE ' NOT NULL' END + ' GO'
FROM INFORMATION_SCHEMA.tables tab
INNER JOIN INFORMATION_SCHEMA.COLUMNS col ON col.table_name = tab.table_name
          AND tab.table_schema = col.table_schema
          AND tab.table_catalog = col.table_catalog
WHERE tab.table_type <> 'VIEW' and col.DATA_TYPE = 'text'

--Generate scripts to set value to value in VARCHAR(MAX) fields
SELECT 'UPDATE [' + tab.table_schema + '].[' + tab.table_name  + '] SET [' + col.column_name + '] = [' + col.column_name + ']'
FROM INFORMATION_SCHEMA.tables tab
INNER JOIN INFORMATION_SCHEMA.COLUMNS col ON col.table_name = tab.table_name
          AND tab.table_schema = col.table_schema
          AND tab.table_catalog = col.table_catalog
WHERE tab.table_type <> 'VIEW' AND col.DATA_TYPE = 'varchar' and col.CHARACTER_MAXIMUM_LENGTH = -1
36
  • 2
    Is your query posted verbatim? Is there a chance you misspelled AdditionalFields in the WHERE clause but not in the SELECT list (which would cause filtering on another field)? Something like WHERE 'g.AdditionalFields' IS NOT NULL would yield similar behavior (since you're not filtering a field but a string constant).
    – Quassnoi
    Apr 24, 2012 at 10:08
  • 1
    The first result isn't surprising. An empty string and NULL are two separate concepts (unless you're weird, like Oracle) Apr 24, 2012 at 10:10
  • 1
    As a debug hint: Just output the PK so you see the row that generates the result.
    – MicSim
    Apr 24, 2012 at 11:02
  • 3
    And while we are at it, you could post a screenshot of the entire ssms window so we can verify your claim more easily. Apr 24, 2012 at 11:36
  • 3
    I've created a connect item for this. I haven't tested to see if this issue occurs in SQL Server 2012 or not though. Apr 28, 2012 at 12:09

5 Answers 5

19
+200

I got a sample code to reproduce the above behavior. The problem arises when you have a TEXT field that stores a value larger than it can fit in a row and if you set it afterwards to NULL and perform the column conversion to VARCHAR(MAX).

The large value gets stored in a separate page. Then you set the value of this field to NULL. If you now convert this column to a VARCHAR(MAX), then SQL Server seems to not get it right. Typically on a TEXT to VARCHAR(MAX) conversion the external pages stay as they are, but maybe because it was set to NULL, the column altering messes things up.

Update: It doesn't seem to have anything to do with the large values in the TEXT column. Short values show the same behavior (extended sample). So it's just the explicit setting to NULL through an UPDATE and the conversion that matters.

CREATE TABLE [dbo].[Test](
    [Id] [int] NOT NULL,
    [Value] [text] NULL,
 CONSTRAINT [PK_Test] PRIMARY KEY CLUSTERED 
(
    [Id] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]

GO

INSERT INTO Test VALUES (1, 'test')
INSERT INTO Test VALUES (2, '')
INSERT INTO Test VALUES (3, NULL)
INSERT INTO Test VALUES (4, '012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789')
INSERT INTO Test VALUES (5, 'short string')
GO

update test SET value = null where ID = 4
update test SET value = null where ID = 5
GO

ALTER TABLE test ALTER COLUMN value varchar(max)
GO

select id, value, len(value) as length
from test
where value is not null
GO

The result is:

1   test    4
2           0
4   NULL    NULL
5   NULL    NULL

An easy fix for this problem would be to reassign the values in the VARCHAR(MAX) columns.

UPDATE Test SET value = value

This seems to put the values in the rows that were previously stored in external pages. (See for reference: NTEXT vs NVARCHAR(MAX) in SQL 2005)

5
  • The Text data type is the issue here; you need to handle them differently where datalength(Value)=0. See stackoverflow.com/questions/33409/… for more details
    – u07ch
    Apr 27, 2012 at 9:34
  • @u07ch: Read the problem carefully. It's not about the TEXT data type, it's about VARCHAR(MAX) NULL values giving TRUE when checked with IS NOT NULL.
    – MicSim
    Apr 27, 2012 at 10:58
  • 2
    +1 for providing a repo. The row where the NULL is inserted explicitly is the only one where the NULL_BITMAP is set to 1 for that column. For text datatypes I've noticed before that updating to NULL doesn't update the bitmap but seems to set some sort of flag elsewhere in the pointer or pointed to structure. If you do select [Id], [Value] from [Test] where [Value] is not null before the conversion to max then correct results are returned but the NULL_BITMAP still says zero so it must look somewhere else to determine that. Apr 28, 2012 at 10:18
  • Great troubleshooting MicSim: you reduce the problem to a specific set of steps, reproduce the unwanted behavior, and find an explanation, congratulations!
    – tcbrazil
    Apr 28, 2012 at 10:54
  • Superb! Thank-you. Better go fix all the Text and Varchar(max) fields in our databases...
    – Colin
    Apr 28, 2012 at 14:47
6

This is just an addition to McSim's answer using SQL Server Internals Viewer to look at the individual stages.

CREATE TABLE [dbo].[Test](
    [Id] [int] NOT NULL PRIMARY KEY ,
    [Value] [text] NULL)


INSERT INTO Test VALUES (1, '')

Row after initial insert

After Insert Main Row

Text value after initial insert

After Insert Text Value

update [Test] SET [Value] = null 

Row after update to NULL

This is identical to the row shown earlier so I haven't repeated the screenshot. Specifically the NULL_BITMAP does not get updated to reflect the new NULL value.

Text value after update to NULL

Text value after Update

The Type bits have changed and Internals Viewer shows this as no longer containing a value for the Data column.

At this point running the following correctly returns no rows

SET STATISTICS IO ON
select [Id]
from [Test]
where [Value] is not null

So SQL Server must follow the text pointer and look at the value there to determine NULL-ability.

ALTER TABLE [Test] ALTER COLUMN [Value] varchar(max)

This is a metadata only change. Both the inrow and out of row data remain unchanged.

However at this point running the following incorrectly returns the row.

SET STATISTICS IO ON
select [Id]
from [Test]
where [Value] is not null

The output of STATISTICS IO

Scan count 1, logical reads 2, ... lob logical reads 1

shows that it still does actually follow the text pointer but presumably in the varchar(max) case there must be a different code path that incorrectly ends up taking the value from the NULL_BITMAP regardless (the value of which has never been updated since the initial insert).

1
  • Many thanks Martin. We'd have got here sooner if I'd had rights to do this on the server. We copied a backup down, but I didn't get time to try yesterday.
    – Colin
    Apr 28, 2012 at 15:03
1

Colin:

I'm pretty sure all this is happening because of the database conversion. Since you need to solve this thing ASAP, my suggestion is to guarantee that your AdditionalFields data is ok first, and try to understand why this happens after:

  1. Do a backup;
  2. Run this T-SQL:

    update grants
    set AdditionalFields = ltrim(rtrim(isnull(AdditionalFields,'')))
    

The isnull function will transform your null values in empty strings, and the left/right trim should guarantee that even fields with more than one space will have the same value after.

Could you run this and feedback us later with the results?

Best regards

1
  • I already know that I can turn numpties into nulls by running an update (I ran the update above inside a transaction) so I'm not sure that this will tell us anything more. I wanted to keep the data in place to help with investigation, and the problem hasn't resulted in any significant issues yet. I don't have enough rights to view the underlying page data, so I think I will try to backup and pull the database file down onto a system where I do have admin rights. Not at work now, so tune in tomorrow...
    – Colin
    Apr 26, 2012 at 18:30
0

As others have pointed out this result is utterly impossible.

  1. Please post a screenshot of the actual execution plan.
  2. Please run dbcc checkdb and post the error messages, if any.

(2) is actually my favorite right now.

1
  • CHECKDB found 0 allocation errors and 0 consistency errors in database
    – Colin
    Apr 26, 2012 at 13:49
0

I suspect the word NULL is neing stored in the db, use select * from blah where mycolumn = 'NULL'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.