3

While debugging some performance issues I discovered that Entity framework was loading a lot of records via lazy loading (900 extra query calls ain't fast!) but I was sure I had the correct include. I've managed to get this down to quite a small test case to demonstrate the confusion I'm having, the actual use case is more complex so I don't have a lot of scope to re-work the signature of what I'm doing but hopefully this is a clear example of the issue I'm having.

Documents have Many MetaInfo rows related. I want to get all documents grouped by MetaInfo rows with a specific value, but I want all the MetaInfo rows included so I don't have to fire off a new request for all the Documents MetaInfo.

So I've got the following Query.

ctx.Configuration.LazyLoadingEnabled = false;

var DocsByCreator = ctx.Documents
    .Include(d => d.MetaInfo) // Load all the metaInfo for each object
    .SelectMany(d => d.MetaInfo.Where(m => m.Name == "Author") // For each Author
        .Select(m => new { Doc = d, Creator = m })) // Create an object with the Author and the Document they authored.
    .ToList(); // Actualize the collection

I expected this to have all the Document / Author pairs, and have all the Document MetatInfo property filled.

That's not what happens, I get the Document objects, and the Authors just fine, but the Documents MetaInfo property ONLY has MetaInfo objects with Name == "Author"

If I move the where clause out of the select many it does the same, unless I move it to after the actualisation (which while here might not be a big deal, it is in the real application as it means we're getting a huge amount more data than we want to deal with.)

After playing with a bunch of different ways to do this I think it really looks like the issue is when you do a select(...new...) as well as the where and the include. Doing the select, or the Where clause after actualisation makes the data appear the way I expected it to.

I figured it was an issue with the MetaInfo property of Document being filtered, so I rewrote it as follows to test the theory and was surprised for find that this also gives the same (I think wrong) result.

ctx.Configuration.LazyLoadingEnabled = false;

var DocsByCreator = ctx.Meta
    .Where(m => m.Name == "Author")
    .Include(m => m.Document.MetaInfo) // Load all the metaInfo for Document
    .Select(m => new { Doc = m.Document, Creator = m })
    .ToList(); // Actualize the collection

Since we're not putting the where on the Document.MetaInfo property I expected this to bypass the problem, but strangely it doesn't the documents still only appear to have "Author" MetaInfo object.

I've created a simple test project and uploaded it to github with a bunch of test cases in, as far as I can tell they should all pass, bug only the ones with premature actualisation pass.

https://github.com/Robert-Laverick/EFIncludeIssue

Anyone got any theories? Am I abusing EF / SQL in some way I'm missing? Is there anything I can do differently to get the same organisation of results? Is this a bug in EF that's just been hidden from view by the LazyLoad being on by default, and it being a bit of an odd group type operation?

1 Answer 1

2

This is a limitation in EF in that Includes will be ignored if the scope of the entities returned is changed from where the include was introduced.

I couldn't find the reference to this for EF6, but it is documented for EF Core. (https://learn.microsoft.com/en-us/ef/core/querying/related-data) (see "ignore includes") I suspect it is a limit in place to stop EF's SQL generation from going completely AWOL in certain scenarios.

So while var docs = context.Documents.Include(d => d.Metas) would return the metas eager loaded against the document; As soon as you .SelectMany() you are changing what EF is supposed to return, so the Include statement is ignored.

If you want to return all documents, and include a property that is their author:

var DocsByCreator = ctx.Documents
    .Include(d => d.MetaInfo)
    .ToList() // Materialize the documents and their Metas.
    .SelectMany(d => d.MetaInfo.Where(m => m.Name == "Author") // For each Author
        .Select(m => new { Doc = d, Creator = m })) // Create an object with the Author and the Document they authored.
    .ToList(); // grab your collection of Doc and Author.

If you only want documents that have authors:

var DocsByCreator = ctx.Documents
    .Include(d => d.MetaInfo)
    .Where(d => d.MetaInfo.Any(m => m.Name == "Author")
    .ToList() // Materialize the documents and their Metas.
    .SelectMany(d => d.MetaInfo.Where(m => m.Name == "Author") // For each Author
        .Select(m => new { Doc = d, Creator = m })) // Create an object with the Author and the Document they authored.
    .ToList(); // grab your collection of Doc and Author.

This means you will want to be sure that all of your filtering logic is done above that first 'ToList() call. Alternatively you can consider resolving the Author meta after the query such as when view models are populated, or an unmapped "Author" property on Document that resolves it. Though I generally avoid unmapped properties because if their use slips into an EF query, you get a nasty error at runtime.

Edit: Based on the requirement to skip & take I would recommend utilizing view models to return data rather than returning entities. Using a view model you can instruct EF to return just the raw data you need, compose the view models with either simple filler code or utilizing Automapper which plays nicely with IQueryable and EF and can handle most deferred cases like this.

For example:

public class DocumentViewModel
{
    public int DocumentId { get; set; }
    public string Name { get; set; }
    public ICollection<MetaViewModel> Metas { get; set; } = new List<MetaViewModel>();
    [NotMapped]
    public string Author // This could be update to be a Meta, or specialized view model.
    {
        get { return Metas.SingleOrDefault(x => x.Name == "Author")?.Value; }
    }
}

public class MetaViewModel
{
    public int MetaId { get; set; }
    public string Name { get; set; }
    public string Value { get; set; }
}

Then the query:

var viewModels = context.Documents
    .Select(x => new DocumentViewModel
    {
        DocumentId = x.DocumentId,
        Name = x.Name,
        Metas = x.Metas.Select(m => new MetaViewModel
        {
            MetaId = m.MetaId,
            Name = m.Name,
            Value = m.Value
         }).ToList()
    }).Skip(pageNumber*pageSize)
    .Take(PageSize)
    .ToList();

The relationship of an "author" to a document is implied, not enforced, at the data level. This solution keeps the entity models "pure" to the data representation and lets the code handle transforming that implied relationship into exposing a document's author.

The .Select() population can be handled by Automapper using .ProjectTo<TViewModel>().

By returning view models rather than entities you can avoid issues like this where .Include() operations get invalidated, plus avoid issues due to the temptation of detaching and reattaching entities between different contexts, plus improve performance and resource usage by only selecting and transmitting the data needed, and avoiding lazy load serialization issues if you forget to disable lazy-load or unexpected #null data with it.

3
  • That does indeed cause "correct" results, but it's not a viable solution in a real case where there could be thousands of rows and we're trying to do paging via skip().take(). The best I've managed to come up with so far is to get just the IDs for the documents, and then have a second query that pulls the documents with MetaInfo via a ID in clause. Oct 11, 2018 at 11:53
  • 1
    Paging implies that these entities would be returned outside of their context scope. I'd recommend using view models as opposed to returning to avoid the issue, and potentially others. I've elaborated on that scenario in the answer above.
    – Steve Py
    Oct 11, 2018 at 21:41
  • Yeah, that's exactly what I was missing, and I'd never seen ProjectTo in automapper before either, for reference I've implemented a ProjectTo version in the repository. Oct 12, 2018 at 10:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.