0

I have the following join statement:

  var query = from holding in dHoldList
              join client in clientList on
                new { holding.ClientNo }
                equals new { client.ClientNo } into clj
              from cl in clj.DefaultIfEmpty()
              join matter in matterList on 
                new { holding.ClientNo } 
                equals new { matter.ClientNo } into mlj
              from ml in mlj.DefaultIfEmpty()
              join stk in stockList on
                new { holding.Sedol }
                equals new { stk.Sedol } into slj
              from sl in slj.DefaultIfEmpty()
              select
                new GeneralHoldingsReport()
                {
                  ClientNo = holding.ClientNo,
                  Depot = holding.Depot,
                  HoldingSedol = holding.Sedol,
                  Value = holding.ValueOfStock,
                  NoOfUnits = holding.QuantityHeld,
                  ClientName = (cl == null ? null : cl.ClientName),
                  CountryOfResidence = (cl == null ? null : cl.CountryOfResidence),
                  BG = (cl == null ? null : cl.BusinessGetter),
                  ClientStockValue = (ml == null ? 0 : ml.FullValueOfPortfolio),
                  StockName = (sl == null ? null : sl.R1.Trim() + " " + sl.R2.Trim())
                };


  var reportList = query.ToList();

However when it runs I am getting an out of memory exception error.

I require the dHoldList to be the main table, with all other tables being left joins into it (i.e. if data is matched in other tables relating to each record in dHoldList then return data, if not just blank.) I believe I am doing this correctly, but obviously not.

Each of these lists contains approx 300k lines, apart from the client which is only 30k, so that may be causing some of the issues here.

1
  • 1
    where are you populating the lists from, database?
    – PMC
    Aug 21, 2012 at 8:03

1 Answer 1

2
  1. You are instantiating a LOT of useless anonymous types in your joins. You can write them as follows:

    join client in clientList on holding.ClientNo equals client.ClientNo into clj 
    

    Maybe this already solves your memory problem?
    This does only apply to LINQ to Objects. If your query is translated into SQL as is the case with EF or LINQ to SQL these anonymous types will not be created.

  2. Think about restructuring the query to make it more readable:

    var query = 
        from holding in dHoldList 
        join client in clientList on holding.ClientNo equals client.ClientNo into clj 
        join matter in matterList on holding.ClientNo equals matter.ClientNo into mlj 
        join stk in stockList on holding.Sedol equals stk.Sedol into slj 
    
        from cl in clj.DefaultIfEmpty() 
        from ml in mlj.DefaultIfEmpty() 
        from sl in slj.DefaultIfEmpty() 
        select new GeneralHoldingsReport() 
        { 
            ClientNo = holding.ClientNo, 
            Depot = holding.Depot, 
            HoldingSedol = holding.Sedol, 
            Value = holding.ValueOfStock, 
            NoOfUnits = holding.QuantityHeld, 
            ClientName = (cl == null ? null : cl.ClientName), 
            CountryOfResidence = (cl == null ? null : cl.CountryOfResidence), 
            BG = (cl == null ? null : cl.BusinessGetter), 
            ClientStockValue = (ml == null ? 0 : ml.FullValueOfPortfolio), 
            StockName = (sl == null ? null : sl.R1.Trim() + " " + sl.R2.Trim()) 
        }; 
    

    This will not have any impact on memory footprint but it does have an impact on readability and maintainability.


If I would write code to achieve your goal, especially with that many objects involved, I wouldn't use a join. I would use hash tables. This would be dramatically faster.

Something like this (untested):

var clients = clientList.ToDictionary(x => x.ClientNo);
var matters = matterList.ToDictionary(x => x.ClientNo);
var stocks = stockList.ToDictionary(x => x.Sedol);

var reportList = new List<GeneralHoldingsReport>(dHoldList.Count);
Client client;
Matter matter;
Stock stock;    

foreach(var holding in dHoldList)
{
    if(!clients.TryGetValue(holding.ClientNo, out client))
        client = null;
    if(!matters.TryGetValue(holding.ClientNo, out matter))
        matter = null;
    if(!stocks.TryGetValue(holding.Sedol, out stock))
        stock = null;
    reportList.Add(new GeneralHoldingsReport()     
                   {     
                       ClientNo = holding.ClientNo,     
                       Depot = holding.Depot,     
                       HoldingSedol = holding.Sedol,     
                       Value = holding.ValueOfStock,     
                       NoOfUnits = holding.QuantityHeld,     
                       ClientName = (client == null ? null :
                                     client.ClientName),     
                       CountryOfResidence = (client == null ? null : 
                                             client.CountryOfResidence),     
                       BG = (client == null ? null : 
                             client.BusinessGetter),     
                       ClientStockValue = (matter == null ? 0 : 
                                           matter.FullValueOfPortfolio),     
                       StockName = (stock == null ? null : 
                                    stock.R1.Trim() + " " 
                                    + stock.R2.Trim())     
                   });
} 

The usage of ToDictionary will result in a crash if there is more than one client or matter per client number and more than one stock per sedol.

7
  • @DavidJohnson: Is this LINQ to objects? Aug 21, 2012 at 9:41
  • Sorry, im not sure what you mean. The lists that I am joining are generic lists if that helps? Aug 21, 2012 at 10:20
  • @DavidJohnson: So, there is no database involved at that point? What is the exact type of the lists? List<T>? IEnumerable<T>? IQueryable<T>? Aug 21, 2012 at 10:56
  • No theres no database involved, it is four List<T>'s...dHoldList, clientList, matterList and stockList... Aug 21, 2012 at 11:17
  • @DavidJohnson: Please see update for an alternate way. Maybe this solves your problem? Aug 21, 2012 at 11:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.