0

I'm splitting a list of (in this example) approx. 190000 items into chunks of 5000 items.

so instead of: List<Object>, count 190000

it becomes: List<List<Object>>, Count 28(Count 5000))

I do this with the following code:

public static IEnumerable<List<Object>> Split(this IEnumerable<Object> sourceList, int chunkSize)
{            
    int numberOfLists = (sourceList.Count() / chunkSize) + 1;
    List<List<Object>> result = new List<List<object>>();

    for (int i = 0; i < numberOfLists; i++)
    {
        List<Object> subList = new List<Object>();
        subList = sourceList.Skip(i * chunkSize).Take(chunkSize).ToList();
        result.Add(subList);
    }

    return result;
}

I call this method (which resides in a helper class) like follows;

var chunkList = (IEnumerable<List<MyObjectClass>>)MyHelper.Split(myObjectList, 5000);

In the above line I explicitly cast the list, that fails in an InvalidCastException. When i use the as operator like follows;

var chunkList = MyHelper.Split(myObjectList, 5000) as IEnumerable<List<MyObjectClass>>;

the result is null.

I expected I could use

List<List<MyObjectClass>> chunkList = MyHelper.Split(myObjectList, 5000) as List<List<MyObjectClass>>

I would like to keep the splitter method as generic as possible. The question is how I can cast the return value correctly. Can someone point out to me how to do this?

Thanks in advance.

3
  • Please add the language tag. Dec 3, 2013 at 15:19
  • I've changed your code to use var, to make lines shorter. Hope you're ok with that. Dec 3, 2013 at 19:40
  • 1
    Off by one error: if sourceList.Count() % chunkSize == 0 you get a spurious empty sub-list at the end.
    – Jon
    Dec 3, 2013 at 19:45

3 Answers 3

2

As others have stated, the problem is that you are attempting to use the type parameter or List<T> in a variant manner, which is not possible. Instead of this you need to make the splitting method generic so that it has its own type parameter that matches that of the list.

That said, you can turn the method into an iterator block so that it only produces the sub-lists on demand:

public static IEnumerable<List<T>> Partition<T>(this IEnumerable<T> source,
                                                int chunkSize)
{
    while (source.Any())
    {
        yield return source.Take(chunkSize).ToList();
        source = source.Skip(chunkSize);
    }
}

This would be used as

var chunkList = sourceList.Partition(5000);

Note that the above version is free of the off-by-one error that your original code and the solutions based on it all share.

If you don't care about lazy evaluation there is also the possibility of using this trick with GroupBy to do the partitioning:

int i = 0;
var chunkList = sourceList
    .GroupBy(o => i++ / chunkSize) // group into partitions
    .Select(Enumerable.ToList)     // transform each partition into a List
    .ToList()                      // force evaluation of query right now
1
  • Hi Jon, thanks a lot! Works like a charm. [11:34:10.674]: 138189 items retrieved in 00:00:18.2368460 [11:34:10.689]: list partitioned in: 00:00:00.0025796
    – Obelix
    Dec 9, 2013 at 10:36
2

you can use type parameter instead of object

public static IEnumerable<List<T>> Split<T>(this IEnumerable<T> sourceList, int chunkSize)   {
    int numberOfLists = (sourceList.Count() / chunkSize) + 1;
    List<List<T>> result = new List<List<T>>();

    for (int i = 0; i < numberOfLists; i++)
    {
        result.Add(sourceList.Skip(i * chunkSize).Take(chunkSize));
    }

    return result;
}

so for use

IEnumerable<List<MyObjectClass>> chunkList = myObjectClassList.Split(5000);
0
1

Instead of taking IEnumerable<Object> as parameter and returning IEnumerable<List<Object>> make your method generic:

public static IEnumerable<List<T>> Split<T>(this IEnumerable<T> sourceList, int chunkSize)
{
    int numberOfLists = (sourceList.Count() / chunkSize) + 1;
    var result = new List<List<T>>(numberOfLists);

    for (int i = 0; i < numberOfLists; i++)
    {
        result.Add(sourceList.Skip(i * chunkSize).Take(chunkSize).ToList());
    }

    return result;
}

You can also use Partition method from moreLINQ library. It is more efficient then your solution, because using Skip().Take() causes unnecessary iteration over the same set of elements every time.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.