49

I tried this earlier today:

public interface IFoo
{
    IEnumerable<int> GetItems_A( ref int somethingElse );
    IEnumerable<int> GetItems_B( ref int somethingElse );
}


public class Bar : IFoo
{
    public IEnumerable<int> GetItems_A( ref int somethingElse )
    {
        // Ok...
    }

    public IEnumerable<int> GetItems_B( ref int somethingElse )
    {
        yield return 7; // CS1623: Iterators cannot have ref or out parameters            

    }
}

What's the rationale behind this?

3

5 Answers 5

62

C# iterators are state machines internally. Every time you yield return something, the place where you left off should be saved along with the state of local variables so that you could get back and continue from there.

To hold this state, C# compiler creates a class to hold local variables and the place it should continue from. It's not possible to have a ref or out value as a field in a class. Consequently, if you were allowed to declare a parameter as ref or out, there would be no way to keep the complete snapshot of the function at the time we had left off.

EDIT: Technically, not all methods that return IEnumerable<T> are considered iterators. Just those that use yield to produce a sequence directly are considered iterators. Therefore, while the splitting the iterator into two methods is a nice and common workaround, it doesn't contradict with what I just said. The outer method (that doesn't use yield directly) is not considered an iterator.

5
  • 3
    "It's not possible to have a ref or out value as a field in a class." --The compiler could easily implement ref parameters to iterators by allocating a single element array in the caller, putting the argument into that, and passing the array to the iterator, and having the iterator operate on array[0]. This would be a very small amount of work on the part of the compiler compared to turning the iterator into a state machine.
    – Jim Balter
    Jun 12, 2014 at 4:07
  • @JimBalter That would be true if the compiler controlled every piece of the code that ran. Unfortunately, that plan would require a different API signature to be generated in the binary--i.e. callers from the outside world passing in "ref" variables would not be able to see them changing. Jun 12, 2014 at 19:02
  • 1
    Um, callers from the outside world can't currently pass in ref parameters at all ... no one can. So there would be no harm in specifying that ref T parameters to iterators are actually represented as T[]. In any case, using an array or a class instance as a box is a simple way to get around the limitation ... simpler than the other solutions offered here.
    – Jim Balter
    Jun 12, 2014 at 23:47
  • @JimBalter they will be if you allow ref T, which is the whole point. Jun 13, 2014 at 2:13
  • 1
    Um, if C# generates a signature with T[], that doesn't outside callers to invoke it with ref T, just as they can't now. It always amuses me when people talk about "the whole point" when that totally ignores what the other person said. I have no interest in further discussion when I encounter that ... over and out.
    – Jim Balter
    Jun 13, 2014 at 4:17
25

If you want to return both an iterator and an int from your method, a workaround is this:

public class Bar : IFoo
{
    public IEnumerable<int> GetItems( ref int somethingElse )
    {
        somethingElse = 42;
        return GetItemsCore();
    }

    private IEnumerable<int> GetItemsCore();
    {
        yield return 7;
    }
}

You should note that none of the code inside an iterator method (i.e. basically a method that contains yield return or yield break) is executed until the MoveNext() method in the Enumerator is called. So if you were able to use out or ref in your iterator method, you would get surprising behavior like this:

// This will not compile:
public IEnumerable<int> GetItems( ref int somethingElse )
{
    somethingElse = 42;
    yield return 7;
}

// ...
int somethingElse = 0;
IEnumerable<int> items = GetItems( ref somethingElse );
// at this point somethingElse would still be 0
items.GetEnumerator().MoveNext();
// but now the assignment would be executed and somethingElse would be 42

This is a common pitfall, a related issue is this:

public IEnumerable<int> GetItems( object mayNotBeNull ){
  if( mayNotBeNull == null )
    throw new NullPointerException();
  yield return 7;
}

// ...
IEnumerable<int> items = GetItems( null ); // <- This does not throw
items.GetEnumerators().MoveNext();                    // <- But this does

So a good pattern is to separate iterator methods into two parts: one to execute immediately and one that contains the code that should be lazily executed.

public IEnumerable<int> GetItems( object mayNotBeNull ){
  if( mayNotBeNull == null )
    throw new NullPointerException();
  // other quick checks
  return GetItemsCore( mayNotBeNull );
}

private IEnumerable<int> GetItemsCore( object mayNotBeNull ){
  SlowRunningMethod();
  CallToDatabase();
  // etc
  yield return 7;
}    
// ...
IEnumerable<int> items = GetItems( null ); // <- Now this will throw

EDIT: If you really want the behavior where moving the iterator would modify the ref-parameter, you could do something like this:

public static IEnumerable<int> GetItems( Action<int> setter, Func<int> getter )
{
    setter(42);
    yield return 7;
}

//...

int local = 0;
IEnumerable<int> items = GetItems((x)=>{local = x;}, ()=>local);
Console.WriteLine(local); // 0
items.GetEnumerator().MoveNext();
Console.WriteLine(local); // 42
2
  • 4
    Re: the edit with getter/setter lambdas, this as a way of simulating pointers to value types (although without address manipulation, of course), more here: incrediblejourneysintotheknown.blogspot.com/2008/05/… Jun 17, 2009 at 8:28
  • Simpler than passing a getter and setter is to put the "ref" parameter in a box -- a class instance if you have multiple "ref" parameters, or simply a single element array if you just have one.
    – Jim Balter
    Jun 12, 2014 at 3:52
5

At a highish level, A ref variable can point to many locations including to value types that are on the stack. The time at which the iterator is initially created by calling the iterator method and when the ref variable would be assigned are two very different times. It is not possible to guarantee that the variable which originally was passed by reference is still around when the iterator actually executes. Hence it is not allowed (or verifiable)

5

Others have explained why your iterator can't have a ref parameter. Here's a simple alternative:

public interface IFoo
{
    IEnumerable<int> GetItems( int[] box );
    ...
}

public class Bar : IFoo
{
    public IEnumerable<int> GetItems( int[] box )
    {
        int value = box[0];
        // use and change value and yield to your heart's content
        box[0] = value;
    }
}

If you have several items to pass in and out, define a class to hold them.

2

I've gotten around this problem using functions, when the value that I need to return is derived from the iterated items:

// One of the problems with Enumerable.Count() is
// that it is a 'terminator', meaning that it will
// execute the expression it is given, and discard
// the resulting sequence. To count the number of
// items in a sequence without discarding it, we 
// can use this variant that takes an Action<int>
// (or Action<long>), invokes it and passes it the
// number of items that were yielded.
//
// Example: This example allows us to find out
//          how many items were in the original
//          source sequence 'items', as well as
//          the number of items consumed by the
//          call to Sum(), without causing any 
//          LINQ expressions involved to execute
//          multiple times.
// 
//   int start = 0;    // the number of items from the original source
//   int finished = 0; // the number of items in the resulting sequence
//
//   IEnumerable<KeyValuePair<string, double>> items = // assumed to be an iterator
//
//   var result = items.Count( i => start = i )
//                   .Where( p => p.Key = "Banana" )
//                      .Select( p => p.Value )
//                         .Count( i => finished = i )
//                            .Sum();
//
//   // by getting the count of items operated 
//   // on by Sum(), we can calculate an average:
// 
//   double average = result / (double) finished; 
//
//   Console.WriteLine( "started with {0} items", start );
//   Console.WriteLine( "finished with {0} items", finished );
//

public static IEnumerable<T> Count<T>( 
    this IEnumerable<T> source, 
    Action<int> receiver )
{
  int i = 0;
  foreach( T item in source )
  {
    yield return item;
    ++i ;
  }
  receiver( i );
}

public static IEnumerable<T> Count<T>( 
    this IEnumerable<T> source, 
    Action<long> receiver )
{
  long i = 0;
  foreach( T item in source )
  {
    yield return item;
    ++i ;
  }
  receiver( i );
}

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.