Let me suggest a different way of looking at your problem.
Going by the description of your game, it seems like the user's main, perhaps only, "verb" (in game design terms) is to draw a line that divides the open area of the field into two sections. If either of these two sections is free of enemies, that section gets filled in; if neither section is free of enemies, the line remains but both sections remain open. There are no other conditions determining whether a section gets filled or not, right?
So the most efficient way to solve this problem, I would think, is simply to draw a continuous line, which may make corners but only moves in horizontal or vertical directions, from one of your enemies to every other enemy in turn. We'll call this line the "probe line". From here on, we're using the approach of Derek's suggested "Ray casting algorithm": We look at the number of times the "probe line" crosses the "border line", and if the number of crossings is ever odd, it means you have at least one enemy on each side of the line, and there's no filling.
Note, though, that there's a difference between the two lines coinciding and the two lines crossing. Picture a probe line that goes from the coordinates (0,10)
to (39,10)
, and a border line that goes down from (5,0)
to (5,10)
and then goes right to (13,10)
. If it goes down from there towards (13,39)
, the two lines are crossing; if instead it goes upwards toward (13,0)
, they're not.
After a lot of thought, I strongly suggest that you store the "border line", and construct the "probe line", in terms of line segments - rather than trying to determine from which cells are filled which line segments created them. That will make it much harder than it has to be.
Finally, one odd game design note to be aware of: unless you constrict the user's control so that he cannot bring the border line back to within one cell of itself, then a single border line drawn by a user might end up sectioning off the field into more than two sections - there could be sections created by the border line looping right back on itself. If you allow that, it could very drastically complicate the calculation of where to fill. Check the following diagram I made via Derek's fiddle (thank you, Derek!):
As you can see, one border line has actually created three sections: one on the upper side of the line, one below the line, and one formed by the line itself. I'm still thinking about how that would affect things algorithmically, for that to be possible.
EDIT: With a) time to think about the above creation-of-multiple-sections-by-loops, and b) the Simulation of Simplicity resource brought up by Derek, I think I can outline the simplest and most efficient algorithm that you're likely to get.
There's one subproblem to it which I'll leave to you, and that is determining what your new sections are after the player's actions have drawn a new line. I leave that to you because it's one that would have had to be solved before a solution to your original problem (how to tell if there are enemies within those sections) could have been called.
The solution, presented here as pseudocode, assumes you have the border of each section stored as line segments between coordinates.
Create a list of the sections.
Create a list of the enemies.
Continue as long as neither list is empty:
For each enemy in the enemy list:
Designate "Point A" as the coordinates of the enemy, PLUS 0.5 to both x and y.
For each section in the section list:
Designate "Point B" as the upper-left-most coordinate, PLUS 0.5 to both x and y.
Count how many of the section border segments cross a line between A and B.
If the answer is even:
remove this section from the section list
skip forward to the next enemy
If any sections remain in the list, they are free of enemies. Fill them in.
The addition of the 0.5 to the coordinates of the "probe line" are thanks to Derek's SoS resource; they eliminate the difficult case where the lines coincide rather than simply crossing or not crossing.