5

I'm trying to create a collection of FTP web requests to download a collection of files.

Was working correctly doing this in a single thread but am trying to do with multiple threads now but am getting a timeout exception. I think I'm missing something pretty simple but cannot seem to work it out

Here is code:

internal static void DownloadLogFiles(IEnumerable<string> ftpFileNames, string localLogsFolder)
{
    BotFinder.DeleteAllFilesFromDirectory(localLogsFolder);

    var ftpWebRequests = new Collection<FtpWebRequest>();

    // Create web request for each log filename
    foreach (var ftpWebRequest in ftpFileNames.Select(filename => (FtpWebRequest) WebRequest.Create(filename)))
    {
        ftpWebRequest.Credentials = new NetworkCredential(BotFinderSettings.FtpUserId, BotFinderSettings.FtpPassword);
        ftpWebRequest.KeepAlive = false;
        ftpWebRequest.UseBinary = true;
        ftpWebRequest.CachePolicy = NoCachePolicy;
        ftpWebRequest.Method = WebRequestMethods.Ftp.DownloadFile;
        ftpWebRequests.Add(ftpWebRequest);
    }

    var threadDoneEvents = new ManualResetEvent[ftpWebRequests.Count];

    for (var x = 0; x < ftpWebRequests.Count; x++)
    {
        var ftpWebRequest = ftpWebRequests[x];
        threadDoneEvents[x] = new ManualResetEvent(false);
        var threadedFtpDownloader = new ThreadedFtpDownloader(ftpWebRequest, threadDoneEvents[x]);
        ThreadPool.QueueUserWorkItem(threadedFtpDownloader.PerformFtpRequest, localLogsFolder);               
    }

    WaitHandle.WaitAll(threadDoneEvents);
}

class ThreadedFtpDownloader
{
    private ManualResetEvent threadDoneEvent;
    private readonly FtpWebRequest ftpWebRequest;

    /// <summary>
    /// 
    /// </summary>
    public ThreadedFtpDownloader(FtpWebRequest ftpWebRequest, ManualResetEvent threadDoneEvent)
    {
        this.threadDoneEvent = threadDoneEvent;
        this.ftpWebRequest = ftpWebRequest;
    }

    /// <summary>
    /// 
    /// </summary>
    /// <param name="localLogsFolder">
    /// 
    /// </param>
    internal void PerformFtpRequest(object localLogsFolder)
    {
        try
        {
            // TIMEOUT IS HAPPENING ON LINE BELOW
            using (var response = ftpWebRequest.GetResponse())
            {
                using (var responseStream = response.GetResponseStream())
                {
                    const int length = 1024*10;
                    var buffer = new Byte[length];
                    var bytesRead = responseStream.Read(buffer, 0, length);

                    var logFileToCreate = string.Format("{0}{1}{2}", localLogsFolder,
                                        ftpWebRequest.RequestUri.Segments[3].Replace("/", "-"),
                                        ftpWebRequest.RequestUri.Segments[4]);

                    using (var writeStream = new FileStream(logFileToCreate, FileMode.OpenOrCreate))
                    {
                        while (bytesRead > 0)
                        {
                            writeStream.Write(buffer, 0, bytesRead);
                            bytesRead = responseStream.Read(buffer, 0, length);
                        }
                    }
                }
            }

            threadDoneEvent.Set();
        }
        catch (Exception exception)
        {
            BotFinder.HandleExceptionAndExit(exception);
        }
    }
}

It seems to be downloading the first two files (using two threads I'm assuming) but then timeout seems to occur when these complete and application tries to move onto next file.

I can confirm that the FTPWebRequest which is timing out is valid and the file exists, I think I may have an open connection or something.


Was going to post a comment but probably easier to read in an answer:

Firstly, if I set the ftpRequest.Timout property to Timeout.Infinite, the timeout issue disappears however having an infinite timeout is probably not best practice. So I'd prefer to go about solving this another way...

Debugging the code, I can see that when it gets to:

ThreadPool.QueueUserWorkItem(threadedFtpDownloader.PerformFtpRequest, localLogsFolder);

It enters into PerformFtpRequest method for each FTP web request and calls the ftpWebRequest.GetResponse() but then only progresses further for the first two requests. The rest of the requests stay active but don't go any further until the first two finish. So this basically means they are left open while waiting for other requests to complete before starting.

I think the solution to this problem would either be allowing all the requests to execute at once (ConnectionLimit property is having no effect here) or to prevent execution from calling GetResponse until it's actually ready to use the response.

Any good ideas on best way to solve this? At the moment all I can seem to think of are hacky solutions which I'd like to avoid :)

Thanks!

1 Answer 1

3

You should get the ServicePoint for the request and set the ConnectionLimit

ServicePoint sp = ftpRequest.ServicePoint;
sp.ConnectionLimit = 10;

The default ConnectionLimit is 2 -- that's why you're seeing that behavior.

UPDATE: See this answer for a more thorough explanation:

How to improve the Performance of FtpWebRequest?

2
  • Thanks mate. I tried adding that code within foreach loop which was creating FtpWebRequests but getting same problem. Is that where I should be setting the connection limit? Dec 12, 2010 at 23:08
  • Added a link to a question/answer with more details. ServicePoint is one of those things you never heard of until you bang your head a few times. Dec 13, 2010 at 0:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.