I'm performing many requests through a collection of URLs so I can check which ones could retrieve a PDF file.
In order to do this, I create a WebRequest
with HEAD
method and checking afterwards the response I get.
When I perform the webRequest.GetResponse()
(sync) everything seems to work fine and every request is being "fired".
On the other hand, when I perform the await webRequest.GetResponseAsync()
for the third time it simply doesn't reach the end of the method.
Since it's actually into a try-catch
in order to handle possible unreachable hosts, it simply ignores that link.
Async
version:
private async Task<IEnumerable<string>> GetLinksContainsDownloadablePdfAsync(IEnumerable<string> linksInMail)
{
var downloadableLinks = new List<string>();
foreach (var link in linksInMail)
{
var headRequest = WebRequest.Create(link);
headRequest.Method = "HEAD";
try
{
var responseTest = await headRequest.GetResponseAsync();
if (responseTest.Headers["Content-Type"].Contains("application/pdf"))
downloadableLinks.Add(link);
}
catch (WebException)
{
//If it's not accesible, just ignore it
}
}
return downloadableLinks;
}
Sync
version:
private IEnumerable<string> GetLinksContainsDownloadablePdf(IEnumerable<string> linksInMail)
{
var downloadableLinks = new List<string>();
foreach (var link in linksInMail)
{
var headRequest = WebRequest.Create(link);
headRequest.Method = "HEAD";
try
{
var responseTest = headRequest.GetResponse();
if (responseTest.Headers["Content-Type"].Contains("application/pdf"))
downloadableLinks.Add(link);
}
catch (WebException)
{
//If it's not accesible, just ignore it
}
}
return downloadableLinks;
}
Could anyone help throwing out some light here?
I'm not getting any Exception
actually and the debugger isn't even reaching the return downloadableLinks
line.
Note this is going to run in a server, so I'm specially interested to be multi-thread friendly.
EDIT: This method is not the only one which is async
and I'm calling it from other async methods
, so I'm supposed to be correctly handling the Task
itself.
This is the method where I'm calling the GetLinksContainsDownloadablePdfAsync
method
protected override async Task<IEnumerable<StoredFile>> ExtractPdfAsync(EmailMessageItem message, DocumentInfo documentInfo)
{
var document = new HtmlDocument();
document.LoadHtml(message.Body.HtmlBody);
var validLinks = await new LinkHelper(document).GetValidLinksInHtmlBodyAsync();
var pdfFiles = await DownloadPdfFromLinksAsync(documentInfo, validLinks);
return pdfFiles;
}
EDIT 2: In order to provide more information to perform tests, these are links found on e-mails, and the e-mail I'm having issues with is an e-mail from a daily online newspaper. I've debugged every single call and noticed it's producing 3 calls since it's diverted with some redirects (3xx). Here is a link to pastebin with almost every link (removed some due to privacy-subscription issues)
return downloadableLinks;
. Run to it. What is the value ofdownloadableLinks.Count
?HttpClient
instead?responseTest.Headers
or calling code...