0

I'm fetching messages from Office365 using Microsoft's Graph API Java client. The queries span a whole year. They're limited to 50 results per page and progress using the returned next page URL. It's done periodically every 5 minutes, but no longer than 2 minutes per job (which iterates over the next url).

Once in a while I get 503 Service Unavailable / 504 Gateway Timeout. Once this happens, the request can't progress and will keep hitting those errors.

According to Microsoft's documentation this should be treated as too many requests, and back off with a delay. There's no Retry-After header when this happens. I've noticed that making the time frame smaller and restarting the query can help sometimes. Also I've seen this raised on Stackoverflow but with no solutions.

I'm wondering is querying for a year too much, even though there's paging? Any ideas for solutions other than backing off, and for how long?

Thanks

Example query which fails:

https://graph.microsoft.com/v1.0/me/messages?$filter=IsDraft+eq+false+and+ReceivedDateTime+ge+2019-03-28T20%3a08%3a51.929Z+and+ReceivedDateTime+lt+2020-02-20T19%3a48%3a37Z&$orderby=ReceivedDateTime+desc&$expand=SingleValueExtendedProperties(%24filter%3did+eq+%27String+0x7D%27)&$select=conversationId%2cchangeKey%2csentDateTime%2creceivedDateTime%2cisRead%2chasAttachments%2cinternetMessageHeaders%2csender%2cfrom%2ctoRecipients%2cccRecipients%2cbccRecipients%2csubject%2cinternetMessageId%2cparentFolderId&$top=50&$skip=51

Edit: Waiting for an hour before sending more requests doesn't seem to help. It looks like the problem is with the large request.

Edit #2: Something which helped decrease the number of errors was to remove the filter=IsDraft+eq+false part from the query string, and filter the drafts on the client side. I still get occasional 503 errors, but much less.

2 Answers 2

1

There is specific guidance on Mailbox limits here https://learn.microsoft.com/en-us/graph/throttling#outlook-service-limits They are based on the individual user mailbox. So if you are calling more than 10,000 API requests in a 10 minute period or calling it in more than 4 concurrent processes you may be hitting this.

You are not getting a 429 response though, which would assume that you are not being throttled by the service.

So I understand your problem more, if you are running this every 5 mins, why are you having to go back a whole year? That is a much more complex query by adding that data range. You could use the Delta queries to just get the changes since the last 5 mins for the Drafts folder https://learn.microsoft.com/en-us/graph/delta-query-overview

2
  • Thanks for the reply. I'll look into delta queries more thoroughly. The intention is to perform a one time import of emails going a year back. The URL I posted is the one returned as nextLink from a previous request, and we're iterating over those links, not restarting the query each time. isDraft is set to false, we're fetching regular emails. Indeed it doesn't look like throttling but more like some problem or delay the server has in paging the data for the response. Shouldn't it be able to handle it? I should mention it usually works smoothly, but for some mailboxes I get this error.
    – Udi S.
    Apr 21, 2020 at 16:58
  • One more clarification - it's not running the same request every 5 minutes, the idea is that every x minutes it pulls a few pages, then stops until the next "job", which will continue where the previous one left off. Advancing through the pages is done by taking the nextLink from the response.
    – Udi S.
    Apr 22, 2020 at 7:08
0

Have you tried to change the order of filter statements? In your case, I would filter by the date first and then filter out all mails, that are drafts:

https://graph.microsoft.com/v1.0/me/messages?$filter=ReceivedDateTime+ge+2019-03-28T20%3a08%3a51.929Z+and+ReceivedDateTime+lt+2020-02-20T19%3a48%3a37Z+and+IsDraft+eq+false&$orderby=ReceivedDateTime+desc&$expand=SingleValueExtendedProperties(%24filter%3did+eq+%27String+0x7D%27)&$select=conversationId%2cchangeKey%2csentDateTime%2creceivedDateTime%2cisRead%2chasAttachments%2cinternetMessageHeaders%2csender%2cfrom%2ctoRecipients%2cccRecipients%2cbccRecipients%2csubject%2cinternetMessageId%2cparentFolderId&$top=50&$skip=51

I have had some throttling issues with MS Graph, which I was able to fix by changing the order of the filter statements. In a ten year old mailbox, there are probably more mails, that are not drafts, than there are mails from the last two years. I always try to apply the narrower filter first.

4
  • Thanks for the answer! A very interesting idea. I tried it and the same error still happens for the mailboxes which experienced this problem, probably those with more emails to iterate over when paging the response.
    – Udi S.
    Apr 2, 2020 at 15:04
  • Is it possible to run the app at night? The throttling limits depend on server load and up to 5 times higher between 8pm and 6am, since there is not a lot of user interaction. Also consider the note in this part of the documentation learn.microsoft.com/en-us/graph/throttling about implementing an exponential backoff retry policy, if the Retry-After header is missing. Also it may help, if you provided some more information about, what your goal is. Maybe there is a better way to implement it, such as using delta queries.
    – Marc
    Apr 2, 2020 at 16:51
  • Sorry for the late reply, and appreciate the answers again. It's actually already running during off hours. I did try backing off for an hour with no change, it seems to be something else other than throttling. The end goal is to import the entire mail history for a period of one year.
    – Udi S.
    Apr 21, 2020 at 16:43
  • We have a similar situation where this simple query GET /me/messages?$filter=(lastModifiedDateTime ge 2020-07-23T13:17:29Z) just never succeeds! Only bringing the timestamp closer to now resolves the problem but we want to be able to get all messages since a certain date.
    – Alexey
    Aug 10, 2020 at 20:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.