5

I am trying to fetch tweets from Twitter timeline using the max_id field present in next_results as demonstrated here.(Twython search API with next_results)

For most of the queries I get the desired number(500+) of tweets but for some I get less than 200 tweets.

When looking into what is returned in 'next_results I found that after returning just 100 tweets, the response has next_results field missing. There cannot be just 200 tweets for the queried topic as I am searching for this topic when it is trending and on Twitter I can see more than 200 tweets.

Is anyone else also facing this issue. Is there any workaround ?

Below is the output where in the second response the 'next_results' field is missing.

TEST_PROGRAM >> Calling twitter to get tweets for  #TheWorstLieEver

{u'count': 100, u'completed_in': 0.066, u'max_id_str': u'432942833725497345', u'since_id_str': u'0', **u'next_results': u'?max_id=432540545630494719&q=%23TheWorstLieEver&count=100&include
_entities=1&result_type=mixed'**, u'refresh_url': u'?since_id=432942833725497345&q=%23TheWorstLieEver&result_type=mixed&include_entities=1', u'since_id': 0, u'query': u'%23TheWorstLieEver
', u'max_id': 432942833725497345L}

{u'count': 100, u'completed_in': 0.111, u'max_id_str': u'432540545630494719', u'since_id_str': u'0', u'refresh_url': u'?since_id=432540545630494719&q=%23TheWorstLieEver&result_type=mixe
d&include_entities=1', u'since_id': 0, u'query': u'%23TheWorstLieEver', u'max_id': 432540545630494719L}
5
  • Hey, did you find a solution? I've got the exact same problem Feb 13, 2014 at 14:26
  • @ColinBernet It seems implementing own max_id navigation fetches more results. What it means is, when we get a response with no next_results we need to manually find the tweet with smallest tweet id and submit it as max_id in the next query. You can find its reference here:link.I tried this but it didnt fetch me any more tweets !!
    – kundan
    Feb 14, 2014 at 5:45
  • @ColinBernet I got the answer. If you are looking for volume, you need to use Streaming API instead. Twitter search API focuses on relevance of query and hence might not return all tweets. [Twitter documentation] (dev.twitter.com/docs/using-search) says Before getting involved, it's important to know that the Search API is focused on relevance and not completeness. This means that some Tweets and users may be missing from search results. If you want to match for completeness you should consider using a Streaming API instead.
    – kundan
    Feb 14, 2014 at 17:49
  • @kundan: Since you found a solution, you should add it as an answer to your question and accept it.
    – Aaron Hill
    Mar 13, 2014 at 18:50
  • @Aaron Thanks for pointing it. I will do it.
    – kundan
    Mar 16, 2014 at 8:15

1 Answer 1

2

I got the answer. If you are looking for volume, you need to use Streaming API instead. Twitter search API focuses on relevance of query and hence might not return all tweets. Twitter documentation says Before getting involved, it's important to know that the Search API is focused on relevance and not completeness. This means that some Tweets and users may be missing from search results. If you want to match for completeness you should consider using a Streaming API instead.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.