I have a list of usernames on Twitter whose profiles are public. I wish to get "all the tweets" they have posted from the day they formed their profile. I checked Twitter4J examples on GitHub.
According to the Twitter API documentation, only the 20 most recent tweets are returned. Is there anyway I could perform my task?
-
Also worth noting is that you can access upto 3200 tweets only using pagination and count - apiwiki.twitter.com/…– AnuragCommented Jun 1, 2010 at 3:39
-
1New link for the Things Every Developer Should Know : dev.twitter.com/overview/general/…– LE GALL BenoîtCommented Oct 20, 2014 at 13:25
3 Answers
To use Twitter4J to get all posts from a user you'll have to make your request over multiple pages..
Below code based of an example on GitHub
Twitter unauthenticatedTwitter = new TwitterFactory().getInstance();
//First param of Paging() is the page number, second is the number per page (this is capped around 200 I think.
Paging paging = new Paging(1, 100);
List<Status> statuses = unauthenticatedTwitter.getUserTimeline("google",paging);
Just loop and keep grabbing new pages until there are no new posts should work.
-
Tyler, Thanks for the response! However, when I run the same code as said above I get this error <code> "[Thu Jun 03 01:29:51 IST 2010]Using class twitter4j.internal.logging.StdOutLoggerFactory as logging factory. [Thu Jun 03 01:29:51 IST 2010]Use twitter4j.internal.http.HttpClientImpl as HttpClient implementation. Failed:Connection refused: connect" </code> I am able to get the public timeline using the getPublicTimeLine() function.– DexterCommented Jun 2, 2010 at 7:32
-
I tried the same using authentication twitter instance of my personal ID which is also public, btw. PS: I am sorry for posting the error in such unformatted format. I have tried all HTML tags including <code> and <blockquote> tags.– DexterCommented Jun 2, 2010 at 7:41
-
1You shouldn't have to authenticate to do this, are you not able to use the code above to get the public timeline of google or your own page? What are you using for paging? Could you try with just "username" as the only parameter to getUserTimeline()– TylerCommented Jun 2, 2010 at 13:37
-
Tyler, Thanks for the response. I am able to retrieve the tweets today! The API is very erratic in it's output as I'm testing the same since the past two days. In any case, The next problem though is that the code works ONLY for the paging parameters 1,100. I tried for different public usernames but in vain. The API limitation is 3200 {16,200} with a count of 200 per request as mentioned on the website. However, I am unable to avail the functionality.– DexterCommented Jun 3, 2010 at 6:28
Here is how to get ALL tweets for a user (or at least up to ~3200):
import java.util.*;
import twitter4j.*;
import twitter4j.conf.*;
public static void main(String[] a) {
ConfigurationBuilder cb = new ConfigurationBuilder();
cb.setOAuthConsumerKey("YOUR KEYS HERE");
cb.setOAuthConsumerSecret("YOUR KEYS HERE");
cb.setOAuthAccessToken("YOUR KEYS HERE");
cb.setOAuthAccessTokenSecret("YOUR KEYS HERE");
Twitter twitter = new TwitterFactory(cb.build()).getInstance();
int pageno = 1;
String user = "cnn";
List statuses = new ArrayList();
while (true) {
try {
int size = statuses.size();
Paging page = new Paging(pageno++, 100);
statuses.addAll(twitter.getUserTimeline(user, page));
if (statuses.size() == size)
break;
}
catch(TwitterException e) {
e.printStackTrace();
}
}
System.out.println("Total: "+statuses.size());
}
-
HI @rednoyz I tried the code but it always give me bad authentication error. 400:The request was invalid. An accompanying error message will explain why. This is the status code will be returned during version 1.0 rate limiting(dev.twitter.com/pages/rate-limiting). In API v1.1, a request without authentication is considered invalid and you will get this response. message - Bad Authentication data. code - 215– virenCommented Sep 24, 2015 at 4:56
-
-
You need to set your authentication keys -- fill in the values in ConfigurationBuilder– rednoyzCommented Feb 20, 2016 at 10:44
-
You'll only need the consumer keys for this, not the access tokens. little detail. Commented Apr 6, 2016 at 12:00
-
I get an error, 'Authentication credentials are missing', unless I use the access tokens as well.– rednoyzCommented Apr 10, 2016 at 2:44
If you read through the Twitter's Documentation, you can retrieve up to 200 tweets at a time if you specify "count=200" in your API request.
You can also use "page=x" to get different paginated results; you could keep doing this until you've retrieved every tweet the user has posted.
I'm not sure how your Java application would create this, but your requests would probably look like:
http://api.twitter.com/1/statuses/user_timeline.xml?id=SomeUsername&count=200&page=1
http://api.twitter.com/1/statuses/user_timeline.xml?id=SomeUsername&count=200&page=2
http://api.twitter.com/1/statuses/user_timeline.xml?id=SomeUsername&count=200&page=3
... etc.
Keep in mind that these requests are rate-limited so you'd need to be careful not to exceed the limit.