24

I am using Tweepy API for extracting Twitter feeds. I want to extract all Twitter feeds of a specific language only. The language filter works only if track filter is provided. The following code returns 406 error:

l = StdOutListener()
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
stream = Stream(auth, l)
stream.filter(languages=["en"])

How can I extract all the tweets from certain language using Tweepy?

1
  • Have you solved this issue. I also want to extract specific language tweet like japanese.
    – tktktk0711
    Sep 22, 2017 at 15:03

7 Answers 7

28

You can't (without special access). Streaming all the tweets (unfiltered) requires a connection to the firehose, which is granted only in specific use cases by Twitter. Honestly, the firehose isn't really necessary--proper use of track can get you more tweets than you know what to do with.

Try using something like this:

stream.filter(languages=["en"], track=["a", "the", "i", "you", "u"]) # etc

Filtering by words like that will get you many, many tweets. If you want real data for the most-used words, check out this article from Time: The 500 Most Frequently Used Words on Twitter. You can use up to 400 keywords, but that will likely approach the 1% limit of tweets at a given time interval. If your track parameter matches 60% of all tweets at a given time, you will still only get 1% (which is a LOT of tweets).

1
  • Thanks!! I was typing stream.filter(lang=["en"], track=["something"])
    – Krishh
    Apr 5, 2016 at 23:15
8

Try lang='en' param in Cursor() e.g.

tweepy.Cursor(.. lang='en')

4

Other than getting filtered tweets directly, you can filter it after getting all tweets of different languages by:

tweets = api.search("python")
for tweet in tweets:
   if tweet.lang == "en":
      print(tweet.text)
      #Do the stuff here

Hope it helps.

2

You can see the arguments for the track method in the github code https://github.com/tweepy/tweepy/blob/master/tweepy/streaming.py

Put languages in a array of ISO_639-1_codes.

They are:

filter(self, follow=None, track=None, is_async=False, locations=None,
               stall_warnings=False, languages=None, encoding='utf8', filter_level=None):

So to track by languages just put:

class Listener(StreamListener):

    def on_data(self, data):
        j = json.loads(data)
        t = {
          'screenName' : j['user']['screen_name'],
          'text:': j['text']
          }
        print(t)
        return(True)

    def on_status(self, status):
        print(status.text)


auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)

stream = Stream(auth=auth, listener=Listener(),wait_on_rate_limit=True,wait_on_rate_limit_notify=True)

stream.filter(track=['Trump'],languages=["en","fr","es"])
1

Tweepy search allows to fetch tweets for specific language. You can use ISO 639-1 code to specify the value for language parameter. Following code will fetch tweets with full text in specified language (English for below example)

    auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
    auth.set_access_token(access_token, access_token_secret)
    api = tweepy.API(auth)
    tweets = api.search(q = keywordtosearch, lang = 'en', count = 100, truncated = False, tweet_mode = 'extended')
    for tweet in tweets:
        print(tweet.full_text)
        #add your code
2
  • tried your code, got AttributeError: 'API' object has no attribute 'search' Oct 12, 2021 at 17:05
  • 1
    replace search with search_tweets
    – Bruno Gois
    Oct 25, 2021 at 1:34
0

This worked for me.

auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
a=input("Enter Tag: ")
tweets = api.search(a, count=200)
a=[]
for tweet in tweets:
    if tweet.lang == "en":
        a.append(tweet.text)
0

With the help of GetOldTweets3 (https://pypi.org/project/GetOldTweets3/), you can download tweets (even old ones) by filtering over few criteria, as shown below:

tweetCriteria = got.manager.TweetCriteria().setQuerySearch('Coronavirus')\
                                       .setSince("2020-02-15")\
                                       .setUntil("2020-03-29")\
                                       .setMaxTweets(5)\
                                       .setNear('India')\
                                       .setLang('en')
tweets = got.manager.TweetManager.getTweets(tweetCriteria)
for tweet in tweets:
    print(tweet.text)
    print(tweet.date)
    print(tweet.geo)
    print(tweet.id)
    print(tweet.permalink)
    print(tweet.username)
    print(tweet.retweets)
    print(tweet.favorites)
    print(tweet.mentions)
    print(tweet.hashtags)
    print('*'*50)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.