9

I am trying to save information from Twits (screen_name, created_at and text) into a pandas DataFrame and then save DataFrame as a csv file.

I am getting an encoding error

import tweepy
from tweepy import OAuthHandler

consumer_key = 'bla'
consumer_secret = 'bla'
access_token = 'bla'
access_secret = 'bla'
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_secret)
api = tweepy.API(auth)

import pandas as pd
import numpy as np
import datetime
import sys

encoding = sys.stdout.encoding or 'utf-8'
columns = ['Screen_Name', 'Time_Stamp', 'Tweet']
todays_date = datetime.datetime.now().date()
tweetDF = pd.DataFrame(columns=columns)

for tweet in tweepy.Cursor(api.search, q="manhattan", lang="en").items(10):
    lenDF = len(tweetDF)
    tweetDF.loc[lenDF] = [tweet.user.screen_name, tweet.created_at, tweet.text]

tweetDF.to_csv("C:/tweetDF")

Here the error:

UnicodeEncodeError                        Traceback (most recent call last)
<ipython-input-11-c0aa5e7ee620> in <module>()

---> 34 tweetDF.to_csv("C:/tweetDF")

C:\Anaconda\lib\site-packages\pandas\core\frame.pyc in to_csv(self, path_or_buf, sep, na_rep, float_format, columns, header, index, index_label, mode, encoding, quoting, quotechar, line_terminator, chunksize, tupleize_cols, date_format, doublequote, escapechar, decimal, **kwds)
   1187                                      escapechar=escapechar,
   1188                                      decimal=decimal)
-> 1189         formatter.save()
   1190 
   1191         if path_or_buf is None:

C:\Anaconda\lib\site-packages\pandas\core\format.pyc in save(self)
   1465 
   1466             else:
-> 1467                 self._save()
   1468 
   1469         finally:

C:\Anaconda\lib\site-packages\pandas\core\format.pyc in _save(self)
   1565                 break
   1566 
-> 1567             self._save_chunk(start_i, end_i)
   1568 
   1569     def _save_chunk(self, start_i, end_i):

C:\Anaconda\lib\site-packages\pandas\core\format.pyc in _save_chunk(self, start_i, end_i)
   1592                                         quoting=self.quoting)
   1593 
-> 1594         lib.write_csv_rows(self.data, ix, self.nlevels, self.cols, self.writer)
   1595 
   1596 # from collections import namedtuple

pandas\lib.pyx in pandas.lib.write_csv_rows (pandas\lib.c:17992)()

UnicodeEncodeError: 'ascii' codec can't encode character u'\u2666' in position 7: ordinal not in range(128)

I tried various encoding enhancements but was not successfull

3
  • 1
    What version of Python are you using?
    – MattDMo
    Commented Oct 10, 2015 at 21:06
  • @MattDMo - version 2.7
    – Toly
    Commented Oct 10, 2015 at 23:31
  • 1
    This post might need an updated answer for Python3.
    – user12217470
    Commented May 31, 2021 at 18:37

1 Answer 1

8

I found the way to fix it and would like to share:

tweetDF.to_csv("C:/tweetDF", sep='\t', encoding = 'utf-8')
1
  • Can you explain why this resolved the issue? I have no idea why this works compared to the original issue. encoding defaults to 'utf-8' already when using this function, as stated in the original documentation for this function pandas.pydata.org/docs/reference/api/… .
    – user12217470
    Commented May 31, 2021 at 18:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.