0

I have a database with lots of tweets that I crawled using Twitter API. Those tweets are in a json format what allows me to convert them into dictionaries, right? So here we go: I'm trying to convert these strings into dictionaries using the json package. But everytime I try to run json.loads(string), it gives me an error:

ValueError: Expecting property name: line 1 column 2 (char 1).

Here's an example of the "json strings" I have in my database.

{u"contributors": "None", u"truncated": False, u"text": u"RT @WhoScored: Most clear-cut chances missed at World Cup 2014: M\xfcller / Higua\xedn / Benzema (5), de Vrij / \xd6zil / Ronaldo (4)", u"in_reply_to_status_id": "None", u"id": 487968527395983360, u"favorite_count": 0, u"source": u"<a href="http://blackberry.com/twitter" rel="nofollow">Twitter for BlackBerry\xae</a>", u"retweeted": False, u"coordinates": "None", u"entities": {u"user_mentions": [{u"id": 99806132, u"indices": [3, 13], u"id_str": u"99806132", u"screen_name": u"WhoScored", u"name": u"WhoScored.com"}], u"symbols": [], u"trends": [], u"hashtags": [], u"urls": []}, u"in_reply_to_screen_name": "None", u"id_str": u"487968527395983360", u"retweet_count": 0, u"in_reply_to_user_id": "None", u"favorited": False, u"retweeted_status": {u"contributors": "None", u"truncated": False, u"text": u"Most clear-cut chances missed at World Cup 2014: M\xfcller / Higua\xedn / Benzema (5), de Vrij / \xd6zil / Ronaldo (4)", u"in_reply_to_status_id": "None", u"id": 487955847025143808, u"favorite_count": 17, u"source": u"<a href="https://about.twitter.com/products/tweetdeck" rel="nofollow">TweetDeck</a>", u"retweeted": False, u"coordinates": "None", u"entities": {u"user_mentions": [], u"symbols": [], u"trends": [], u"hashtags": [], u"urls": []}, u"in_reply_to_screen_name": "None", u"id_str": u"487955847025143808", u"retweet_count": 59, u"in_reply_to_user_id": "None", u"favorited": False, u"user": {u"follow_request_sent": "None", u"profile_use_background_image": True, u"default_profile_image": False, u"id": 99806132, u"verified": True, u"profile_image_url_https": u"https://pbs.twimg.com/profile_images/477005408557486083/9MVR7GdF_normal.jpeg", u"profile_sidebar_fill_color": u"DDEEF6", u"profile_text_color": u"333333", u"followers_count": 425860, u"profile_sidebar_border_color": u"C0DEED", u"id_str": u"99806132", u"profile_background_color": u"272727", u"listed_count": 3245, u"profile_background_image_url_https": u"https://pbs.twimg.com/profile_background_images/439356280/123abc.jpg", u"utc_offset": 3600, u"statuses_count": 24118, u"description": u"The largest detailed football statistics website, covering Europe"s top leagues and more. Follow @WSTipster for betting tips. Powered by Opta data.", u"friends_count": 67, u"location": u"London", u"profile_link_color": u"0084B4", u"profile_image_url": u"http://pbs.twimg.com/profile_images/477005408557486083/9MVR7GdF_normal.jpeg", u"following": "None", u"geo_enabled": False, u"profile_banner_url": u"https://pbs.twimg.com/profile_banners/99806132/1402565693", u"profile_background_image_url": u"http://pbs.twimg.com/profile_background_images/439356280/123abc.jpg", u"name": u"WhoScored.com", u"lang": u"en", u"profile_background_tile": False, u"favourites_count": 250, u"screen_name": u"WhoScored", u"notifications": "None", u"url": u"http://whoscored.com", u"created_at": u"Sun Dec 27 23:22:45 +0000 2009", u"contributors_enabled": False, u"time_zone": u"London", u"protected": False, u"default_profile": False, u"is_translator": False}, u"geo": "None", u"in_reply_to_user_id_str": "None", u"possibly_sensitive": False, u"lang": u"ro", u"created_at": u"Sat Jul 12 13:45:14 +0000 2014", u"filter_level": u"low", u"in_reply_to_status_id_str": "None", u"place": "None"}, u"user": {u"follow_request_sent": "None", u"profile_use_background_image": True, u"default_profile_image": False, u"id": 498676612, u"verified": False, u"profile_image_url_https": u"https://pbs.twimg.com/profile_images/485720258934603776/BmUaZHax_normal.jpeg", u"profile_sidebar_fill_color": u"DDEEF6", u"profile_text_color": u"333333", u"followers_count": 192, u"profile_sidebar_border_color": u"C0DEED", u"id_str": u"498676612", u"profile_background_color": u"C0DEED", u"listed_count": 1, u"profile_background_image_url_https": u"https://pbs.twimg.com/profile_background_images/654833637/l9fp65m0xqzsmoneg8pz.jpeg", u"utc_offset": "None", u"statuses_count": 6468, u"description": u"Garut 10 July | Asda islamic school | Farmasi | @judikajude  | path : Dicky Darul Majid", u"friends_count": 153, u"location": u"", u"profile_link_color": u"B30000", u"profile_image_url": u"http://pbs.twimg.com/profile_images/485720258934603776/BmUaZHax_normal.jpeg", u"following": "None", u"geo_enabled": True, u"profile_banner_url": u"https://pbs.twimg.com/profile_banners/498676612/1404927261", u"profile_background_image_url": u"http://pbs.twimg.com/profile_background_images/654833637/l9fp65m0xqzsmoneg8pz.jpeg", u"name": u"DICKY ", u"lang": u"en", u"profile_background_tile": False, u"favourites_count": 74, u"screen_name": u"DickyDarulMajid", u"notifications": "None", u"url": "None", u"created_at": u"Tue Feb 21 09:21:59 +0000 2012", u"contributors_enabled": False, u"time_zone": "None", u"protected": False, u"default_profile": False, u"is_translator": False}, u"geo": "None", u"in_reply_to_user_id_str": "None", u"possibly_sensitive": False, u"lang": u"ro", u"created_at": u"Sat Jul 12 14:35:37 +0000 2014", u"filter_level": u"medium", u"in_reply_to_status_id_str": "None", u"place": "None"}

And here is the code:

import sys, codecs, json
encode = sys.stdin.encoding

all_entries = Tweet.objects.all()[1427:1435]

for entry in all_entries:
    tweet = entry.tweet.encode(encode)
    json_acceptable_string = tweet.replace("'", "\"")
    json_acceptable_string = json_acceptable_string.replace("None", "\"None\"")
    data = json.loads(json_acceptable_string)
    print data

Traceback:

Traceback (most recent call last):
File "/home/kiko/workspace/SA_WorldCup/main.py", line 6, in <module>
util.tweets_count()
File "/home/kiko/workspace/SA_WorldCup/util/__init__.py", line 25, in tweets_count
data = json.loads(json_acceptable_string, object_hook=JSONObject)
File "/usr/lib/python2.7/json/__init__.py", line 351, in loads
return cls(encoding=encoding, **kw).decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting property name: line 1 column 2 (char 1)

I tried many times, but with no succeed. Can you help me out? Thanks a lot.

2 Answers 2

0

It looks like your "json" is actually python literal syntax. In that case, it might be easier to ast.literal_eval the string1.

As a side benefit, then you don't have to do any (possibly sketchy) replacements of None with "None". (Consider a tweet which says "None of your base are belong to us")

1although it would probably be even better to make sure that proper json is being dumped into the database to begin with...

0
0

use eval()

In [13]: a = "[{'start_city': '1', 'end_city': 'aaa', 'number': 1},\
...:      {'start_city': '2', 'end_city': 'bbb', 'number': 1},\
...:      {'start_city': '3', 'end_city': 'ccc', 'number': 1}]"
In [14]: eval(a)
Out[14]:
[{'end_city': 'aaa', 'number': 1, 'start_city': '1'},
 {'end_city': 'bbb', 'number': 1, 'start_city': '2'},
 {'end_city': 'ccc', 'number': 1, 'start_city': '3'}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.