566

I am getting error Expecting value: line 1 column 1 (char 0) when trying to decode JSON.

The URL I use for the API call works fine in the browser, but gives this error when done through a curl request. The following is the code I use for the curl request.

The error happens at return simplejson.loads(response_json)

response_json = self.web_fetch(url)
response_json = response_json.decode('utf-8')
return json.loads(response_json)


def web_fetch(self, url):
    buffer = StringIO()
    curl = pycurl.Curl()
    curl.setopt(curl.URL, url)
    curl.setopt(curl.TIMEOUT, self.timeout)
    curl.setopt(curl.WRITEFUNCTION, buffer.write)
    curl.perform()
    curl.close()
    response = buffer.getvalue().strip()
    return response

Traceback:

File "/Users/nab/Desktop/myenv2/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  111.                         response = callback(request, *callback_args, **callback_kwargs)
File "/Users/nab/Desktop/pricestore/pricemodels/views.py" in view_category
  620.     apicall=api.API().search_parts(category_id= str(categoryofpart.api_id), manufacturer = manufacturer, filter = filters, start=(catpage-1)*20, limit=20, sort_by='[["mpn","asc"]]')
File "/Users/nab/Desktop/pricestore/pricemodels/api.py" in search_parts
  176.         return simplejson.loads(response_json)
File "/Users/nab/Desktop/myenv2/lib/python2.7/site-packages/simplejson/__init__.py" in loads
  455.         return _default_decoder.decode(s)
File "/Users/nab/Desktop/myenv2/lib/python2.7/site-packages/simplejson/decoder.py" in decode
  374.         obj, end = self.raw_decode(s)
File "/Users/nab/Desktop/myenv2/lib/python2.7/site-packages/simplejson/decoder.py" in raw_decode
  393.         return self.scan_once(s, idx=_w(s, idx).end())

Exception Type: JSONDecodeError at /pricemodels/2/dir/
Exception Value: Expecting value: line 1 column 1 (char 0)
11
  • Why decode the response? (simple)json can handle UTF-8 encoded JSON just fine.
    – Martijn Pieters
    May 15, 2013 at 19:24
  • 4
    Last but not least, what does print repr(response_json) tell you is being passed to .loads()?
    – Martijn Pieters
    May 15, 2013 at 19:25
  • 7
    One more: why use simplejson when you can just use the stdlib json (which is the same library as simplejson)?
    – Martijn Pieters
    May 15, 2013 at 19:26
  • When I do print repr(response_json) it just says u'' May 15, 2013 at 19:32
  • 3
    That is an empty string. Your web_fetch() call failed.
    – Martijn Pieters
    May 15, 2013 at 19:33

24 Answers 24

266

Your code produced an empty response body, you'd want to check for that or catch the exception raised. It is possible the server responded with a 204 No Content response, or a non-200-range status code was returned (404 Not Found, etc.). Check for this.

Note:

  • There is no need to use simplejson library, the same library is included with Python as the json module.

  • There is no need to decode a response from UTF8 to unicode, the simplejson / json .loads() method can handle UTF8 encoded data natively.

  • pycurl has a very archaic API. Unless you have a specific requirement for using it, there are better choices.

Either the requests or httpx offers much friendlier APIs, including JSON support. If you can, replace your call with:

import requests

response = requests.get(url)
response.raise_for_status()  # raises exception when not a 2xx response
if response.status_code != 204:
    return response.json()

Of course, this won't protect you from a URL that doesn't comply with HTTP standards; when using arbirary URLs where this is a possibility, check if the server intended to give you JSON by checking the Content-Type header, and for good measure catch the exception:

if (
    response.status_code != 204 and
    response.headers["content-type"].strip().startswith("application/json")
):
    try:
        return response.json()
    except ValueError:
        # decide how to handle a server that's misbehaving to this extent
1
  • 1
    This is useful for determining when the json exists or to use response.text which is the alternative response.
    – D.L
    Feb 7, 2022 at 13:16
195

Be sure to remember to invoke json.loads() on the contents of the file, as opposed to the file path of that JSON:

json_file_path = "/path/to/example.json"

with open(json_file_path, 'r') as j:
     contents = json.loads(j.read())

I think a lot of people are guilty of doing this every once in a while (myself included):

contents = json.load(json_file_path)
3
  • 18
    I think in that case one should use json.load() instead.
    – Coddy
    Dec 14, 2019 at 18:26
  • 5
    The problem with json.load() is, if the file is empty then you will get the same error as the question. If you load the json content from HTTP response, you can check it via HTTP status code, but if you load it from file, then you will need to check if the file is empty or not. So it is better to use j.read() then check if it's empty.
    – jackblk
    Jun 16, 2022 at 8:40
  • 1
    I added an extra check on the file.is_empty() before loading. p = Path("example.json"); if p.exists() and p.stat().st_size > 0 ...
    – Pramit
    Oct 28, 2022 at 19:10
102

Check the response data-body, whether actual data is present and a data-dump appears to be well-formatted.

In most cases your json.loads- JSONDecodeError: Expecting value: line 1 column 1 (char 0) error is due to :

  • non-JSON conforming quoting
  • XML/HTML output (that is, a string starting with <), or
  • incompatible character encoding

Ultimately the error tells you that at the very first position the string already doesn't conform to JSON.

As such, if parsing fails despite having a data-body that looks JSON like at first glance, try replacing the quotes of the data-body:

import sys, json
struct = {}
try:
  try: #try parsing to dict
    dataform = str(response_json).strip("'<>() ").replace('\'', '\"')
    struct = json.loads(dataform)
  except:
    print repr(resonse_json)
    print sys.exc_info()

Note: Quotes within the data must be properly escaped

4
  • 6
    In the comments it was clear the OP got an empty response. Since requests.get(url).json() Just Works, the JSON isn't malformed either.
    – Martijn Pieters
    Sep 22, 2014 at 21:19
  • 3
    JSONDecodeError: Expecting value: line 1 column 1 (char 0) specifically occurs when an empty string is passed to json decode
    – wesinat0r
    May 11, 2020 at 16:38
  • 5
    JSONDecodeError: Expecting value: line 1 column 1 (char 0) also happens when the first line in the json response is invalid. Example response from running an az cli command is ["WARNING: The default kind for created storage account will change to 'StorageV2' from 'Storage' in the future", '{',. This gave me the error that lead me here. The rest of the response IS a valid json object. Just that first line breaks things.
    – SeaDude
    May 21, 2020 at 5:40
  • 1
    In some cases it is necessary to pass the header as headers: { 'Accept': 'application/json', 'Content-Type': 'application/json', } along with the request URL in order to receive the valid JSON response.
    – Dev
    Apr 4, 2021 at 15:18
53

With the requests lib JSONDecodeError can happen when you have an http error code like 404 and try to parse the response as JSON !

You must first check for 200 (OK) or let it raise on error to avoid this case. I wish it failed with a less cryptic error message.

NOTE: as Martijn Pieters stated in the comments servers can respond with JSON in case of errors (it depends on the implementation), so checking the Content-Type header is more reliable.

3
  • Sorry for the old comment, but could you link to an example? I'm trying to take my skills from "perform action", to "attempt to perform action, return response, react accordingly".
    – dcclassics
    Oct 30, 2017 at 22:57
  • @dcclassics: Example: it fails on server-side and server responds by showing an error page (HTML) instead of answering with JSON, so the code parsing the answer will attempt to read JSON but will fail on the HTML tags. Nov 8, 2017 at 11:12
  • 1
    Servers can and do include JSON bodies in error responses. It’s not just 200 OK responses. You want to check the Content-Type header.
    – Martijn Pieters
    Aug 20, 2018 at 16:40
28

Check encoding format of your file and use corresponding encoding format while reading file. It will solve your problem.

with open("AB.json", encoding='utf-8', errors='ignore') as json_data:
     data = json.load(json_data, strict=False)
2
  • 6
    This worked for me with the small change of encoding='utf-8', so I suppose sometimes you need to try a few things out. Jun 14, 2019 at 11:08
  • Cannot agree more, it really depends on the setup...
    – Kris Stern
    Nov 30, 2020 at 1:30
26

I had the same issue trying to read json files with

json.loads("file.json")

I solved the problem with

with open("file.json", "r") as read_file:
   data = json.load(read_file)

maybe this can help in your case

1
  • I think it's important to make sure that the FILE PATH must be checked. Jun 7, 2022 at 2:29
18

A lot of times, this will be because the string you're trying to parse is blank:

>>> import json
>>> x = json.loads("")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

You can remedy by checking whether json_string is empty beforehand:

import json

if json_string:
    x = json.loads(json_string)
else:
    # Your code/logic here 
    x = {}
4
  • 1
    While debugging further up in my code I was calling response.read() and then was dismayed when another call resulted in Expecting value: line 1 etc. Removed the debug statement and problem solved.
    – Joe
    Jun 20, 2019 at 20:46
  • To debug, you can also use this nice website jsonlint.com
    – Roelant
    Oct 24, 2019 at 11:53
  • it can also be an empty binary string starting with b. it's not te same if you use a if to detect it : b''
    – Patrice G
    Apr 8, 2022 at 9:16
  • @PatriceG That is called a bytes literal and it works the same way in Python 3.7+ that I've tested
    – Alex W
    Sep 7, 2022 at 12:58
17

I encounterred the same problem, while print out the json string opened from a json file, found the json string starts with '', which by doing some reserach is due to the file is by default decoded with UTF-8, and by changing encoding to utf-8-sig, the mark out is stripped out and loads json no problem:

open('test.json', encoding='utf-8-sig')
1
  • Solved my problem. Almost identical as your description, some weird character string at the start. Thanks a lot.
    – Bowen Liu
    Oct 20, 2020 at 20:16
8

This is the minimalist solution I found when you want to load json file in python

import json
data = json.load(open('file_name.json'))

If this give error saying character doesn't match on position X and Y, then just add encoding='utf-8' inside the open round bracket

data = json.load(open('file_name.json', encoding='utf-8'))

Explanation open opens the file and reads the containts which later parse inside json.load.

Do note that using with open() as f is more reliable than above syntax, since it make sure that file get closed after execution, the complete sytax would be

with open('file_name.json') as f:
    data = json.load(f)
6

There may be embedded 0's, even after calling decode(). Use replace():

import json
struct = {}
try:
    response_json = response_json.decode('utf-8').replace('\0', '')
    struct = json.loads(response_json)
except:
    print('bad json: ', response_json)
return struct
2
  • response_json not defined
    – Maths12
    May 4, 2021 at 15:59
  • The original poster refers to the response json as response_json .
    – bryan
    May 17, 2021 at 18:52
6

I had the same issue, in my case I solved like this:

import json

with open("migrate.json", "rb") as read_file:
   data = json.load(read_file)
5

I was having the same problem with requests (the python library). It happened to be the accept-encoding header.

It was set this way: 'accept-encoding': 'gzip, deflate, br'

I simply removed it from the request and stopped getting the error.

5

Just check if the request has a status code 200. So for example:

if status != 200:
    print("An error has occured. [Status code", status, "]")
else:
    data = response.json() #Only convert to Json when status is OK.
    if not data["elements"]:
        print("Empty JSON")
    else:
        "You can extract data here"
5

In my case I was doing file.read() two times in if and else block which was causing this error. so make sure to not do this mistake and hold contain in variable and use variable multiple times.

4

I had exactly this issue using requests. Thanks to Christophe Roussy for his explanation.

To debug, I used:

response = requests.get(url)
logger.info(type(response))

I was getting a 404 response back from the API.

2
  • 1
    It can be simplified to response.status_code or print(response.status_code). Jan 17, 2019 at 13:58
  • 1
    Same. I was getting a 403 error. So my response looks like this, <Response [403]> which isn't a valid json format
    – Gidi9
    Aug 17, 2021 at 11:04
4

In my case it occured because i read the data of the file using file.read() and then tried to parse it using json.load(file).I fixed the problem by replacing json.load(file) with json.loads(data)

Not working code

with open("text.json") as file:
    data=file.read()
    json_dict=json.load(file)

working code

with open("text.json") as file:
   data=file.read()
   json_dict=json.loads(data)
4
  • Your file variable is the fp described in docs.python.org/3/library/json.html#basic-usage, so use it like so: json.load(file)
    – Jesse H.
    Apr 14, 2022 at 20:09
  • @JesseH. isn't it what i have done in the not working code part? It throws the error Apr 15, 2022 at 9:06
  • The reason it is not working is you are using .read() method on file. Read more about that here: stackoverflow.com/a/3906148/12678216 . You should remove that line, and see if it works.
    – Jesse H.
    Apr 15, 2022 at 17:13
  • @JesseH. Yes i have realized that .read() method is causing the issue so i have suggested a workaround in the answer. Why do you think this is a bad answer? Apr 16, 2022 at 2:36
2

For me, it was not using authentication in the request.

2

I did:

  1. Open test.txt file, write data
  2. Open test.txt file, read data

So I didn't close file after 1.

I added

outfile.close()

and now it works

1
  • I also had a similar problem using two with statement, even though it should handle the closing by itself
    – G M
    Nov 19, 2021 at 13:14
1

For me it was server responding with something other than 200 and the response was not json formatted. I ended up doing this before the json parse:

# this is the https request for data in json format
response_json = requests.get() 

# only proceed if I have a 200 response which is saved in status_code
if (response_json.status_code == 200):  
     response = response_json.json() #converting from json to dictionary using json library
1
  • This was the problem for me. The status code was 500 (internal server error) instead of 200, so no json was returned and therefore there was nothing in line 1 col 1 of the json. Always good to check that the request status code is what you expect it to be.
    – thposs
    Feb 27, 2020 at 18:47
1

I received such an error in a Python-based web API's response .text, but it led me here, so this may help others with a similar issue (it's very difficult to filter response and request issues in a search when using requests..)

Using json.dumps() on the request data arg to create a correctly-escaped string of JSON before POSTing fixed the issue for me

requests.post(url, data=json.dumps(data))
1

In my case it is because the server is giving http error occasionally. So basically once in a while my script gets the response like this rahter than the expected response:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<h1>502 Bad Gateway</h1>
<p>The proxy server received an invalid response from an upstream server.<hr/>Powered by Tengine</body>
</html>

Clearly this is not in json format and trying to call .json() will yield JSONDecodeError: Expecting value: line 1 column 1 (char 0)

You can print the exact response that causes this error to better debug. For example if you are using requests and then simply print the .text field (before you call .json()) would do.

0

If you are a Windows user, Tweepy API can generate an empty line between data objects. Because of this situation, you can get "JSONDecodeError: Expecting value: line 1 column 1 (char 0)" error. To avoid this error, you can delete empty lines.

For example:

 def on_data(self, data):
        try:
            with open('sentiment.json', 'a', newline='\n') as f:
                f.write(data)
                return True
        except BaseException as e:
            print("Error on_data: %s" % str(e))
        return True

Reference: Twitter stream API gives JSONDecodeError("Expecting value", s, err.value) from None

1
  • 1
    I don't think empty lines are a problem. It clearly states that the error is on line 1 column 1. I think this workaround works because it is removing the BOM from the file. You can quickly verify it: 1. Check the size of your original file (right click > Properties), it can be 134.859 bytes 2. Open the original file with Notepad++ 3. Change the Encoding from "UTF-8-BOM" to "UTF-8". Save 4. Check the size again. It ca be 134.856 (3 bytes less)
    – Alex 75
    Mar 22, 2020 at 12:23
0

if you use headers and have "Accept-Encoding": "gzip, deflate, br" install brotli library with pip install. You don't need to import brotli to your py file.

0

In my case it was a simple solution of replacing single quotes with double. You can find my answer here

1
  • 2
    While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. Feb 23, 2022 at 8:35

Not the answer you're looking for? Browse other questions tagged or ask your own question.