1

I am streaming data from a sensor using socket library in Json format, and trying to parse it and load it into Database. When I print the stream I get this Json in this format:

b'[{"metadata":{"timezone":{"location":"Etc/UTC"},"serial_number":"00:07:32:52:09:fc","device_type":"SPIDER"},"timestamp":"2019-08-29T13:53:05.895Z","framenumber":"2290718","tracked_objects":[{"id":2592,"is_at_border":true,"type":"PERSON","position":{"x":233,"y":262,"type":"FOOT","coordinate_system":"PROCESSING_IN_PIXEL"},"person_data":{"height":1728}}]}]'

Based on my research prefix b stands for the byte type. So when I try to parse it with code below:

while True:
    message, address = server_socket.recvfrom(1024)
    message = message.upper()

    # loading json file.
    objs_json = json.loads(message)
    # using if looop to prevent script of trying to to parse data without any object being tracked.
    if "tracked_objects" in objs_json:
        # Parsing json file with json_normalize object
        objs_df = json_normalize(
            objs_json, record_path='tracked_objects',
            meta=[['metadata', 'serial_number'], 'timestamp']
        )
        # Renaming columns
        objs_df = objs_df.rename(
            columns={
                "id": "object_id", "position.x": "x_pos", 
                "position.y": "y_pos", "person_data.height": "height",
                "metadata.serial_number": "serial_number",
                "timestamp": "timestamp"
            }
        )
        # Selecting columns of interest
        objs_df = objs_df.loc[:, ["timestamp", "serial_number", "object_id", "x_pos", "y_pos", "height"]]
        # Writting the data into SQlite db
        objs_df.to_sql('data_object', con=engine, if_exists='append', index=False)
    # In case there is no tracks, print No Tracks in console.
    else:
        print("No Tracks")

I get this error message:

    Traceback (most recent call last):
  File "/home/pi/ProRail-PMS/Test_Spider2.py", line 20, in <module>
    objs_json = json.loads(message)
  File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 215 (char 214)

However if I save that data into json file and remove the prefix b my parsing code works. How do i go around this so when I receive the data from socket library I want to be able to parse it and feed it into database?

2 Answers 2

6

At first I wanted to comment that it works for me, but then I noticed how you get the message and what you do with it:

Remove message = message.upper():

>>> message = b'[{"metadata":{"timezone":{"location":"Etc/UTC"},"serial_number":"00:07:32:52:09:fc","device_type":"SPIDER"},"timestamp":"2019-08-29T13:53:05.895Z","framenumber":"2290718","tracked_objects":[{"id":2592,"is_at_border":true,"type":"PERSON","position":{"x":233,"y":262,"type":"FOOT","coordinate_system":"PROCESSING_IN_PIXEL"},"person_data":{"height":1728}}]}]'
>>> json.loads(message)
[{'metadata': {'timezone': {'location': 'Etc/UTC'}, 'serial_number': '00:07:32:52:09:fc', 'device_type': 'SPIDER'}, 'timestamp': '2019-08-29T13:53:05.895Z', 'framenumber': '2290718', 'tracked_objects': [{'id': 2592, 'is_at_border': True, 'type': 'PERSON', 'position': {'x': 233, 'y': 262, 'type': 'FOOT', 'coordinate_system': 'PROCESSING_IN_PIXEL'}, 'person_data': {'height': 1728}}]}]
>>
>>
>>> message = message.upper()
b'[{"METADATA":{"TIMEZONE":{"LOCATION":"ETC/UTC"},"SERIAL_NUMBER":"00:07:32:52:09:FC","DEVICE_TYPE":"SPIDER"},"TIMESTAMP":"2019-08-29T13:53:05.895Z","FRAMENUMBER":"2290718","TRACKED_OBJECTS":[{"ID":2592,"IS_AT_BORDER":TRUE,"TYPE":"PERSON","POSITION":{"X":233,"Y":262,"TYPE":"FOOT","COORDINATE_SYSTEM":"PROCESSING_IN_PIXEL"},"PERSON_DATA":{"HEIGHT":1728}}]}]'
>>> json.loads(message)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 217 (char 216)
>>> message[217:]
b'RUE,"TYPE":"PERSON","POSITION":{"X":233,"Y":262,"TYPE":"FOOT","COORDINATE_SYSTEM":"PROCESSING_IN_PIXEL"},"PERSON_DATA":{"HEIGHT":1728}}]}]'

Your upper breaks the True value that is unquoted (because it's a boolean, not a string). ;)

2
  • so with your suggestion i am able to avoide that error however it is not parsing my data. I am getting "No Tracks" message as it seems to not finding the ""tracked_objects" within Json file. Did i write that part correctly?
    – Slavisha84
    Aug 29, 2019 at 20:48
  • Be wary of types - your json is a single-element list with a dict inside. Do objs_json = json.loads(message)[0] to get the dictionary. Then your code will see "tracked_objects" part. Dunno about the rest. I don't have time to analyse your whole code. Good luck!
    – h4z3
    Aug 30, 2019 at 8:29
-1

Imported it into Notepad++ since the JSON file is of type byte. Saved it out as JSON and it saved normally as a standard JSON file. Used Python json.load(f) to bring it in from my drive and then it can be used.

1
  • 1
    Hi Bob, welcome to SO! Please read up on answering questions before writing your next answer :)
    – Diggy.
    May 13, 2020 at 16:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.