I need to receive a protocol buffers message on my python - tornado server and get the stuff out of the binary message.
postContent = self.request.body
message = prototemp.ReqMessage()
message.ParseFromString(postContent)
It works perfectly using a test tool. When i run it in sandbox environment and simulate 1000 requests from my client, it works in certain cases, but in most of the requests, it throws an exception -
File "server1.py", line 21, in post
message.ParseFromString(postContent)
File "/usr/lib/python2.6/site-packages/protobuf-2.4.1-py2.6.egg/google/protobuf/message.py", line 179, in ParseFromString
self.MergeFromString(serialized)
File "/usr/lib/python2.6/site-packages/protobuf-2.4.1-py2.6.egg/google/protobuf/internal/python_message.py", line 755, in MergeFromString
if self._InternalParse(serialized, 0, length) != length:
File "/usr/lib/python2.6/site-packages/protobuf-2.4.1-py2.6.egg/google/protobuf/internal/python_message.py", line 782, in InternalParse
pos = field_decoder(buffer, new_pos, end, self, field_dict)
File "/usr/lib/python2.6/site-packages/protobuf-2.4.1-py2.6.egg/google/protobuf/internal/decoder.py", line 544, in DecodeField
if value._InternalParse(buffer, pos, new_pos) != new_pos:
File "/usr/lib/python2.6/site-packages/protobuf-2.4.1-py2.6.egg/google/protobuf/internal/python_message.py", line 782, in InternalParse
pos = field_decoder(buffer, new_pos, end, self, field_dict)
File "/usr/lib/python2.6/site-packages/protobuf-2.4.1-py2.6.egg/google/protobuf/internal/decoder.py", line 410, in DecodeField
field_dict[key] = local_unicode(buffer[pos:new_pos], 'utf-8')
UnicodeDecodeError: 'utf8' codec can't decode byte 0xce in position 1: invalid continuation byte
In some other cases it gives these errors -
UnicodeDecodeError: 'utf8' codec can't decode byte 0xbf in position 3: invalid start byte
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe7 in position 3: unexpected end of data
What could be the reason ?
pdb
to view what the variables are at that point? Because it's telling you the problem: there's some character at the indicated position in the string that can't be encoded with utf-8. So either you need to handle that character. (and if you can figure out what it is and whether you have to deal with it in general, you'll be able to handle it) – Jeff Tratner Jun 9 '12 at 5:13