9

I'm trying to write some python code which can create multipart mime http requests in the client, and then appropriately interpret then on the server. I have, I think, partially succeeded on the client end with this:

from email.mime.multipart import MIMEMultipart, MIMEBase
import httplib
h1 = httplib.HTTPConnection('localhost:8080')
msg = MIMEMultipart()
fp = open('myfile.zip', 'rb')
base = MIMEBase("application", "octet-stream")
base.set_payload(fp.read())
msg.attach(base)
h1.request("POST", "http://localhost:8080/server", msg.as_string())

The only problem with this is that the email library also includes the Content-Type and MIME-Version headers, and I'm not sure how they're going to be related to the HTTP headers included by httplib:

Content-Type: multipart/mixed; boundary="===============2050792481=="
MIME-Version: 1.0

--===============2050792481==
Content-Type: application/octet-stream
MIME-Version: 1.0

This may be the reason that when this request is received by my web.py application, I just get an error message. The web.py POST handler:

class MultipartServer:
    def POST(self, collection):
        print web.input()

Throws this error:

Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/web.py-0.34-py2.6.egg/web/application.py", line 242, in process
    return self.handle()
  File "/usr/local/lib/python2.6/dist-packages/web.py-0.34-py2.6.egg/web/application.py", line 233, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/local/lib/python2.6/dist-packages/web.py-0.34-py2.6.egg/web/application.py", line 415, in _delegate
    return handle_class(cls)
  File "/usr/local/lib/python2.6/dist-packages/web.py-0.34-py2.6.egg/web/application.py", line 390, in handle_class
    return tocall(*args)
  File "/home/richard/Development/server/webservice.py", line 31, in POST
    print web.input()
  File "/usr/local/lib/python2.6/dist-packages/web.py-0.34-py2.6.egg/web/webapi.py", line 279, in input
    return storify(out, *requireds, **defaults)
  File "/usr/local/lib/python2.6/dist-packages/web.py-0.34-py2.6.egg/web/utils.py", line 150, in storify
    value = getvalue(value)
  File "/usr/local/lib/python2.6/dist-packages/web.py-0.34-py2.6.egg/web/utils.py", line 139, in getvalue
    return unicodify(x)
  File "/usr/local/lib/python2.6/dist-packages/web.py-0.34-py2.6.egg/web/utils.py", line 130, in unicodify
    if _unicode and isinstance(s, str): return safeunicode(s)
  File "/usr/local/lib/python2.6/dist-packages/web.py-0.34-py2.6.egg/web/utils.py", line 326, in safeunicode
    return obj.decode(encoding)
  File "/usr/lib/python2.6/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 137-138: invalid data

My line of code is represented by the error line about half way down:

  File "/home/richard/Development/server/webservice.py", line 31, in POST
    print web.input()

It's coming along, but I'm not sure where to go from here. Is this a problem with my client code, or a limitation of web.py (perhaps it just can't support multipart requests)? Any hints or suggestions of alternative code libraries would be gratefully received.

EDIT

The error above was caused by the data not being automatically base64 encoded. Adding

encoders.encode_base64(base)

Gets rid of this error, and now the problem is clear. HTTP request isn't being interpreted correctly in the server, presumably because the email library is including what should be the HTTP headers in the body instead:

<Storage {'Content-Type: multipart/mixed': u'', 
          ' boundary': u'"===============1342637378=="\n'
          'MIME-Version: 1.0\n\n--===============1342637378==\n'
          'Content-Type: application/octet-stream\n'
          'MIME-Version: 1.0\n' 
          'Content-Transfer-Encoding: base64\n'
          '\n0fINCs PBk1jAAAAAAAAA.... etc

So something is not right there.

Thanks

Richard

1
  • 1
    @TokenMacGuy - yes. yes it is.
    – Richard J
    Dec 14, 2010 at 13:54

3 Answers 3

1

I used this package by Will Holcomb http://pypi.python.org/pypi/MultipartPostHandler/0.1.0 to make multi-part requests with urllib2, it may help you out.

1
  • Great, thanks, I'll take a look at that. Certainly has the right sort of name :) Cheers, R
    – Richard J
    Dec 14, 2010 at 13:55
1

After a bit of exploration, the answer to this question has become clear. The short answer is that although the Content-Disposition is optional in a Mime-encoded message, web.py requires it for each mime-part in order to correctly parse out the HTTP request.

Contrary to other comments on this question, the difference between HTTP and Email is irrelevant, as they are simply transport mechanisms for the Mime message and nothing more. Multipart/related (not multipart/form-data) messages are common in content exchanging webservices, which is the use case here. The code snippets provided are accurate, though, and led me to a slightly briefer solution to the problem.

# open an HTTP connection
h1 = httplib.HTTPConnection('localhost:8080')

# create a mime multipart message of type multipart/related
msg = MIMEMultipart("related")

# create a mime-part containing a zip file, with a Content-Disposition header
# on the section
fp = open('file.zip', 'rb')
base = MIMEBase("application", "zip")
base['Content-Disposition'] = 'file; name="package"; filename="file.zip"'
base.set_payload(fp.read())
encoders.encode_base64(base)
msg.attach(base)

# Here's a rubbish bit: chomp through the header rows, until hitting a newline on
# its own, and read each string on the way as an HTTP header, and reading the rest
# of the message into a new variable
header_mode = True
headers = {}
body = []
for line in msg.as_string().splitlines(True):
    if line == "\n" and header_mode == True:
        header_mode = False
    if header_mode:
        (key, value) = line.split(":", 1)
        headers[key.strip()] = value.strip()
    else:
        body.append(line)
body = "".join(body)

# do the request, with the separated headers and body
h1.request("POST", "http://localhost:8080/server", body, headers)

This is picked up perfectly well by web.py, so it's clear that email.mime.multipart is suitable for creating Mime messages to be transported by HTTP, with the exception of its header handling.

My other overall conern is in scalability. Neither this solution nor the others proposed here scale well, as they read the contents of a file into a variable before bundling up in the mime message. A better solution would be one which could serialise on demand as the content is piped out over the HTTP connection. It's not urgent for me to fix that, but I'll come back here with a solution if I get to it.

1
  • 1. I think the preferred way to set the header is something like base.add_header('Content-Disposition','file',name='package',...). 2. It's better to search for \n\n (and also \r\n\r\n, with e.g. re.search('\r?\n\r?\n',...)) so you don't have to split and join the body. 3. Header lines can be folded. 4. Technically the \n that terminates the headers is not part of the body, though this isn't harmful. 5. I'm not entirely sure that the RFC 5322 and RFC 2316 grammars are 100% compatible (in particular WRT "characters" vs octets).
    – tc.
    Apr 17, 2012 at 17:46
-1

There is a number of things wrong with your request. As TokenMacGuy suggests, multipart/mixed is unused in HTTP; use multipart/form-data instead. In addition, parts should have a Content-disposition header. A python fragment to do that can be found in the Code Recipes.

5
  • thanks - as you can see, I'm still working on this; I haven't figured out where multipart/mixed is being set yet. Likewise I haven't yet employed the Content-Disposition header as I'm still working on getting it into an HTTP request in the first place. My question is about how to construct such a request in the first place. Cheers, R.
    – Richard J
    Dec 14, 2010 at 13:56
  • See the recipe. Forget about email.mime - HTTP is not email. Dec 14, 2010 at 18:30
  • Hi Martin; FYI I've demonstrated that the difference between HTTP and email is irrelevant here - they are simply transports, and mime is the same in either case. See my alternative answer. Thanks for the pointers. R
    – Richard J
    Dec 21, 2010 at 23:27
  • 2
    That's wrong, multipart/mixed is used in HTTP: docs.couchdb.org/en/latest/replication/…
    – fiatjaf
    Feb 6, 2015 at 3:52
  • even if rare, multipart/mixed is used in python. See Google APIs: developers.google.com/drive/api/v3/performance#details
    – Lorenzo P
    Aug 3, 2019 at 14:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.