0

New to python and I think I've hit the unicode lesson wall. My stack:

  • cx-Oracle (5.1.3)
  • Flask (0.10.1)
  • Flask-SQLAlchemy (2.0)
  • Flask-WTF (0.12)
  • openpyxl (2.2.6)
  • SQLAlchemy (1.0.8)

Python 3.4.3 virtualenv. I'm working on a small we application that allows users to upload excel workbooks and then the application will write the rows to an oracle db. I started having this error code when I was using python 2.7.10. Most of what I read suggested that it was due to my default interpreter encoding for python 2.X. So I moved the code into a python 3.4.3 virtual environment. This error is still happening.

In the code below I'm zipping the column type with the cell type to be inserted into the row. After creating the list of dictionaries, sqlalchemy is called to do inserts. executemany is called as a result.

This error occurs when it hits special characters like the copyright, registered trademark sign, etc that are in the excel worksheet. The if condition below is causing the trouble. Since those special characters are part of a text string, I want python 3 to treat is as a string and then insert it.

From all my readings it looks like python 3 should use utf-8 encoding by default. I've read about all the packages I use and they all use utf-8 encoding. So I don't understand why it tries to use ascii to encode this string & hence this error. I have tried doing an explicit .encode('utf-8') on the value but that causes another problem when the excel cell type is a 'None'. That means it is an empty cell. Then I get the Nonetype has no encode attribute error.

output_data = []
for row in allrows:
   row_dict = {}
        for fobject,cobject in zip(sortedtablecolumnsobject,row):
        ########################################################
        ##Validation & assignment
        ########################################################
            if fobject.type.name == "Text":
                row_dict[fobject.name] = str(cobject.value)
         output_data.append(row_dict)

This is where the table reflection & insertion takes place:

    m = db.MetaData()
    t = db.Table(tableinfo.table_name,m,autoload = True, autoload_with = db.engine)
    db.engine.execute(t.insert(),output_data)

I think one of the packages is using ascii encoding without my approval :-) Please help.

Edit:

>import sys, locale
>sys.stdout.encoding
>'UTF-8'
>locale.getpreferredencoding(False)
>'UTF-8'

Trace below:

   Traceback (most recent call last):
      File "/home/vagrant/Development/dutmyway/env/lib/python3.4/site-packages/flask                                                                            /app.py", line 1836, in __call__
        return self.wsgi_app(environ, start_response)
      File "/home/vagrant/Development/dutmyway/env/lib/python3.4/site-packages/flask                                                                            /app.py", line 1820, in wsgi_app
        response = self.make_response(self.handle_exception(e))
      File "/home/vagrant/Development/dutmyway/env/lib/python3.4/site-packages/flask                                                                            /app.py", line 1403, in handle_exception
        reraise(exc_type, exc_value, tb)
      File "/home/vagrant/Development/dutmyway/env/lib/python3.4/site-packages/flask                                                                            /_compat.py", line 33, in reraise
        raise value
      File "/home/vagrant/Development/dutmyway/env/lib/python3.4/site-packages/flask                                                                            /app.py", line 1817, in wsgi_app
        response = self.full_dispatch_request()
      File "/home/vagrant/Development/dutmyway/env/lib/python3.4/site-packages/flask                                                                            /app.py", line 1477, in full_dispatch_request
        rv = self.handle_user_exception(e)
      File "/home/vagrant/Development/dutmyway/env/lib/python3.4/site-packages/flask                                                                            /app.py", line 1381, in handle_user_exception
        reraise(exc_type, exc_value, tb)
      File "/home/vagrant/Development/dutmyway/env/lib/python3.4/site-packages/flask                                                                            /_compat.py", line 33, in reraise
        raise value
      File "/home/vagrant/Development/dutmyway/env/lib/python3.4/site-packages/flask                                                                            /app.py", line 1475, in full_dispatch_request
        rv = self.dispatch_request()
      File "/home/vagrant/Development/dutmyway/env/lib/python3.4/site-packages/flask                                                                            /app.py", line 1461, in dispatch_request
        return self.view_functions[rule.endpoint](**req.view_args)
      File "/home/vagrant/Development/dutmyway/app/views.py", line 220, in upload
        db.engine.execute(t.insert(),output_data)
      File "/home/vagrant/Development/dutmyway/env/lib/python3.4/site-packages/sqlal                                                                            chemy/engine/base.py", line 1991, in execute
        return connection.execute(statement, *multiparams, **params)
      File "/home/vagrant/Development/dutmyway/env/lib/python3.4/site-packages/sqlal                                                                            chemy/engine/base.py", line 914, in execute
        return meth(self, multiparams, params)
      File "/home/vagrant/Development/dutmyway/env/lib/python3.4/site-packages/sqlal                                                                            chemy/sql/elements.py", line 323, in _execute_on_connection
        return connection._execute_clauseelement(self, multiparams, params)
      File "/home/vagrant/Development/dutmyway/env/lib/python3.4/site-packages/sqlal                                                                            chemy/engine/base.py", line 1010, in _execute_clauseelement
        compiled_sql, distilled_params
      File "/home/vagrant/Development/dutmyway/env/lib/python3.4/site-packages/sqlal                                                                            chemy/engine/base.py", line 1146, in _execute_context
        context)
      File "/home/vagrant/Development/dutmyway/env/lib/python3.4/site-packages/sqlal                                                                            chemy/engine/base.py", line 1344, in _handle_dbapi_exception
        util.reraise(*exc_info)
      File "/home/vagrant/Development/dutmyway/env/lib/python3.4/site-packages/sqlal                                                                            chemy/util/compat.py", line 182, in reraise
        raise value
      File "/home/vagrant/Development/dutmyway/env/lib/python3.4/site-packages/sqlal                                                                            chemy/engine/base.py", line 1116, in _execute_context
        context)
      File "/home/vagrant/Development/dutmyway/env/lib/python3.4/site-packages/sqlal                                                                            chemy/dialects/oracle/cx_oracle.py", line 961, in do_executemany
        cursor.executemany(statement, parameters)
4
  • 1
    Provide the full traceback. What is locale.getpreferredencoding(False)? What is sys.stdout.encoding?
    – jfs
    Oct 14, 2015 at 6:26
  • 1
    Excel tends to escape things like the copyright sign as entity rather than a utf-8 codepoint so you may need to add some code to handle that. Oct 14, 2015 at 10:24
  • @J.F.Sebastian I added what you asked for. As you can see, everything is in utf-8. So I don't get how/why ascii is being used to try encode while inserting into oracle.
    – okyere
    Oct 14, 2015 at 18:42
  • It is not the full traceback. Try a debugger, to localize the error and produce a minimal code example. There are multiple different character encodings that may be used at the same time for different purposes e.g., check db connection encoding. Also, are you sure that you've provided results for the same environment that is used to run your db code?
    – jfs
    Oct 14, 2015 at 19:19

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.