18

I have a custom EncryptedCharField, which I want to basically appear as a CharField when interfacing UI, but before storing/retrieving in the DB it encrypts/decrypts it.

The custom fields documentation says to:

  1. add __metaclass__ = models.SubfieldBase
  2. override to_python to convert the data from it's raw storage into the desired format
  3. override get_prep_value to convert the value before storing ot the db.

So you think this would be easy enough - for 2. just decrypt the value, and 3. just encrypt it.

Based loosely on a django snippet, and the documentation this field looks like:

class EncryptedCharField(models.CharField):
  """Just like a char field, but encrypts the value before it enters the database, and    decrypts it when it
  retrieves it"""
  __metaclass__ = models.SubfieldBase
  def __init__(self, *args, **kwargs):
    super(EncryptedCharField, self).__init__(*args, **kwargs)
    cipher_type = kwargs.pop('cipher', 'AES')
    self.encryptor = Encryptor(cipher_type)

  def get_prep_value(self, value):
     return encrypt_if_not_encrypted(value, self.encryptor)

  def to_python(self, value):
    return decrypt_if_not_decrypted(value, self.encryptor)


def encrypt_if_not_encrypted(value, encryptor):
  if isinstance(value, EncryptedString):
    return value
  else:
    encrypted = encryptor.encrypt(value)
    return EncryptedString(encrypted)

def decrypt_if_not_decrypted(value, encryptor):
  if isinstance(value, DecryptedString):
    return value
  else:
    encrypted = encryptor.decrypt(value)
    return DecryptedString(encrypted)


class EncryptedString(str):
  pass

class DecryptedString(str):
  pass

and the Encryptor looks like:

class Encryptor(object):
  def __init__(self, cipher_type):
    imp = __import__('Crypto.Cipher', globals(), locals(), [cipher_type], -1)
    self.cipher = getattr(imp, cipher_type).new(settings.SECRET_KEY[:32])

  def decrypt(self, value):
    #values should always be encrypted no matter what!
    #raise an error if tthings may have been tampered with
    return self.cipher.decrypt(binascii.a2b_hex(str(value))).split('\0')[0]

  def encrypt(self, value):
    if value is not None and not isinstance(value, EncryptedString):
      padding  = self.cipher.block_size - len(value) % self.cipher.block_size
      if padding and padding < self.cipher.block_size:
        value += "\0" + ''.join([random.choice(string.printable) for index in range(padding-1)])
      value = EncryptedString(binascii.b2a_hex(self.cipher.encrypt(value)))
    return value

When saving a model, an error, Odd-length string, occurs, as a result of attempting to decrypt an already decrypted string. When debugging, it appears as to_python ends up being called twice, the first with the encrypted value, and the second time with the decrypted value, but not actually as a type Decrypted, but as a raw string, causing the error. Furthermore get_prep_value is never called.

What am I doing wrong?

This should not be that hard - does anyone else think this Django field code is very poorly written, especially when it comes to custom fields, and not that extensible? Simple overridable pre_save and post_fetch methods would easily solve this problem.

1
  • Can I ask how did you solved it, because I need to do something similar to an IntegerField. This is an old question so are you still using this custom encryption or some third party application?
    – copser
    Commented Sep 22, 2017 at 12:53

5 Answers 5

11
+50

I think the issue is that to_python is also called when you assign a value to your custom field (as part of validation may be, based on this link). So the problem is to distinguish between to_python calls in the following situations:

  1. When a value from the database is assigned to the field by Django (That's when you want to decrypt the value)
  2. When you manually assign a value to the custom field, e.g. record.field = value

One hack you could use is to add prefix or suffix to the value string and check for that instead of doing isinstance check.

I was going to write an example, but I found this one (even better :)).

Check BaseEncryptedField: https://github.com/django-extensions/django-extensions/blob/2.2.9/django_extensions/db/fields/encrypted.py (link to an older version because the field was removed in 3.0.0; see Issue #1359 for reason of deprecation)

Source: Django Custom Field: Only run to_python() on values from DB?

2
  • I like that hack idea!!!! Sucks we have to hack it due to poor design, but I will try that out
    – Oved D
    Commented Nov 13, 2012 at 17:27
  • The problem is that Django uses to_python for validation too and that's why to_python is not good for translation where user input is of the same data type as database type means different things, e.g. both are str data, but db one is encrypted and input is decrypted.
    – maulik13
    Commented Nov 13, 2012 at 19:40
4

You should be overriding to_python, like the snippet did.

If you take a look at the CharField class you can see that it doesn't have a value_to_string method:

The docs say that the to_python method needs to deal with three things:

  • An instance of the correct type
  • A string (e.g., from a deserializer).
  • Whatever the database returns for the column type you're using.

You are currently only dealing with the third case.

One way to handle this is to create a special class for a decrypted string:

class DecryptedString(str):
   pass

Then you can detect this class and handle it in to_python():

def to_python(self, value):
    if isinstance(value, DecryptedString):
        return value

    decrypted = self.encrypter.decrypt(encrypted)
    return DecryptedString(decrypted)

This prevents you from decrypting more than once.

4
  • I tried that, but it seems like that is being called when the item is saved, thus breaking the app since it's already a decrypted value that's trying to decrypt.
    – Oved D
    Commented Oct 25, 2012 at 21:01
  • Figured out this is because my get_db_prep_value is calling the base method get_db_prep_value, which calls get_prep_value, and in CharField, this is overwritten to call to_python causing this error
    – Oved D
    Commented Oct 25, 2012 at 21:04
  • to_python get's called in the base field method clean() - and as far as I understand this happens when the field is validated before saving. So I'm not sure how to get this not to be called?
    – Oved D
    Commented Oct 25, 2012 at 21:21
  • This works in details views, but not when actually updating the model. I will post my updated code above.
    – Oved D
    Commented Nov 7, 2012 at 5:20
3

You forgot to set the metaclass:

class EncryptedCharField(models.CharField):
    __metaclass__ = models.SubfieldBase

The custom fields documentation explains why this is necessary.

1
  • This causes wierd behavior, as when adding this, to_python always gets called twice, with the original type being lost and a double decryption happens, resulting in an error.
    – Oved D
    Commented Nov 7, 2012 at 5:21
2

Since this question was originally answered, a number of packages have been written to solve this exact problem.

For example, as of 2018, the package django-encrypted-model-fields handles this with a syntax like

from encrypted_model_fields.fields import EncryptedCharField

class MyModel(models.Model):
    encrypted_char_field = EncryptedCharField(max_length=100)
    ...

As a rule of thumb, it's usually a bad idea to roll your own solution to a security challenge when a more mature solution exists out there -- the community is a better tester and maintainer than you are.

2
  • Is it possible to replace existing custom class with above class? if yes then how? I have a project where we need encryption and description but I was unable to find any existing lib so I had to create my own class using github.com/defrex/django-encrypted-fields But now I wish to replace it existing new lib.
    – Avi
    Commented Mar 24, 2022 at 11:08
  • You would need to need to find a way to migrate the data to the new encrypted field. For example 1) add new field as optional 2) create data migration script that for each object reads old field and writes to new field 3) delete the old field and make the new field requried Commented Mar 24, 2022 at 13:51
1

You need to add a to_python method that deals with a number of cases, including passing on an already decrypted value

(warning: snippet is cut from my own code - just for illustration)

def to_python(self, value):
    if not value:
        return
    if isinstance(value, _Param): #THIS IS THE PASSING-ON CASE
        return value
    elif isinstance(value, unicode) and value.startswith('{'):
        param_dict = str2dict(value)
    else:
        try:
            param_dict = pickle.loads(str(value))
        except:
            raise TypeError('unable to process {}'.format(value))
    param_dict['par_type'] = self.par_type
    classname = '{}_{}'.format(self.par_type, param_dict['rule'])
    return getattr(get_module(self.par_type), classname)(**param_dict)

By the way:

Instead of get_db_prep_value you should use get_prep_value (the former is for db specific conversions - see https://docs.djangoproject.com/en/1.4/howto/custom-model-fields/#converting-python-objects-to-query-values )

1
  • done...but this doesn't solve the issue as it's never called. Updated code is above
    – Oved D
    Commented Nov 7, 2012 at 5:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.