19

As a project grows, so do dependencies and event chains, especially in overridden save() methods and post_save and pre_save signals.

Example:

An overridden A.save creates two related objects to A - B and C. When C is saved, the post_save signal is invoked that does something else, etc...

How can these event chins be made more clear? Is there a way to visualize (generate automatically) such chains/flows? I'm not looking for ERD nor a Class diagram. I need to be sure that doing one thing one place won't affect something on the other side of the project, so simple visualization would be the best.

EDIT

To be clear, I know that it would be almost impossible to check dynamically generated signals. I just want to check all (not dynamically generated) post_save, pre_save, and overridden save methods and visualize them so I can see immediately what is happening and where when I save something.

5 Answers 5

10

This is not the full solution, but I hope it can be a good starting point. Consider this code:

from django.db import models
from django.db.models.signals import pre_save
from django.dispatch import receiver

class A(models.Model):
    def save(self, *args, **kwargs):
        if not self.pk:
            C.objects.create()

class B(models.Model):
    pass

class C(models.Model):
    b = models.ForeignKey(B, on_delete=models.CASCADE, blank=True)

@receiver(pre_save, sender=C)
def pre_save_c(sender, instance, **kwargs):
    if not instance.pk:
        b = B.objects.create()
        instance.b = b

We can get the dependencies for the app name list using inspect, django get_models(), and signals in this manner:

import inspect
import re
from collections import defaultdict

from django.apps import apps
from django.db.models import signals

RECEIVER_MODELS = re.compile('sender=(\w+)\W')
SAVE_MODELS = re.compile('(\w+).objects.')

project_signals = defaultdict(list)
for signal in vars(signals).values():
    if not isinstance(signal, signals.ModelSignal):
        continue
    for _, receiver in signal.receivers:
        rcode = inspect.getsource(receiver())
        rmodel = RECEIVER_MODELS.findall(rcode)
        if not rmodel:
            continue
        auto_by_signals = [
            '{} auto create -> {}'.format(rmodel[0], cmodel)
            for cmodel in SAVE_MODELS.findall(rcode)
        ]
        project_signals[rmodel[0]].extend(auto_by_signals)

for model in apps.get_models():
    is_self_save = 'save' in model().__class__.__dict__.keys()
    if is_self_save:
        scode = inspect.getsource(model.save)
        model_name = model.__name__
        for cmodel in SAVE_MODELS.findall(scode):
            print('{} auto create -> {}'.format(model_name, cmodel))
            for smodels in project_signals.get(cmodel, []):
                print(smodels)

This gives:

A auto create -> C
C auto create -> B

Updated: change method to found overridden save by the instance class dict.

is_self_save = 'save' in model().__class__.__dict__.keys()
1
  • 1
    Interesting ! A simpler solution, more limited but possible more general, could be a script which lists all Models either overriding save() or sending a pre_save/post_save signal Commented Apr 11, 2019 at 11:08
6
+100

(Too long to fit into a comment, lacking code to be a complete answer)

I can't mock up a ton of code right now, but another interesting solution, inspired by Mario Orlandi's comment above, would be some sort of script that scans the whole project and searches for any overridden save methods and pre and post save signals, tracking the class/object that creates them. It could be as simple as a series of regex expressions that look for class definitions followed by any overridden save methods inside.

Once you have scanned everything, you could use this collection of references to create a dependency tree (or set of trees) based on the class name and then topologically sort each one. Any connected components would illustrate the dependencies, and you could visualize or search these trees to see the dependencies in a very easy, natural way. I am relatively naive in django, but it seems you could statically track dependencies this way, unless it is common for these methods to be overridden in multiple places at different times.

0

If you only want to track models saves, and not interested in other things happening inside overridden save methods and signals, you can use a mechanism like angio. You can register a global post_save receiver, without a sender argument, one that will be called for all model saves, and print the saved model name in that function. Then, write a script to just call save for all existing models. Something like the following could work:

@receiver(models.signals.post_save)
def global_post_save(sender, instance, created, *args, **kwargs):
    print(' --> ' + str(sender.__name__))

from django.apps import apps
for model in apps.get_models():
    instance = model.objects.first()
    if instance:
        print('Saving ' + str(model.__name__))
        instance.save()
        print('\n\n')

With the following model structure;

class A(models.Model):
    ...
    def save(self, *args, **kwargs):
        B.objects.create()

@receiver(post_save, sender=B)
def post_save_b(sender, instance, **kwargs):
    C.objects.create()

The script would print:

Saving A
 --> A
 --> B
 --> C

Saving B
 --> B
 --> C

Saving C
 --> C

This is just a basic sketch of what could be done, and can be improved according to the structure of your application. This assumes you already have an entry in the db for each model. Though not changing anything, this approach also saves things in the database, so would be better run on a test db.

0

Assuming your ultimate goal is to track changes in database when an instance of some model is saved, one potential solution can be scanning database for changes instead of source code. The upside of this approach is that it can also cover dynamic code. And downside is, obviously, it will ONLY cover database changes.

This can be achieved using simple testing techniques. Assuming following models..

from django.db import models
from django.db.models.signals import pre_save, post_save
from django.dispatch import receiver


class B(models.Model):
    def save(self, *args, **kwargs):
        X.objects.create()
        super().save(*args, **kwargs)


class C(models.Model):
    y = models.OneToOneField('Y', on_delete=models.CASCADE)


class D(models.Model):
    pass


class X(models.Model):
    pass


class Y(models.Model):
    related = models.ForeignKey('Z', on_delete=models.CASCADE)


class Z(models.Model):
    pass


@receiver(pre_save, sender=D)
def pre_save_d(*args, instance, **kwargs):
    Z.objects.create()


@receiver(post_save, sender=C)
def pre_save_c(*args, instance, **kwargs):
    Y.objects.create(related=Z.objects.create())

I can write up a test case which takes count all database instances, creates an instance of a model, takes count again and calculates the difference. Database instances can be created using factories like mommy. Here is a simple but working example of this technique.

class TestModelDependency(TestCase):
    def test_dependency(self):
        models = apps.get_models()
        models = [model for model in models if model._meta.app_label == 'model_effects']

        for model in models:
            kwargs = self.get_related_attributes(model)

            initial_count = self.take_count(models)
            mommy.make(model, **kwargs)
            final_count = self.take_count(models)

            diff = self.diff(initial_count, final_count)

            print(f'Creating {model._meta.model_name}')
            print(f'Created {" | ".join(f"{v} instance of {k}" for k, v in diff.items())}')

            call_command('flush', interactive=False)

    @staticmethod
    def take_count(models):
        return {model._meta.model_name: model.objects.count() for model in models}

    @staticmethod
    def diff(initial, final):
        result = dict()
        for k, v in final.items():
            i = initial[k]
            d = v - i
            if d != 0:
                result[k] = d
        return result

    @staticmethod
    def get_related_attributes(model):
        kwargs = dict()
        for field in model._meta.fields:
            if any(isinstance(field, r) for r in [ForeignKey, OneToOneField]):
                kwargs[field.name] = mommy.make(field.related_model)
        return kwargs

And my output is

Creating b
Created 1 instance of b | 1 instance of x
Creating c
Created 1 instance of c | 1 instance of y | 1 instance of z
Creating d
Created 1 instance of d | 1 instance of z
Creating x
Created 1 instance of x
Creating y
Created 1 instance of y
Creating z
Created 1 instance of z

For large applications it can be slow, but I use in memory sqlite database for testing and it runs pretty fast.

0

I'working in a Django app that does something similar, but while I get it done, I will comment about the use case you've presented here:

I need to be sure that doing one thing one place won't affect something on the other side of the project ...

You surely could write tests with some dummy signal handlers in order to know if the execution of certain code, triggers unwanted behavior, for instance:

# I use pytest, put this example is suitable also for 
# django's TestCase and others
class TestSome:

    # For Django TestCase this would be setUp
    def setup_method(self, test_method):

        self.singals_info = []

        def dummy_handler(*args, **kwargs):
            # collect_info is a function you must implement, it would
            # gather info about signal, sender, instance, etc ... and
            # save that info in (for example) self.signals_info.
            # You can then use that info for test assertions.
            self.collect_info(*args, **kwargs)

        # connect your handler to every signal you want to control
        post_save.connect(dummy_handler)


    def test_foo():
         # Your normal test here ...
         some_value = some_tested_function()

         # Check your signals behave 
         assert self.signals_behave(self.signals_info)

Why this is better than having a script that show's the events chain?

Well, as you say, when the need for things like this emerges, is because the size of the project is very big, and if you use a tool like you are asking for, you can end with a result like this:

Save A -> Creates B -> Creates C
Save B -> Creates D
Save B -> Creates C
.
.
.
# Imagine here 3 or 4 more lines.

You will end up solving a puzzle every time you want to add some code, that saves/modify something.

However ...

It would be better, to write your code, and then, some test fails (resolving the puzzle for you) and showing to you exactly where will your code miss-behave.

Conclusion:

Implements those tests and your life will be easier.

Best scenario using tests: Write your code, and if no test fail, you're ready to tackle your next programming task.

Worst scenario using tests: Write your code, some test fail, as you know where exactly your code broke, just fix it.

Best scenario using the tool: Analyze the tool output, write your code, everything is ok.

Worst scenario using the tool: Analyze the tool output, write your code, something fails, repeat until all ok.

So, a tool like that would be helpful? Of course, but is not the right tool to ensure things are well, use tests for that.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.