19

There is large python project where one attribute of one class just have wrong value in some place.

It should be sqlalchemy.orm.attributes.InstrumentedAttribute, but when I run tests it is constant value, let's say string.

There is some way to run python program in debug mode, and run some check (if variable changed type) after each step throught line of code automatically?

P.S. I know how to log changes of attribute of class instance with help of inspect and property decorator. Possibly here I can use this method with metaclasses...

But sometimes I need more general and powerfull solution...

Thank you.

P.P.S. I need something like there: https://stackoverflow.com/a/7669165/816449, but may be with more explanation of what is going on in that code.

0

6 Answers 6

13

Well, here is a sort of slow approach. It can be modified for watching for local variable change (just by name). Here is how it works: we do sys.settrace and analyse the value of obj.attr each step. The tricky part is that we receive 'line' events (that some line was executed) before line is executed. So, when we notice that obj.attr has changed, we are already on the next line and we can't get the previous line frame (because frames aren't copied for each line, they are modified ). So on each line event I save traceback.format_stack to watcher.prev_st and if on the next call of trace_command value has changed, we print the saved stack trace to file. Saving traceback on each line is quite an expensive operation, so you'd have to set include keyword to a list of your projects directories (or just the root of your project) in order not to watch how other libraries are doing their stuff and waste cpu.

watcher.py

import traceback

class Watcher(object):
    def __init__(self, obj=None, attr=None, log_file='log.txt', include=[], enabled=False):
        """
            Debugger that watches for changes in object attributes
            obj - object to be watched
            attr - string, name of attribute
            log_file - string, where to write output
            include - list of strings, debug files only in these directories.
               Set it to path of your project otherwise it will take long time
               to run on big libraries import and usage.
        """

        self.log_file=log_file
        with open(self.log_file, 'wb'): pass
        self.prev_st = None
        self.include = [incl.replace('\\','/') for incl in include]
        if obj:
            self.value = getattr(obj, attr)
        self.obj = obj
        self.attr = attr
        self.enabled = enabled # Important, must be last line on __init__.

    def __call__(self, *args, **kwargs):
        kwargs['enabled'] = True
        self.__init__(*args, **kwargs)

    def check_condition(self):
        tmp = getattr(self.obj, self.attr)
        result = tmp != self.value
        self.value = tmp
        return result

    def trace_command(self, frame, event, arg):
        if event!='line' or not self.enabled:
            return self.trace_command
        if self.check_condition():
            if self.prev_st:
                with open(self.log_file, 'ab') as f:
                    print >>f, "Value of",self.obj,".",self.attr,"changed!"
                    print >>f,"###### Line:"
                    print >>f,''.join(self.prev_st)
        if self.include:
            fname = frame.f_code.co_filename.replace('\\','/')
            to_include = False
            for incl in self.include:
                if fname.startswith(incl):
                    to_include = True
                    break
            if not to_include:
                return self.trace_command
        self.prev_st = traceback.format_stack(frame)
        return self.trace_command
import sys
watcher = Watcher()
sys.settrace(watcher.trace_command)

testwatcher.py

from watcher import watcher
import numpy as np
import urllib2
class X(object):
    def __init__(self, foo):
        self.foo = foo

class Y(object):
    def __init__(self, x):
        self.xoo = x

    def boom(self):
        self.xoo.foo = "xoo foo!"
def main():
    x = X(50)
    watcher(x, 'foo', log_file='log.txt', include =['C:/Users/j/PycharmProjects/hello'])
    x.foo = 500
    x.goo = 300
    y = Y(x)
    y.boom()
    arr = np.arange(0,100,0.1)
    arr = arr**2
    for i in xrange(3):
        print 'a'
        x.foo = i

    for i in xrange(1):
        i = i+1

main()
2
  • Yes, this is very slow, but still faster than manual pdb, thank you.
    – Bunyk
    Commented Nov 16, 2012 at 9:37
  • yeah, fixed. By the way, if you are ok with the next line instead of the actual, it can be done in a much more quicker way, check this out: gist.github.com/4086770 it shows either next line or the actual one, depending on whether line event follows the line event Commented Nov 16, 2012 at 12:07
10

There's a very simple way to do this: use watchpoints.

Basically you only need to do

from watchpoints import watch
watch(your_object.attr)

That's it. Whenever the attribute is changed, it will print out the line that changed it and how it's changed. Super easy to use.

It also has more advanced features, for example, you can call pdb when the variable is changed, or use your own callback functions instead of print it to stdout.

2

A simpler way to watch for an object's attribute change (which can also be a module-level variable or anything accessible with getattr) would be to leverage hunter library, a flexible code tracing toolkit. To detect state changes we need a predicate which can look like the following:

import traceback


class MutationWatcher:

    def __init__(self, target, attrs):
        self.target = target
        self.state = {k: getattr(target, k) for k in attrs}

    def __call__(self, event):
        result = False
        for k, v in self.state.items():
            current_value = getattr(self.target, k)
            if v != current_value:
                result = True
                self.state[k] = current_value
                print('Value of attribute {} has chaned from {!r} to {!r}'.format(
                    k, v, current_value))

        if result:
            traceback.print_stack(event.frame)

        return result

Then given a sample code:

class TargetThatChangesWeirdly:
    attr_name = 1


def some_nested_function_that_does_the_nasty_mutation(obj):
    obj.attr_name = 2


def some_public_api(obj):
    some_nested_function_that_does_the_nasty_mutation(obj)

We can instrument it with hunter like:

# or any other entry point that calls the public API of interest
if __name__ == '__main__':
    obj = TargetThatChangesWeirdly()

    import hunter
    watcher = MutationWatcher(obj, ['attr_name'])
    hunter.trace(watcher, stdlib=False, action=hunter.CodePrinter)

    some_public_api(obj)

Running the module produces:

Value of attribute attr_name has chaned from 1 to 2
  File "test.py", line 44, in <module>
    some_public_api(obj)
  File "test.py", line 10, in some_public_api
    some_nested_function_that_does_the_nasty_mutation(obj)
  File "test.py", line 6, in some_nested_function_that_does_the_nasty_mutation
    obj.attr_name = 2
                                 test.py:6     return        obj.attr_name = 2
                                               ...       return value: None

You can also use other actions that hunter supports. For instance, Debugger which breaks into pdb (debugger on an attribute change).

2

Try using __setattr__ to override the function that is called when an attribute assignment is attempted. Documentation for __setattr__

1
  • There is one thing to note here - this works only with attributes of instances of class that defines __setattr__. To use it with class attributes we need to redefine metaclass for class, and who knows what magic we need to make it work with variables defined in module.
    – Bunyk
    Commented Jan 14, 2016 at 22:22
1

You can use the python debugger module (part of the standard library)

To use, just import pdb at the top of your source file:

import pdb

and then set a trace wherever you want to start inspecting the code:

pdb.set_trace()

You can then step through the code with n, and investigate the current state by running python commands.

2
  • 2
    Sorry, I forgot to add, I want this think to work automatically. So I start debugger, give it my condition, for example type(some.module.SomeClass.my_attribute) == str), and find first line where condition doesn't met. And there are millions lines of code, and I don't know where variable is changed.
    – Bunyk
    Commented Nov 15, 2012 at 18:11
  • this solution would require to debug each row of the code. What if you have thousands of rows of code?
    – Angelo
    Commented Nov 16, 2021 at 21:14
1
def __setattr__(self, name, value):
    if name=="xxx":
       util.output_stack('xxxxx')
    super(XXX, self).__setattr__(name, value)

This sample code helped me.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.