3

I want to check if class c uses a certain module m. I am able to get two things: - the module's usage in that file(f), - the starting line number of the classes in file f.

However, in order to know whether a particular class uses the module, I need to know the starting and end line of class c. I don't know how to get the end line of class c.

I tried going through the documentation of ast but could not find any method of finding the scope of a class.

My current code is:

source_code = "car"
source_code_data = pyclbr.readmodule(source_code)

This gives the following in source_code_data variable: source_code_data value in debugger mode1

As you can see in the image, there is lineno but no end lineno to denote the end line of a class.

I expect to get the end line of the class in order to know its scope and finally knowing the usage of the module. Currently, the module has this structure:

Module: vehicles used in file: /Users/aviralsrivastava/dev/generate_uml/inheritance_and_dependencies/car.py at: [('vehicles.Vehicle', 'Vehicle', 5), ('vehicles.Vehicle', 'Vehicle', 20)]

So, with this information, I will be able to know if a module is used in a range: used in a class.

My whole code base is:

import ast
import os
import pyclbr
import subprocess
import sys
from operator import itemgetter

from dependency_collector import ModuleUseCollector
from plot_uml_in_excel import WriteInExcel


class GenerateUML:
    def __init__(self):
        self.class_dict = {}  # it will have a class:children mapping.

    def show_class(self, name, class_data):
        print(class_data)
        self.class_dict[name] = []
        self.show_super_classes(name, class_data)

    def show_methods(self, class_name, class_data):
        methods = []
        for name, lineno in sorted(class_data.methods.items(),
                                   key=itemgetter(1)):
            # print('  Method: {0} [{1}]'.format(name, lineno))
            methods.append(name)
        return methods

    def show_super_classes(self, name, class_data):
        super_class_names = []
        for super_class in class_data.super:
            if super_class == 'object':
                continue
            if isinstance(super_class, str):
                super_class_names.append(super_class)
            else:
                super_class_names.append(super_class.name)
        for super_class_name in super_class_names:
            if self.class_dict.get(super_class_name, None):
                self.class_dict[super_class_name].append(name)
            else:
                self.class_dict[super_class_name] = [name]
        # adding all parents for a class in one place for later usage: children
        return super_class_names

    def get_children(self, name):
        if self.class_dict.get(name, None):
            return self.class_dict[name]
        return []


source_code = "car"
source_code_data = pyclbr.readmodule(source_code)
print(source_code_data)
generate_uml = GenerateUML()
for name, class_data in sorted(source_code_data.items(), key=lambda x: x[1].lineno):
    print(
        "Class: {}, Methods: {}, Parent(s): {}, File: {}".format(
            name,
            generate_uml.show_methods(
                name, class_data
            ),
            generate_uml.show_super_classes(name, class_data),
            class_data.file
        )
    )
print('-----------------------------------------')
# create a list with all the data
# the frame of the list is: [{}, {}, {},....] where each dict is: {"name": <>, "methods": [], "children": []}
agg_data = []
files = {}
for name, class_data in sorted(source_code_data.items(), key=lambda x: x[1].lineno):
    methods = generate_uml.show_methods(name, class_data)
    children = generate_uml.get_children(name)
    # print(
    #     "Class: {}, Methods: {}, Child(ren): {}, File: {}".format(
    #         name,
    #         methods,
    #         children,
    #         class_data.file
    #     )
    # )
    agg_data.append(
        {
            "Class": name,
            "Methods": methods,
            "Children": children,
            "File": class_data.file
        }
    )
    files[class_data.file] = name
print('-----------------------------------------')
# print(agg_data)


for data_index in range(len(agg_data)):
    agg_data[data_index]['Dependents'] = None
    module = agg_data[data_index]["File"].split('/')[-1].split('.py')[0]
    used_in = []
    for file_ in files.keys():
        if file_ == agg_data[data_index]["File"]:
            continue
        collector = ModuleUseCollector(module)
        source = open(file_).read()
        collector.visit(ast.parse(source))
        print('Module: {} used in file: {} at: {}'.format(
            module, file_, collector.used_at))
        if len(collector.used_at):
            used_in.append(files[file_])
    agg_data[data_index]['Dependents'] = used_in
'''
# checking the dependencies
dependencies = []
for data_index in range(len(agg_data)):
    collector = ModuleUseCollector(
        agg_data[data_index]['File'].split('/')[-1].split('.py')[0]
    )
    collector.visit(ast.parse(source))
    dependencies.append(collector.used_at)

agg_data[source_code_index]['Dependencies'] = dependencies
# next thing, for each class, find the dependency in each class.

'''
print('------------------------------------------------------------------')
print('FINAL')
for data in agg_data:
    print(data)
    print('-----------')
    print('\n')

# The whole data is now collected and we need to form the dataframe of it:
'''
write_in_excel = WriteInExcel(file_name='dependency_1.xlsx')
df = write_in_excel.create_pandas_dataframe(agg_data)
write_in_excel.write_df_to_excel(df, 'class_to_child_and_dependents')
'''
'''
print(generate_uml.class_dict)
write_in_excel = WriteInExcel(
    classes=generate_uml.class_dict, file_name='{}.xlsx'.format(source_code))
write_in_excel.form_uml_sheet_for_classes()
'''

2 Answers 2

1

Edit: Invasive Method

I did manage to find a way to do this with pyclbr but it involves changing parts of the source code. Essentially you make the stack (which is normally a list) a custom class. When an item is removed from the stack (when its scope has ended) the ending line number is added. I tried to make it as un invasive as possible.

First define a stack class in the pyclbr module:

class Stack(list):

    def __init__(self):
        import inspect
        super().__init__()

    def __delitem__(self, key):
        frames = inspect.stack()
        setattr(self[key][0], 'endline', frames[1].frame.f_locals['start'][0] - 1)
        super().__delitem__(key)

Then you change the stack in the _create_tree function which is originally on line 193:

stack = Stack()

Now you can access the ending line as well by using

class_data.endline



I was not able to find a solution that uses the libraries you are using. So I wrote a simple parser that finds the start and end lines of every class contained in a module. You must pass in either the text wrapper itself or the list created by readlines() on your file. Its also worth noting that if a class continues until the end of the file then the end is -1.

import re

def collect_classes(source):
    classes = {}
    current_class = None
    for lineno, line in enumerate(source, start=1):
        if current_class and not line.startswith(' '):
            if line != '\n':
                classes[current_class]['end'] = lineno - 1
                current_class = None

        if line.startswith('class'):
                current_class = re.search(r'class (.+?)(?:\(|:)', line).group(1)
                classes[current_class] = {'start': lineno}

    if current_class:
        classes[current_class]['end'] = -1

    return classes

import datetime
file = open(datetime.__file__, 'r') # opening the source of datetime for a test
scopes = collect_classes(file)
print(scopes)

This outputs:

{'timedelta': {'start': 454, 'end': 768}, 'date': {'start': 774, 'end': 1084}, 'tzinfo': {'start': 1092, 'end': 1159}, 'time': {'start': 1162, 'end': 1502}, 'datetime': {'start': 1509, 'end': 2119}, 'timezone': {'start': 2136, 'end': 2251}}

2
  • Glad I could help! Which method did you end up using? Sep 28, 2019 at 23:37
  • Invasive method. It made more sense to me. I was done with my own implementation but found yours better. Sep 29, 2019 at 0:09
1

Thanks! The endline is really a missing feature there.

The discussion on this led to an alternative (still invasive) method to achieve this but that is almost a one-liner that does not require to subclass list and override the del operator:

https://github.com/python/cpython/pull/16466#issuecomment-539664180

Basically, just add

stack[-1].endline = start - 1

before all del stack[-1] in pyclbr.py

3
  • that didn't work. start is a tuple, isn't it? also, stack does not have an attribute endline. Apologies for being too dumb here but if you could help me out with the correct(exact) code, that'd be a great help. Feb 5, 2020 at 15:50
  • Ah yes, you are right start is a tuple containing (lineno, indent) So just use stack[-1].endline = start[0] - 1 In Python, you are allowed to add an attribute to object during runtime even if this is frowned upon. To be nice and square, you should also modify the __init__ functions of the structures where it is relevant. I think just Object ? Sorry on my side I ended up using a different, non-invasive method (re-walk the whole ast and add the last lineno of their last desendant to each module) so I can not test this code right now. Feb 7, 2020 at 2:34
  • 1
    Made the changes: github.com/python/cpython/pull/16466/files Feb 8, 2020 at 2:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.