0

i'm working with openpyxl on a .xlsx file which has around 10K products, of which some are "regular items" and some are products that need to be ordered when required. For the project I'm doing I would like to delete all of the rows containing the items that need to be ordered.

I tested this with a small sample size of the actual workbook and did have the code working the way I wanted to. However when I tried this in the actual workbook with 10K rows it seems to be taking forever to delete those rows (it has been running for nearly and hour now).

Here's the code that I used:

wb = openpyxl.load_workbook('prod.xlsx')
sheet = wb.get_sheet_by_name('Sheet1')
def clean_workbook():
    for row in sheet:
        for cell in row:
            if cell.value == 'ordered':
                sheet.delete_rows(cell.row)

I would like to know is there a faster way of doing this with some tweaks in my code? Or is there a better way to just read just the regular stock from the workbook without deleting the unwanted items?

2
  • Rather than looping through each of the cells for the row, couldn't you just jump to the column/cell that indicates the status? You'll save a lot of time there. Nov 26, 2020 at 20:52
  • You shouldn't delete rows while looping over them as this will cause problems. Nov 27, 2020 at 9:05

2 Answers 2

1

Deleting rows in loops can be slow because openpyxl has to update all the cells below the row being deleted. Therefore, you should do this as little as possible. One way is to collect a list of row numbers, check for contiguous groups and then delete using this list from the bottom.

A better approach might be to loop through ws.values and write to a new worksheet filtering out the relevant rows. Copy any other relevant data such as formatting, etc. Then you can delete the original worksheet and rename the new one.

ws1 = wb['My Sheet']
ws2 = wb.create_sheet('My Sheet New')

for row in ws1.values:
    if row[x] == "ordered": # we can assume this is always the same column
       continue
    ws2.append(row)

del wb["My Sheet"]
ws2.title = "My Sheet"

For more sophisticated filtering you will probably want to load the values into a Pandas dataframe, make the changes and then write to a new sheet.

1
  • Thanks Charlie, got it working nice and fast with this tip!
    – MikkoBoy
    Nov 27, 2020 at 17:16
0

You can open with read-only mode, and import all content into a list, then modify in list is always a lot more faster than working in excel. After you modify the list, made a new worksheet and upload your list back to excel. I did this way with my 100k items excel .

5
  • Yes, but what if there are other worksheets in the file that need preserving. Nov 27, 2020 at 9:06
  • You can still apply the same concept, the different is, you create a new worksheet in current file, and use write-only mode to import list to this worksheet, then the rest of other worksheet can preserve.
    – Kai
    Nov 27, 2020 at 10:24
  • And also, the whole process involve read-only, write-only modes and also create a new worksheet, or delete current one, so you have to open and close excel in separate times. But the whole process is faster as you are dealing a large data set in list.
    – Kai
    Nov 27, 2020 at 10:29
  • Depends a lot on the workbook. Combining read-only and write-only modes will minimise memory use but will lose macros, charts, formatting, etc. Nov 27, 2020 at 16:24
  • yes, you are right, so this way is only for raw data.
    – Kai
    Nov 27, 2020 at 23:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.