41

I am trying to use python to create the files needed to run some other software in batch. For part of this I need to produce a text file that loads the needed data files into the software. My problem is that the files I need to enter into this text file are stored in a set of structured folders.

I need to loop over a set of folders (up to 20), which each could contain up to 3 more folders which contain the files I need. The bottom level of the folders contain a set of files needed for each run of the software. The text file should have the path+name of these files printed line by line, add an instruction line and then move to the next set of files from a folder and so on until all of sub level folders have been checked.

3 Answers 3

84

Charles' answer is good, but can be improved upon to increase speed and efficiency. Each item produced by os.walk() (See docs) is a tuple of three items. Those items are:

  1. The working directory
  2. A list of strings naming any sub-directories present in the working directory
  3. A list of files present in the working directory

Knowing this, much of Charles' code can be condensed with the modification of a forloop:

import os

def list_files(dir):
    r = []
    for root, dirs, files in os.walk(dir):
        for name in files:
            r.append(os.path.join(root, name))
    return r
3
  • so what's the diff btwn root and dir? and what should i put for dir? the parent folder of all the sub folders i want to walk through right? this code is not working for me :( returning empty array... Jul 10, 2017 at 23:12
  • @alwaysaskingquestions root is the first item of a tuple generated by os.walk(). dir is the directory of the files you want to iterate through. In my answer's code, the subfolders (indicated by tuple item dirs) are ignored. Only the files and their root directories are used.
    – L. Teder
    Sep 27, 2017 at 14:36
  • 4
    Just use a list comprehension: return [os.path.join(r, n) for r, _, f in os.walk(dir) for n in f].
    – Martijn Pieters
    Sep 24, 2018 at 15:54
34

Use os.walk(). The following will output a list of all files within the subdirectories of "dir". The results can be manipulated to suit you needs:

import os                                                                                                             
                                                                                                                      
def list_files(dir):                                                                                                  
    r = []                                                                                                            
    subdirs = [x[0] for x in os.walk(dir)]                                                                            
    for subdir in subdirs:                                                                                            
        files = os.walk(subdir).next()[2]                                                                             
        if (len(files) > 0):                                                                                          
            for file in files:                                                                                        
                r.append(os.path.join(subdir, file))                                                                         
    return r                                                                                                          

For python 3, change next() to __next__().

1
  • What if the dir path is in ADLS? How can we walk through adls folder using python? Sep 21, 2022 at 10:53
2

This will help to list specific file extension. In my sub-folders i have many files but i am only interested parquet files.

import os
dir = r'/home/output/'
def list_files(dir):
r = []
for root, dirs, files in os.walk(dir):
    for name in files:
        filepath = root + os.sep + name
        if filepath.endswith(".snappy.parquet"):
            r.append(os.path.join(root, name))
return r

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.