4

I have a basic file/folder structure on the Desktop where the "Test" folder contains "Folder 1", which in turn contains 2 subfolders:

  • An "Original files" subfolder which contains shapefiles (.shp).
  • A "Processed files" subfolder which is empty.

I am attempting to write a script which looks into each parent folder (Folder 1, Folder 2 etc) and if it finds an Original Files subfolder, it will run a function and output the results into the Processed files subfolder.

I made a simple diagram to showcase this where if Folder 1 contains the relevant subfolders then the function will run; if Folder 2 does not contain the subfolders then it's simply ignored:

Folder structure

I looked into the following posts but having some trouble:

The following is the script which seems to run happily, annoying thing is that it doesn't produce an error so this real noob can't see where the problem is:

import os, sys

from os.path import expanduser
home = expanduser("~")

for subFolders, files in os.walk(home + "\Test\\" + "\*Original\\"):
 if filename.endswith('.shp'):

    output = home + "\Test\\" + "\*Processed\\" + filename

    # do_some_function, output  
2
  • 1
    why not using "python -m pdb yourscript.py" to see if it did what you want or not? Eclipse/PyDev will let you have a nicer GUI debugging experience.
    – Dyno Fu
    Commented Mar 26, 2015 at 16:17
  • @DynoHongjunFu - Awesome, many thanks for this. I think anything would give me a nicer debugging experience :)
    – Joseph
    Commented Mar 26, 2015 at 16:20

1 Answer 1

2

I guess you mixed something up in your os.walk()-loop.

I just created a simple structure as shown in your question and used this code to get what you're looking for:

root_dir = '/path/to/your/test_dir'
original_dir = 'Original files'
processed_dir = 'Processed files'

for path, subdirs, files in os.walk(root_dir):
    if original_dir in path:
        for file in files:
            if file.endswith('shp'):
                print('original dir: \t' + path)
                print('original file: \t' + path + os.path.sep + file)
                print('processed dir: \t' + os.path.sep.join(path.split(os.path.sep)[:-1]) + os.path.sep + processed_dir)
                print('processed file: ' + os.path.sep.join(path.split(os.path.sep)[:-1]) + os.path.sep + processed_dir + os.path.sep + file)
                print('')

I'd suggest to only use wildcards in a directory-crawling script if you are REALLY sure what your directory tree looks like. I'd rather use the full names of the folders to search for, as in my script.

Update: Paths

Whenever you use paths, take care of your path separators - the slashes.

On windows systems, the backslash is used for that:

C:\any\path\you\name

Most other systems use a normal, forward slash:

/the/path/you/want

In python, a forward slash could be used directly, without any problem:

path_var = '/the/path/you/want'

...as opposed to backslashes. A backslash is a special character in python strings. For example, it's used for the newline-command: \n

To clarify that you don't want to use it as a special character, but as a backslash itself, you either have to "escape" it, using another backslash: '\\'. That makes a windows path look like this:

path_var = 'C:\\any\\path\\you\\name'

...or you could mark the string as a "raw" string (or "literal string") with a proceeding r. Note that by doing that, you can't use special characters in that string anymore.

path_var = r'C:\any\path\you\name'

In your comment, you used the example root_dir = home + "\Test\\". The backslash in this string is used as a special character there, so python tries to make sense out of the backslash and the following character: \T. I'm not sure if that has any meaning in python, but \t would be converted to a tab-stop. Either way - that will not resolve to the path you want to use.

I'm wondering why your other example works. In "C:\Users\me\Test\\", the \U and \m should lead to similar errors. And you also mixed single and double backslashes.

That said...

When you take care of your OS path separators and trying around with new paths now, also note that python does a lot of path-concerning things for you. For example, if your script reads a directory, as os.walk() does, on my windows system the separators are already processed as double backslashes. There's no need for me to check that - it's usually just hardcoded strings, where you'll have to take care.

And finally: The Python os.path module provides a lot of methods to handle paths, seperators and so on. For example, os.path.sep (and os.sep, too) wil be converted in the correct seperator for the system python is running on. You can also build paths using os.path.join().

And finally: The home-directory

You use expanduser("~") to get the home-path of the current user. That should work fine, but if you're using an old python version, there could be a bug - see: expanduser("~") on Windows looks for HOME first

So check if that home-path is resolved correct, and then build your paths using the power of the os-module :-)

Hope that helps!

4
  • Thanks buddy, I will test this out and report back!
    – Joseph
    Commented Mar 30, 2015 at 10:00
  • Awesome, tested your script and works great! Although I did have an issue when using root_dir = home + "\Test\\". Nothing printed. But seems to work when I used the direct path ("C:\Users\me\Test\\"). Any idea as to why? Other than that, fantastic and I will take your wildcard advice into account!
    – Joseph
    Commented Mar 30, 2015 at 10:25
  • 1
    @Joseph: I'm updating my answer and provide some informations about path names. Glad i could help :-)
    – xph
    Commented Mar 30, 2015 at 10:41
  • This is a brilliant updated answer! Many thanks for this as it really clears up the PATH issues. I will play around with the different methods. Hope your answer gets loads of upvotes!
    – Joseph
    Commented Mar 30, 2015 at 11:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.