189

I often use python to process directories of data. Recently, I have noticed that the default order of the lists has changed to something almost nonsensical. For example, if I am in a current directory containing the following subdirectories: run01, run02, ... run19, run20, and then I generate a list from the following command:

dir = os.listdir(os.getcwd())

then I usually get a list in this order:

dir = ['run01', 'run18', 'run14', 'run13', 'run12', 'run11', 'run08', ... ]

and so on. The order used to be alphanumeric. But this new order has remained with me for a while now.

What is determining the (displayed) order of these lists?

0

14 Answers 14

221

You can use the builtin sorted function to sort the strings however you want. Based on what you describe,

sorted(os.listdir(whatever_directory))

Alternatively, you can use the .sort method of a list:

lst = os.listdir(whatever_directory)
lst.sort()

I think should do the trick.

Note that the order that os.listdir gets the filenames is probably completely dependent on your filesystem.

10
  • 2
    Does not change the order if dealing with number-first filenames (ie 59.9780radps-0096 is still before 9.9746radps-0082). I think it's because everything is a string, so the decimal is not treated properly.
    – Elliot
    Sep 23, 2014 at 20:22
  • 2
    Or use the natsort library, which I just found.
    – Elliot
    Sep 23, 2014 at 20:39
  • 13
    Only sorted(listdir) worked for me. listdir.sort() gave me: TypeError: 'NoneType' object is not iterable
    – paul_h
    Nov 11, 2017 at 10:17
  • 2
    @paul_h -- listdir.sort() won't work for statements like for i in listdir.sort(), because list.sort() method change the order of items in lists IN PLACE, which means process the list itself but won't return anything but None. So you need to use a_list = listdir('some_path'); a_list.sort() then do for i in a_list
    – Sean_Syue
    Jul 14, 2018 at 3:10
  • 1
    Do you know how to change the order to ascending or descending using .sort ?
    – Alex B
    Jul 16, 2018 at 20:56
85

I think the order has to do with the way the files are indexed on your FileSystem. If you really want to make it adhere to some order you can always sort the list after getting the files.

0
60

Per the documentation:

os.listdir(path)

Return a list containing the names of the entries in the directory given by path. The list is in arbitrary order. It does not include the special entries '.' and '..' even if they are present in the directory.

Order cannot be relied upon and is an artifact of the filesystem.

To sort the result, use sorted(os.listdir(path)).

0
58

Python for whatever reason does not come with a built-in way to have natural sorting (meaning 1, 2, 10 instead of 1, 10, 2), so you have to write it yourself:

import re
def sorted_alphanumeric(data):
    convert = lambda text: int(text) if text.isdigit() else text.lower()
    alphanum_key = lambda key: [ convert(c) for c in re.split('([0-9]+)', key) ] 
    return sorted(data, key=alphanum_key)

You can now use this function to sort a list:

dirlist = sorted_alphanumeric(os.listdir(...))

PROBLEMS: In case you use the above function to sort strings (for example folder names) and want them sorted like Windows Explorer does, it will not work properly in some edge cases.
This sorting function will return incorrect results on Windows, if you have folder names with certain 'special' characters in them. For example this function will sort 1, !1, !a, a, whereas Windows Explorer would sort !1, 1, !a, a.

So if you want to sort exactly like Windows Explorer does in Python you have to use the Windows built-in function StrCmpLogicalW via ctypes (this of course won't work on Unix):

from ctypes import wintypes, windll
from functools import cmp_to_key

def winsort(data):
    _StrCmpLogicalW = windll.Shlwapi.StrCmpLogicalW
    _StrCmpLogicalW.argtypes = [wintypes.LPWSTR, wintypes.LPWSTR]
    _StrCmpLogicalW.restype  = wintypes.INT

    cmp_fnc = lambda psz1, psz2: _StrCmpLogicalW(psz1, psz2)
    return sorted(data, key=cmp_to_key(cmp_fnc))

This function is slightly slower than sorted_alphanumeric().

Bonus: winsort can also sort full paths on Windows.

Alternatively, especially if you use Unix, you can use the natsort library (pip install natsort) to sort by full paths in a correct way (meaning subfolders at the correct position).

You can use it like this to sort full paths:

from natsort import natsorted, ns
dirlist = natsorted(dirlist, alg=ns.PATH | ns.IGNORECASE)

Starting with version 7.1.0 natsort supports os_sorted which internally uses either the beforementioned Windows API or Linux sorting and should be used instead of natsorted().

4
  • Works perfectly fine. print( sorted_aphanumeric(["1", "10", "2", "foo_10", "foo_8"]) ) -> ['1', '2', '10', 'foo_8', 'foo_10']. Exactly as expected.
    – user136036
    Jan 20, 2020 at 13:04
  • There is a longstanding open issue on natsorted to get Windows Explorer matching functionality implemented. Perhaps you should contribute a solution? github.com/SethMMorton/natsort/issues/41 Apr 1, 2020 at 14:54
  • The winsort function was exactly what I needed :) Mar 6, 2021 at 15:40
  • worked great on files name finishing a _1, _2,..._10, _11, etc
    – Antony P.
    Feb 6 at 21:05
22

I think by default the order is determined with the ASCII value. The solution to this problem is this

dir = sorted(os.listdir(os.getcwd()), key=len)
1
  • 2
    None of the above worked for me, that "key-len" seemed to be the last remaining trick, thanks so much.
    – Spider999
    Feb 20, 2021 at 11:33
10

Use natsort library:

Install the library with the following command for Ubuntu and other Debian versions

Python 2

sudo pip install natsort

Python 3

sudo pip3 install natsort

Details of how to use this library is found here

from natsort import natsorted

files = ['run01', 'run18', 'run14', 'run13', 'run12', 'run11', 'run08']
natsorted(files)

[out]:
['run01', 'run08', 'run11', 'run12', 'run13', 'run14', 'run18']
  • This is not a duplicate of answer. natsort was added as an edit on 2020-01-27.
1
  • 2
    That is more accurate than sorted()! Thanks Nov 7, 2019 at 15:11
5
aaa = ['row_163.pkl', 'row_394.pkl', 'row_679.pkl', 'row_202.pkl', 'row_1449.pkl', 'row_247.pkl', 'row_1353.pkl', 'row_749.pkl', 'row_1293.pkl', 'row_1304.pkl', 'row_78.pkl', 'row_532.pkl', 'row_9.pkl', 'row_1435.pkl']                                                                                                                                                                                                                                                                                                 
sorted(aaa, key=lambda x: int(os.path.splitext(x.split('_')[1])[0]))

As In case of mine requirement I have the case like row_163.pkl here os.path.splitext('row_163.pkl') will break it into ('row_163', '.pkl') so need to split it based on '_' also.

but in case of your requirement you can do something like

sorted(aa, key = lambda x: (int(re.sub('\D','',x)),x))

where

aa = ['run01', 'run08', 'run11', 'run12', 'run13', 'run14', 'run18']

and also for directory retrieving you can do sorted(os.listdir(path))

and for the case of like 'run01.txt' or 'run01.csv' you can do like this

sorted(files, key=lambda x : int(os.path.splitext(x)[0]))
1
  • Unarguably best answer here.
    – Amit Amola
    Aug 17, 2020 at 11:06
5

It's probably just the order that C's readdir() returns. Try running this C program:

#include <dirent.h>
#include <stdio.h>

int main(void){
   DIR *dirp;
   struct dirent* de;
   dirp = opendir(".");
   while(de = readdir(dirp)) // Yes, one '='.
        printf("%s\n", de->d_name);
   closedir(dirp);
   return 0;
}

The build line should be something like gcc -o foo foo.c.

P.S. Just ran this and your Python code, and they both gave me sorted output, so I can't reproduce what you're seeing.

1
  • 1
    The reason that you're seeing soted output may depend on a lot of factors, such as OS, filesystem, time of creation of files, actions during the last defragmentation, ... Jan 27, 2011 at 7:34
3

From the documentation:

The list is in arbitrary order, and does not include the special entries '.' and '..' even if they are present in the directory.

This means that the order is probably OS/filesystem dependent, has no particularly meaningful order, and is therefore not guaranteed to be anything in particular. As many answers mentioned: if preferred, the retrieved list can be sorted.

Cheers :)

0
3

The proposed combination of os.listdir and sorted commands generates the same result as ls -l command under Linux. The following example verifies this assumption:

user@user-PC:/tmp/test$ touch 3a 4a 5a b c d1 d2 d3 k l p0 p1 p3 q 410a 409a 408a 407a
user@user-PC:/tmp/test$ ls -l
total 0
-rw-rw-r-- 1 user user 0 Feb  15 10:31 3a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 407a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 408a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 409a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 410a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 4a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 5a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 b
-rw-rw-r-- 1 user user 0 Feb  15 10:31 c
-rw-rw-r-- 1 user user 0 Feb  15 10:31 d1
-rw-rw-r-- 1 user user 0 Feb  15 10:31 d2
-rw-rw-r-- 1 user user 0 Feb  15 10:31 d3
-rw-rw-r-- 1 user user 0 Feb  15 10:31 k
-rw-rw-r-- 1 user user 0 Feb  15 10:31 l
-rw-rw-r-- 1 user user 0 Feb  15 10:31 p0
-rw-rw-r-- 1 user user 0 Feb  15 10:31 p1
-rw-rw-r-- 1 user user 0 Feb  15 10:31 p3
-rw-rw-r-- 1 user user 0 Feb  15 10:31 q

user@user-PC:/tmp/test$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.listdir( './' )
['d3', 'k', 'p1', 'b', '410a', '5a', 'l', 'p0', '407a', '409a', '408a', 'd2', '4a', 'p3', '3a', 'q', 'c', 'd1']
>>> sorted( os.listdir( './' ) )
['3a', '407a', '408a', '409a', '410a', '4a', '5a', 'b', 'c', 'd1', 'd2', 'd3', 'k', 'l', 'p0', 'p1', 'p3', 'q']
>>> exit()
user@user-PC:/tmp/test$ 

So, for someone who wants to reproduce the result of the well-known ls -l command in their python code, sorted( os.listdir( DIR ) ) works pretty well.

0
2

I found "sort" does not always do what I expected. eg, I have a directory as below, and the "sort" give me a very strange result:

>>> os.listdir(pathon)
['2', '3', '4', '5', '403', '404', '407', '408', '410', '411', '412', '413', '414', '415', '416', '472']
>>> sorted([ f for f in os.listdir(pathon)])
['2', '3', '4', '403', '404', '407', '408', '410', '411', '412', '413', '414', '415', '416', '472', '5']

It seems it compares the first character first, if that is the biggest, it would be the last one.

3
  • 2
    This is expected behavior. ('5' > '403') is True.
    – AXO
    Dec 1, 2015 at 18:16
  • 2
    @AXO is correct, because at this point you're comparing the alphanumeric sort, not quantitative values of the numbers. In order to get a sort similar to your expectation, you may want to use number padding on your folders... ['002', '003', '004', '005', '403', '404', '405', '406']
    – Andrew
    Sep 8, 2017 at 17:32
  • Don't use padding, just use a string to number conversion as sorting key: sorted(os.listdir(pathon), key=int) will properly return ['2', '3', '4', '5', '403', ...].
    – Guimoute
    Mar 3 at 23:33
1
In [6]: os.listdir?

Type:       builtin_function_or_method
String Form:<built-in function listdir>
Docstring:
listdir(path) -> list_of_strings
Return a list containing the names of the entries in the directory.
path: path of directory to list
The list is in **arbitrary order**.  It does not include the special
entries '.' and '..' even if they are present in the directory.
4
  • 1
    This explains why they are seeing the behaviour, without offering a solution. Feb 21, 2013 at 13:39
  • 2
    OP just want to know why, not how.
    – Denis
    Feb 21, 2013 at 13:41
  • @Denis thanks for pointing this out - I didn't notice it before
    – Dimitris
    Feb 21, 2013 at 13:41
  • @DanielWatkins OK, Not it isnt.)
    – Denis
    Feb 21, 2013 at 13:43
1

To answer the question directly, you can use the following code.

dir = ['run01', 'run18', 'run14', 'run13', 'run12', 'run11', 'run08']
for file in sorted(dir, key=lambda x:int(x.replace('run', ''))):
    print(file)

It will print:

run01
run08
run11
run12
run13
run14
run18

This approach uses the Python built-in method sorted, and, through the key argument, it specifies the sorting criterium, that is, the list item without 'run' casted to an integer.

0

ls by default previews the files sorted by name. (ls options can be used to sort by date, size, ...)

files = list(os.popen("ls"))
files = [file.strip("\n") for file in files]

Using ls would have much better performance when the directory contains so many files.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.