0

There are 10,000 files in a folder. Few files are created on 2018-06-01, few on 2018-06-09, like that.

I need to find all files which are created on 2018-06-09. But it is taking to much time (almost 2 hours) to read each file and get the file creation date and then get the files which are created on 2018-06-09.

for file in os.scandir(Path):
    if file.is_file():
        file_ctime = datetime.fromtimestamp(os.path.getctime(file)).strftime('%Y- %m- %d %H:%M:%S')
        if file_ctime[0:4] == '2018-06-09':
            # ...  

2 Answers 2

1

You could try using os.listdir(path) to get all the files and dirs from the given path.

Once you have all the files and directories you could use filter and a lambda function to create a new list of only the files with the desired timestamp.

You could then iterate through that list to do what work you need to on the correct files.

0

Let's start with the most basic thing - why are you building a datetime only to re-format it as string and then do a string comparison?

Then there is the whole point of using os.scandir() over os.listdir() - os.scandir() returns a os.DirEntry which caches file stats through the os.DirEntry.stat() call.

In dependence of checks you need to perform, os.listdir() might even perform better if you expect to do a lot of filtering on the filename as then you won't need to build up a whole os.DirEntry just to discard it.

So, to optimize your loop, if you don't expect a lot of filtering on the name:

for entry in os.scandir(Path):
    if entry.is_file() and 1528495200 <= entry.stat().st_ctime < 1528581600:
        pass  # do whatever you need with it

If you do, then better stick with os.listdir() as:

import stat

for entry in os.listdir(Path):
    # do your filtering on the entry name first...
    path = os.path.join(Path, entry)  # build path to the listed entry...
    stats = os.stat(path)  # cache the file entry statistics
    if stat.S_ISREG(stats.st_mode) and 1528495200 <= stats.st_ctime < 1528581600:
        pass  # do whatever you need with it

If you want to be flexible with the timestamps, use datetime.datetime.timestamp() beforehand to get the POSIX timestamps and then you can compare them against what stat_result.st_ctime returns directly without conversion.

However, even your original, non-optimized approach should be significantly faster than 2 hours for a mere 10k entries. I'd check the underlying filesystem, too, something seems wrong there.

4
  • i want to compare specific date with file timestamp, so that's why i reformatted the file timestamp into a variable and comparing with specific date. Any suggestion to compare the specific date with file timestamp date ? Jun 13, 2018 at 8:37
  • @GowthamChinatalacheruvu - The above is comparing against a specific date (2018-06-09), it's the equivalent of your code. As I wrote, if you want to be flexible with timestamps just build your own timestamps from dates - 1528495200 is what you'll get if you issue datetime.datetime(2018, 6, 9).timestamp() while 1528581600 is what you'll get if you issue datetime.datetime(2018, 6, 10).timestamp(). Store them in variables (min_ts and max_ts for example) and use them in the conditional (i.e. min_ts <= entry.stat().st_ctime < max_ts) and you get your flexibility.
    – zwer
    Jun 13, 2018 at 9:24
  • when i use stat().st_ctime we will get the timestamp(date+time). When i am comparing this value with my specific date, which will fail. Any solution to get only date from st_ctime? Jun 13, 2018 at 9:42
  • @GowthamChinatalacheruvu - Which is the reason why I am using a range above (from one timestamp to another) capturing the whole date span of 2018-06-09 - from 2018-06-09 00:00:00 to 2018-06-09 23:59:59. I don't see why you need to convert it into a date just to do the comparison? If you want to get the date you can use datetime.date.fromtimestamp() but that will needlessly increase execution time of your code which you are trying to avoid based on your question.
    – zwer
    Jun 13, 2018 at 13:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.