127

I have such bash script:

array=( '2015-01-01', '2015-01-02' )

for i in "${array[@]}"
do
    python /home/user/executeJobs.py {i} &> /home/user/${i}.log
done

Now I want to loop through a range of dates, e.g. 2015-01-01 until 2015-01-31.

How to achieve in Bash?

Update:

Nice-to-have: No job should be started before a previous run has completed. In this case, when executeJobs.py is completed bash prompt $ will return.

e.g. could I incorporate wait%1 in my loop?

17
  • Are you on a platform with GNU date? Jan 29, 2015 at 23:01
  • 1
    check this link: glatter-gotz.com/blog/2011/02/19/…
    – qqibrow
    Jan 29, 2015 at 23:03
  • 1
    BTW, since you have a Python interpreter handy, this would be much, much easier to do in a reliable and portable way using the datetime Python module. Jan 29, 2015 at 23:03
  • 3
    2015-01-01 until 2015-01-31 does not span dates in more than one month, so it's a very simple case.
    – Wintermute
    Jan 29, 2015 at 23:14
  • 2
    ...so, if you're actually seeing a need to wait (as in, bugs happening due to concurrent processes when you don't), then you have something more interesting / more complicated going on, which needs a more complicated solution (like asking the subprocess to inherit a lockfile), which is enough complexity and sufficiently unrelated to date arithmetic that it should be a separate question. Jan 29, 2015 at 23:33

10 Answers 10

275

Using GNU date:

d=2015-01-01
while [ "$d" != 2015-02-20 ]; do 
  echo $d
  d=$(date -I -d "$d + 1 day")

  # mac option for d decl (the +1d is equivalent to + 1 day)
  # d=$(date -j -v +1d -f "%Y-%m-%d" $d +%Y-%m-%d)
done

Note that because this uses string comparison, it requires full ISO 8601 notation of the edge dates (do not remove leading zeros). To check for valid input data and coerce it to a valid form if possible, you can use date as well:

# slightly malformed input data
input_start=2015-1-1
input_end=2015-2-23

# After this, startdate and enddate will be valid ISO 8601 dates,
# or the script will have aborted when it encountered unparseable data
# such as input_end=abcd
startdate=$(date -I -d "$input_start") || exit -1
enddate=$(date -I -d "$input_end")     || exit -1

d="$startdate"
while [ "$d" != "$enddate" ]; do 
  echo $d
  d=$(date -I -d "$d + 1 day")
done

One final addition: To check that $startdate is before $enddate, if you only expect dates between the years 1000 and 9999, you can simply use string comparison like this:

while [[ "$d" < "$enddate" ]]; do

To be on the very safe side beyond the year 10000, when lexicographical comparison breaks down, use

while [ "$(date -d "$d" +%Y%m%d)" -lt "$(date -d "$enddate" +%Y%m%d)" ]; do

The expression $(date -d "$d" +%Y%m%d) converts $d to a numerical form, i.e., 2015-02-23 becomes 20150223, and the idea is that dates in this form can be compared numerically.

18
  • 1
    Sure, why not. It's just a shell loop, that it uses dates as iterator doesn't change what you can do inside it.
    – Wintermute
    Jan 29, 2015 at 23:29
  • 1
    @SirBenBenji, ...that said, %1 is a job control construct, and job control is turned off in noninteractive scripts unless you explicitly turn it on yourself. The proper way to refer to individual subprocesses inside a script is by PID, and even then, waiting for processes to complete is automatic unless they're explicitly backgrounded by your code (as with a &), or they self-detach (in which case wait won't even work, and the PID given to the shell will be invalidated by the double-fork process used to self-background). Jan 29, 2015 at 23:31
  • 1
    After taking a closer look, it appears that leap seconds are excluded from the UNIX timestamp, so that some timestamps refer to a space of two seconds. This apparently makes ´gettimeofday` very interesting to implement in the sub-second range, and I suppose we should count ourselves lucky that leap seconds have never been removed from a year. This means I have to correct myself: Adding 86400 seconds to a unix timestamp is arguably always the same as adding a day, since there is no way to refer to 2016-12-31T23:59:60 specifically. TIL.
    – Wintermute
    Dec 17, 2017 at 1:28
  • 2
    Running your first code (sh test.sh) gives me an error: date: illegal option -- I usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ... [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
    – dorien
    Jan 22, 2018 at 12:56
  • 4
    For macOS it won't work, first install gnu date apple.stackexchange.com/questions/231224/… May 24, 2019 at 9:28
33

Brace expansion:

for i in 2015-01-{01..31} …

More:

for i in 2015-02-{01..28} 2015-{04,06,09,11}-{01..30} 2015-{01,03,05,07,08,10,12}-{01..31} …

Proof:

$ echo 2015-02-{01..28} 2015-{04,06,09,11}-{01..30} 2015-{01,03,05,07,08,10,12}-{01..31} | wc -w
 365

Compact/nested:

$ echo 2015-{02-{01..28},{04,06,09,11}-{01..30},{01,03,05,07,08,10,12}-{01..31}} | wc -w
 365

Ordered, if it matters:

$ x=( $(printf '%s\n' 2015-{02-{01..28},{04,06,09,11}-{01..30},{01,03,05,07,08,10,12}-{01..31}} | sort -n -t"-" -k1 -k2 -k3) )
$ echo "${#x[@]}"
365

Since it's unordered, you can just tack leap years on:

$ echo {2015..2030}-{02-{01..28},{04,06,09,11}-{01..30},{01,03,05,07,08,10,12}-{01..31}} {2016..2028..4}-02-29 | wc -w
5844
8
  • 5
    What about leap years?
    – Wintermute
    Jan 29, 2015 at 23:16
  • Can I then use python /home/user/executeJobs.py 2015-01-{01..31} &> /home/user/2015-01-{01..31}.log ? Jan 29, 2015 at 23:16
  • 1
    @SirBenBenji That depends on executeJobs.py.
    – kojiro
    Jan 29, 2015 at 23:18
  • I must say executeJobs needs the date parameter and I need to wait on each run to complete. It is a Big Data job and should under no circumstances started before each previous run has completed. I should have thought of this before, sorry for forgetting about it. Jan 29, 2015 at 23:20
  • Oh Wow. Such power in simplicity.
    – crafter
    Feb 22, 2022 at 13:57
21
start='2019-01-01'
end='2019-02-01'

start=$(date -d $start +%Y%m%d)
end=$(date -d $end +%Y%m%d)

while [[ $start -le $end ]]
do
        echo $(date -d $start +%Y-%m-%d)
        start=$(date -d"$start + 1 day" +"%Y%m%d")

done
2
  • 2
    Careful, this one plays with the fact, that bash doesn't check variable types. It only works with the date format YMD - because that concatenates the date into something that looks like an integer, which make's two dates comparable. While 20200201 would be considered less than 20200202, it would not work with 2020-02-01 and 2020-02-02. OP asked for a dash-separated date.
    – n.r.
    Feb 26, 2021 at 14:29
  • @n.r. Good cautionary advise. Now it works also with dashed-separated date. Mar 31, 2021 at 6:10
6

The previous solution by @Gilli is pretty clever, because it plays with the fact, that you can simple format two dates make them look like integers. Then you can use -le / less-equal - which usually works with numeric data only.

Problem is, that this binds you to the date format YMD, like 20210201. If you need something different, like 2021-02-01 (that's what OP implicated as a requirement), the script will not work:

start='2021-02-01'
end='2021-02-05'

start=$(date -d $start +%Y-%m-%d)
end=$(date -d $end +%-Y%m-%d)

while [[ $start -le $end ]]
do
        echo $start
        start=$(date -d"$start + 1 day" +"%Y-%m-%d")

done

The output will look like this:

2021-02-01
2021-02-02
2021-02-03
2021-02-04
2021-02-05
2021-02-06
2021-02-07
./loop.sh: line 16: [[: 2021-02-08: value too great for base (error token is "08")

To fix that and use this loop for custom date formats, you need to work with one additional variable, let's call it "d_start":

d_start='2021-02-01'
end='2021-02-05'

start=$(date -d $d_start +%Y%m%d)
end=$(date -d $end +%Y%m%d)

while [[ $start -le $end ]]
do
        echo $d_start
        start=$(date -d"$start + 1 day" +"%Y%m%d")
        d_start=$(date -d"$d_start + 1 day" +"%Y-%m-%d")

done

This will lead to this output:

2021-02-01
2021-02-02
2021-02-03
2021-02-04
2021-02-05
5

I needed to loop through dates on AIX, BSDs, Linux, OS X and Solaris. The date command is one of the least portable and most miserable commands to use across platforms I have encountered. I found it easier to write a my_date command that just worked everywhere.

The C program below takes a starting date, and adds or subtracts days from it. If no date is supplied, it adds or subtracts days from the current date.

The my_date command allows you to perform the following everywhere:

start="2015-01-01"
stop="2015-01-31"

echo "Iterating dates from ${start} to ${stop}."

while [[ "${start}" != "${stop}" ]]
do
    python /home/user/executeJobs.py {i} &> "/home/user/${start}.log"
    start=$(my_date -s "${start}" -n +1)
done

And the C code:

#include <stdio.h>
#include <stdlib.h>
#include <stdbool.h>
#include <time.h>

int show_help();

int main(int argc, char* argv[])
{
    int eol = 0, help = 0, n_days = 0;
    int ret = EXIT_FAILURE;

    time_t startDate = time(NULL);
    const time_t ONE_DAY = 24 * 60 * 60;

    for (int i=0; i<argc; i++)
    {
        if (strcmp(argv[i], "-l") == 0)
        {
            eol = 1;
        }
        else if (strcmp(argv[i], "-n") == 0)
        {
            if (++i == argc)
            {
                show_help();
                ret = EXIT_FAILURE;
                goto finish;
            }

            n_days = strtoll(argv[i], NULL, 0);
        }
        else if (strcmp(argv[i], "-s") == 0)
        {
            if (++i == argc)
            {
                show_help();
                ret = EXIT_FAILURE;
                goto finish;
            }

            struct tm dateTime;
            memset (&dateTime, 0x00, sizeof(dateTime));

            const char* start = argv[i];
            const char* end = strptime (start, "%Y-%m-%d", &dateTime);

            /* Ensure all characters are consumed */
            if (end - start != 10)
            {
                show_help();
                ret = EXIT_FAILURE;
                goto finish;
            }

            startDate = mktime (&dateTime);
        }
    }

    if (help == 1)
    {
        show_help();
        ret = EXIT_SUCCESS;
        goto finish;
    }

    char buff[32];
    const time_t next = startDate + ONE_DAY * n_days;
    strftime(buff, sizeof(buff), "%Y-%m-%d", localtime(&next));

    /* Paydirt */
    if (eol)
        fprintf(stdout, "%s\n", buff);
    else
        fprintf(stdout, "%s", buff);

    ret = EXIT_SUCCESS;

finish:

    return ret;
}

int show_help()
{
    fprintf(stderr, "Usage:\n");
    fprintf(stderr, "  my_date [-s date] [-n [+|-]days] [-l]\n");
    fprintf(stderr, "    -s date: optional, starting date in YYYY-MM-DD format\n");
    fprintf(stderr, "    -n days: optional, number of days to add or subtract\n");
    fprintf(stderr, "    -l: optional, add new-line to output\n");
    fprintf(stderr, "\n");
    fprintf(stderr, "  If no options are supplied, then today is printed.\n");
    fprintf(stderr, "\n");
    return 0;
}
5

If you're stuck with busybox date that is used in many distributions like alpine that are commonly used in docker containers, I've found working with timestamps to be the most reliable approach:

STARTDATE="2019-12-30"
ENDDATE="2020-01-04"

start=$(date -d $STARTDATE +%s)
end=$(date -d $ENDDATE +%s)

d="$start"
while [[ $d -le $end ]]
do
    date -d @$d +%Y-%m-%d

    d=$(( $d + 86400 ))
done

This will output:

2019-12-30
2019-12-31
2020-01-01
2020-01-02
2020-01-03
2020-01-04

Unix timestamps don't include leap seconds, so 1 day equals always exactly 86400 seconds.

2
  • This is the only one works inside some container ...
    – Eric
    Sep 15, 2021 at 11:22
  • 1
    @user218867 right this is because most lightweight container OSes like Alpine only have the date packed into busybox but no GNU date to save space.
    – Gellweiler
    Sep 16, 2021 at 19:45
4

I had the same issue and I tried some of the above answers, maybe they are ok, but none of those answers fixed on what I was trying to do, using macOS.

I was trying to iterate over dates in the past, and the following is what worked for me:

#!/bin/bash

# Get the machine date
newDate=$(date '+%m-%d-%y')

# Set a counter variable
counter=1 

# Increase the counter to get back in time
while [ "$newDate" != 06-01-18 ]; do
  echo $newDate
  newDate=$(date -v -${counter}d '+%m-%d-%y')
  counter=$((counter + 1))
done

Hope it helps.

2
  • I recommend not using a variable name that happens to coincide with a very powerful bit-for-bit copy command, but that's just me. Aug 3, 2018 at 0:20
  • Simpler way is to use gdate instead of date in macOS.
    – northtree
    Apr 23, 2020 at 2:11
4

Bash is best written by leveraging pipes(|). This should result in memory efficient and concurrent(faster) processing. I would write the following:

seq 0 100 | xargs printf "20 Aug 2020 - %sdays\n" \
  | xargs -d '\n' -l date -d

The following will print the date of 20 aug 2020 and print the dates of the 100 days before it.

This oneliner can be made into a utility.

#!/usr/bin/env bash

# date-range template <template>

template="${1:--%sdays}"

export LANG;

xargs printf "$template\n" | xargs -d '\n' -l date -d

By default we choose to iterate into the past 1 day at a time.

$ seq 10 | date-range
Mon Mar  2 17:42:43 CET 2020
Sun Mar  1 17:42:43 CET 2020
Sat Feb 29 17:42:43 CET 2020
Fri Feb 28 17:42:43 CET 2020
Thu Feb 27 17:42:43 CET 2020
Wed Feb 26 17:42:43 CET 2020
Tue Feb 25 17:42:43 CET 2020
Mon Feb 24 17:42:43 CET 2020
Sun Feb 23 17:42:43 CET 2020
Sat Feb 22 17:42:43 CET 2020

Let's say we want to generate dates up to a certain date. We don't know yet how many iterations we need to get there. Let's say Tom was born 1 Jan 2001. We want to generate each date till a certain one. We can achieve this by using sed.

seq 0 $((2**63-1)) | date-range | sed '/.. Jan 2001 /q'

The $((2**63-1)) trick is used to create a big integer.

Once sed exits it will also exit the date-range utility.

One can also iterate using a 3 month interval:

$ seq 0 3 12 | date-range '+%smonths'
Tue Mar  3 18:17:17 CET 2020
Wed Jun  3 19:17:17 CEST 2020
Thu Sep  3 19:17:17 CEST 2020
Thu Dec  3 18:17:17 CET 2020
Wed Mar  3 18:17:17 CET 2021
1
2

If one wants to loop from input date to any range below can be used, also it will print output in format of yyyyMMdd...

#!/bin/bash
in=2018-01-15
while [ "$in" != 2018-01-25 ]; do
  in=$(date -I -d "$in + 1 day")
  x=$(date -d "$in" +%Y%m%d)
  echo $x
done
0

This might also help. Based on Gilli answer, but a different solution of the issue with an integer conversion.

Basically, while verifying the input, LoopEachDay stores the "end" date in seconds and compares with it firstly converting the current day into seconds(date -d "$dateIteration" '+%s'), too.

#/bin/bash

RegexVerify()
{
    regex="$1";
    shift;

    if [[ "$@" =~ $regex ]];
    then
        return 0;
    fi

    return 1;
}

VerifyDateISO8601()
{
    if RegexVerify '^[0-9]{4}-(0?[1-9]|10|11|12)-(0?[1-9]|[12][0-9]|3[01])$' "$1";
    then
        return 0;
    fi

    return 1;
}

# Iterate each day
#
# * The *first* argument is an ISO8601 start date.
# * The *second* argument is an ISO8601 end date or an empty string which assumes
# the current date.
LoopEachDay()
{
    if ! VerifyDateISO8601 "$1";
    then
        return 1;
    fi

    if ! VerifyDateISO8601 "$2" && [ "$2" != '' ];
    then
        return 2;
    fi

    dateIteration="$(date -d "$1" '+%Y-%m-%d')";
    dateIterationEndSeconds="$(date -d "$2" '+%s')";

    while (("$(date -d "$dateIteration" '+%s')" <= dateIterationEndSeconds))
    do
        printf $'%s\n' "$dateIteration"; # A work with "$dateIteration"

        dateIteration="$(date -d "$dateIteration + 1 day" '+%Y-%m-%d')";
    done
}

LoopEachDay '2021-13-01' '';
printf $'Exit code: %s\n\n' "$?";

# Exit code: 1

LoopEachDay '2021-04-01' '';

# 2021-04-01
# 2021-04-02
# 2021-04-03
# 2021-04-04
# 2021-04-05
# 2021-04-06
# 2021-04-07
# 2021-04-08

printf $'\n';
LoopEachDay '2021-04-03' '2021-04-06';

# 2021-04-03
# 2021-04-04
# 2021-04-05
# 2021-04-06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.