154

We require a script that simulates associative arrays or map-like data structure for shell scripting. Can anyone let's know how it is done?

1

17 Answers 17

201

Another option, if portability is not your main concern, is to use associative arrays that are built in to the shell. This should work in bash 4.0 (available now on most major distros, though not on OS X unless you install it yourself), ksh, and zsh:

declare -A newmap
newmap[name]="Irfan Zulfiqar"
newmap[designation]=SSE
newmap[company]="My Own Company"

echo ${newmap[company]}
echo ${newmap[name]}

Depending on the shell, you may need to do a typeset -A newmap instead of declare -A newmap, or in some it may not be necessary at all.

6
  • Thanks for you posting answer, I think that would the best way to do it for guys who would be using bash 4.0 or above. Apr 2, 2009 at 6:07
  • I'd add a little kludge to make sure BASH_VERSION is set, and >= 4. And yes, BASH 4 is really, really cool!
    – Tim Post
    Apr 14, 2009 at 8:46
  • I'm using something like this. What's the best way to "catch" the error where the array index/subscript doesn't exist? For example, what if I was taking the subscript as a command line option, and the user made a typo and entered "designatio"? I get a "bad array subscript" error but don't how to validate the input at the time of array lookup, if that's possible?
    – Jer
    Feb 5, 2014 at 19:02
  • 3
    @Jer It's pretty obscure, but to determine if a variable is set in the shell, you can use test -z ${variable+x} (the x doesn't matter, that could be any string). For an associative array in Bash, you can do similar; use test -z ${map[key]+x}. Feb 6, 2014 at 5:00
  • I had to use declare -a newmap in iTerm2 3.4
    – harshainfo
    May 11, 2022 at 4:20
120

Another non-bash 4 way.

#!/bin/bash

# A pretend Python dictionary with bash 3 
ARRAY=( "cow:moo"
        "dinosaur:roar"
        "bird:chirp"
        "bash:rock" )

for animal in "${ARRAY[@]}" ; do
    KEY=${animal%%:*}
    VALUE=${animal#*:}
    printf "%s likes to %s.\n" "$KEY" "$VALUE"
done

echo -e "${ARRAY[1]%%:*} is an extinct animal which likes to ${ARRAY[1]#*:}\n"

You could throw an if statement for searching in there as well. if [[ $var =~ /blah/ ]]. or whatever.

5
  • 3
    This method is good when you don't have Bash 4 indeed. But I think the line that fetches the VALUE would be safer this way: VALUE=${animal#*:}. With only one # character, the matching will stop on the first ":". That allows values to contain ":", too. Nov 14, 2012 at 14:31
  • @Ced-le-pingouin ~ That's a great point! I didn't catch that. I've edited my post to reflect your suggested improvements.
    – Bubnoff
    Nov 15, 2012 at 17:22
  • 1
    It a pretty hackish emulation of associative arrays using BASH parameter substitution. The "key" param-sub substitutes everything before the colon and the value pattern substitutes everything after the colon. Similar to a regex wildcard match. So NOT a true associative array. Not recommended unless you need an easy to understand way to do hash/associative array-like functionality in BASH 3 or below. It works though! More here: tldp.org/LDP/abs/html/parameter-substitution.html#PSOREX2
    – Bubnoff
    Jan 27, 2014 at 18:51
  • 1
    This does not implement an associative array because it does not provide a way to look up an item by the key. It only provides a way to find each key (and value) from a numeric index. (An item could be found by key by iterating through the array, but that is not what is desired for an associative array.) Dec 22, 2018 at 12:03
  • @EricPostpischil True. It is only a hack. It allows a person to use familiar syntax in the setup but still requires iterating through the array as you say. I've tried to be clear in my previous comment that it is definitely not an associative array and I don't even recommend it if you have alternatives. The only point in its favor, in my view, is that it is easy to write and use for those familiar with other languages like Python. If you are at a point where you actually want to implement associative arrays in BASH 3 then you may need to retrace your steps a bit.
    – Bubnoff
    Dec 26, 2018 at 19:48
38

I think that you need to step back and think about what a map, or associative array, really is. All it is is a way to store a value for a given key, and get that value back quickly and efficiently. You may also want to be able to iterate over the keys to retrieve every key value pair, or delete keys and their associated values.

Now, think about a data structure you use all the time in shell scripting, and even just in the shell without writing a script, that has these properties. Stumped? It's the filesystem.

Really, all you need to have an associative array in shell programming is a temp directory. mktemp -d is your associative array constructor:

prefix=$(basename -- "$0")
map=$(mktemp -dt ${prefix})
echo >${map}/key somevalue
value=$(cat ${map}/key)

If you don't feel like using echo and cat, you can always write some little wrappers; these ones are modelled off of Irfan's, though they just output the value rather than setting arbitrary variables like $value:

#!/bin/sh

prefix=$(basename -- "$0")
mapdir=$(mktemp -dt ${prefix})
trap 'rm -r ${mapdir}' EXIT

put() {
  [ "$#" != 3 ] && exit 1
  mapname=$1; key=$2; value=$3
  [ -d "${mapdir}/${mapname}" ] || mkdir "${mapdir}/${mapname}"
  echo $value >"${mapdir}/${mapname}/${key}"
}

get() {
  [ "$#" != 2 ] && exit 1
  mapname=$1; key=$2
  cat "${mapdir}/${mapname}/${key}"
}

put "newMap" "name" "Irfan Zulfiqar"
put "newMap" "designation" "SSE"
put "newMap" "company" "My Own Company"

value=$(get "newMap" "company")
echo $value

value=$(get "newMap" "name")
echo $value

edit: This approach is actually quite a bit faster than the linear search using sed suggested by the questioner, as well as more robust (it allows keys and values to contain -, =, space, qnd ":SP:"). The fact that it uses the filesystem does not make it slow; these files are actually never guaranteed to be written to the disk unless you call sync; for temporary files like this with a short lifetime, it's not unlikely that many of them will never be written to disk.

I did a few benchmarks of Irfan's code, Jerry's modification of Irfan's code, and my code, using the following driver program:

#!/bin/sh

mapimpl=$1
numkeys=$2
numvals=$3

. ./${mapimpl}.sh    #/ <- fix broken stack overflow syntax highlighting

for (( i = 0 ; $i < $numkeys ; i += 1 ))
do
    for (( j = 0 ; $j < $numvals ; j += 1 ))
    do
        put "newMap" "key$i" "value$j"
        get "newMap" "key$i"
    done
done

The results:

    $ time ./driver.sh irfan 10 5

    real    0m0.975s
    user    0m0.280s
    sys     0m0.691s

    $ time ./driver.sh brian 10 5

    real    0m0.226s
    user    0m0.057s
    sys     0m0.123s

    $ time ./driver.sh jerry 10 5

    real    0m0.706s
    user    0m0.228s
    sys     0m0.530s

    $ time ./driver.sh irfan 100 5

    real    0m10.633s
    user    0m4.366s
    sys     0m7.127s

    $ time ./driver.sh brian 100 5

    real    0m1.682s
    user    0m0.546s
    sys     0m1.082s

    $ time ./driver.sh jerry 100 5

    real    0m9.315s
    user    0m4.565s
    sys     0m5.446s

    $ time ./driver.sh irfan 10 500

    real    1m46.197s
    user    0m44.869s
    sys     1m12.282s

    $ time ./driver.sh brian 10 500

    real    0m16.003s
    user    0m5.135s
    sys     0m10.396s

    $ time ./driver.sh jerry 10 500

    real    1m24.414s
    user    0m39.696s
    sys     0m54.834s

    $ time ./driver.sh irfan 1000 5

    real    4m25.145s
    user    3m17.286s
    sys     1m21.490s

    $ time ./driver.sh brian 1000 5

    real    0m19.442s
    user    0m5.287s
    sys     0m10.751s

    $ time ./driver.sh jerry 1000 5

    real    5m29.136s
    user    4m48.926s
    sys     0m59.336s

6
  • 3
    I dont think you should be using file system for maps, that basically using IO for something that you can done fairly fast in memory. Apr 2, 2009 at 6:05
  • 10
    The files won't necessarily ever be written to the disk; unless you call sync, the operating system may just leave them in memory. Your code is calling out to sed and doing several linear searches, which are all very slow. I did some quick benchmarks, and my version is 5-35 times faster. Apr 2, 2009 at 6:47
  • on the other hand, bash4's native arrays are significantly better an approach and in bash3 you can still keep everything off the disk without forking by use of declare and indirection.
    – lhunath
    Aug 12, 2010 at 13:23
  • 7
    "fast" and "shell" don't really go together anyway: certainly not for the sort of speed issues we're talking about at the "avoid miniscule IO" level. You could search for and use /dev/shm to guarantee no IO.
    – jmtd
    Apr 1, 2011 at 16:30
  • I get "basename: illegal option -- b" for the first line with the prefix. how can I fix that?
    – Bren
    May 2, 2017 at 3:25
20

To add to Irfan's answer, here is a shorter and faster version of get() since it requires no iteration over the map contents:

get() {
    mapName=$1; key=$2

    map=${!mapName}
    value="$(echo $map |sed -e "s/.*--${key}=\([^ ]*\).*/\1/" -e 's/:SP:/ /g' )"
}
1
  • 18
    forking a subshell and sed is hardly optimal. Bash4 supports this natively and bash3 has better alternatives.
    – lhunath
    Aug 12, 2010 at 13:21
8

Yet another non-bash-4 (i.e., bash 3, Mac-compatible) way:

val_of_key() {
    case $1 in
        'A1') echo 'aaa';;
        'B2') echo 'bbb';;
        'C3') echo 'ccc';;
        *) echo 'zzz';;
    esac
}

for x in 'A1' 'B2' 'C3' 'D4'; do
    y=$(val_of_key "$x")
    echo "$x => $y"
done

Prints:

A1 => aaa
B2 => bbb
C3 => ccc
D4 => zzz

The function with the case acts like an associative array. Unfortunately it cannot use return, so it has to echo its output, but this is not a problem, unless you are a purist that shuns forking subshells.

7

Bash4 supports this natively. Do not use grep or eval, they are the ugliest of hacks.

For a verbose, detailed answer with example code see: https://stackoverflow.com/questions/3467959

7
####################################################################
# Bash v3 does not support associative arrays
# and we cannot use ksh since all generic scripts are on bash
# Usage: map_put map_name key value
#
function map_put
{
    alias "${1}$2"="$3"
}

# map_get map_name key
# @return value
#
function map_get
{
    alias "${1}$2" | awk -F"'" '{ print $2; }'
}

# map_keys map_name 
# @return map keys
#
function map_keys
{
    alias -p | grep $1 | cut -d'=' -f1 | awk -F"$1" '{print $2; }'
}

Example:

mapName=$(basename $0)_map_
map_put $mapName "name" "Irfan Zulfiqar"
map_put $mapName "designation" "SSE"

for key in $(map_keys $mapName)
do
    echo "$key = $(map_get $mapName $key)
done
5

For Bash 3, there is a particular case that has a nice and simple solution:

If you don't want to handle a lot of variables, or keys are simply invalid variable identifiers, and your array is guaranteed to have less than 256 items, you can abuse function return values. This solution does not require any subshell as the value is readily available as a variable, nor any iteration so that performance screams. Also it's very readable, almost like the Bash 4 version.

Here's the most basic version:

hash_index() {
    case $1 in
        'foo') return 0;;
        'bar') return 1;;
        'baz') return 2;;
    esac
}

hash_vals=("foo_val"
           "bar_val"
           "baz_val");

hash_index "foo"
echo ${hash_vals[$?]}

Remember, use single quotes in case, else it's subject to globbing. Really useful for static/frozen hashes from the start, but one could write an index generator from a hash_keys=() array.

Watch out, it defaults to the first one, so you may want to set aside zeroth element:

hash_index() {
    case $1 in
        'foo') return 1;;
        'bar') return 2;;
        'baz') return 3;;
    esac
}

hash_vals=("",           # sort of like returning null/nil for a non existent key
           "foo_val"
           "bar_val"
           "baz_val");

hash_index "foo" || echo ${hash_vals[$?]}  # It can't get more readable than this

Caveat: the length is now incorrect.

Alternatively, if you want to keep zero-based indexing, you can reserve another index value and guard against a non-existent key, but it's less readable:

hash_index() {
    case $1 in
        'foo') return 0;;
        'bar') return 1;;
        'baz') return 2;;
        *)   return 255;;
    esac
}

hash_vals=("foo_val"
           "bar_val"
           "baz_val");

hash_index "foo"
[[ $? -ne 255 ]] && echo ${hash_vals[$?]}

Or, to keep the length correct, offset index by one:

hash_index() {
    case $1 in
        'foo') return 1;;
        'bar') return 2;;
        'baz') return 3;;
    esac
}

hash_vals=("foo_val"
           "bar_val"
           "baz_val");

hash_index "foo" || echo ${hash_vals[$(($? - 1))]}
1
  • BUT if I want to assign to hash_index "foo" || echo ${hash_vals[$(($? - 1))]} to a variable, this code output=foo" || echo ${hash_vals[$(($? - 1))]}. So if not mind, could you tell me the right way to assign it.
    – Bowen Peng
    Aug 3, 2021 at 15:38
4

Now answering this question.

Following scripts simulates associative arrays in shell scripts. Its simple and very easy to understand.

Map is nothing but a never ending string that has keyValuePair saved as --name=Irfan --designation=SSE --company=My:SP:Own:SP:Company

spaces are replaced with ':SP:' for values

put() {
    if [ "$#" != 3 ]; then exit 1; fi
    mapName=$1; key=$2; value=`echo $3 | sed -e "s/ /:SP:/g"`
    eval map="\"\$$mapName\""
    map="`echo "$map" | sed -e "s/--$key=[^ ]*//g"` --$key=$value"
    eval $mapName="\"$map\""
}

get() {
    mapName=$1; key=$2; valueFound="false"

    eval map=\$$mapName

    for keyValuePair in ${map};
    do
        case "$keyValuePair" in
            --$key=*) value=`echo "$keyValuePair" | sed -e 's/^[^=]*=//'`
                      valueFound="true"
        esac
        if [ "$valueFound" == "true" ]; then break; fi
    done
    value=`echo $value | sed -e "s/:SP:/ /g"`
}

put "newMap" "name" "Irfan Zulfiqar"
put "newMap" "designation" "SSE"
put "newMap" "company" "My Own Company"

get "newMap" "company"
echo $value

get "newMap" "name"
echo $value

edit: Just added another method to fetch all keys.

getKeySet() {
    if [ "$#" != 1 ]; 
    then 
        exit 1; 
    fi

    mapName=$1; 

    eval map="\"\$$mapName\""

    keySet=`
           echo $map | 
           sed -e "s/=[^ ]*//g" -e "s/\([ ]*\)--/\1/g"
          `
}
1
  • 1
    You're eval'ing data as though it's bash code, and what's more: you fail to quote it properly. Both cause masses of bugs and arbitrary code injection.
    – lhunath
    Aug 12, 2010 at 13:25
2

You can use dynamic variable names and let the variables names work like the keys of a hashmap.

For example, if you have an input file with two columns, name, credit, as the example bellow, and you want to sum the income of each user:

Mary 100
John 200
Mary 50
John 300
Paul 100
Paul 400
David 100

The command bellow will sum everything, using dynamic variables as keys, in the form of map_${person}:

while read -r person money; ((map_$person+=$money)); done < <(cat INCOME_REPORT.log)

To read the results:

set | grep map

The output will be:

map_David=100
map_John=500
map_Mary=150
map_Paul=500

Elaborating on these techniques, I'm developing on GitHub a function that works just like a HashMap Object, shell_map.

In order to create "HashMap instances" the shell_map function is able create copies of itself under different names. Each new function copy will have a different $FUNCNAME variable. $FUNCNAME then is used to create a namespace for each Map instance.

The map keys are global variables, in the form $FUNCNAME_DATA_$KEY, where $KEY is the key added to the Map. These variables are dynamic variables.

Bellow I'll put a simplified version of it so you can use as example.

#!/bin/bash

shell_map () {
    local METHOD="$1"

    case $METHOD in
    new)
        local NEW_MAP="$2"

        # loads shell_map function declaration
        test -n "$(declare -f shell_map)" || return

        # declares in the Global Scope a copy of shell_map, under a new name.
        eval "${_/shell_map/$2}"
    ;;
    put)
        local KEY="$2"  
        local VALUE="$3"

        # declares a variable in the global scope
        eval ${FUNCNAME}_DATA_${KEY}='$VALUE'
    ;;
    get)
        local KEY="$2"
        local VALUE="${FUNCNAME}_DATA_${KEY}"
        echo "${!VALUE}"
    ;;
    keys)
        declare | grep -Po "(?<=${FUNCNAME}_DATA_)\w+((?=\=))"
    ;;
    name)
        echo $FUNCNAME
    ;;
    contains_key)
        local KEY="$2"
        compgen -v ${FUNCNAME}_DATA_${KEY} > /dev/null && return 0 || return 1
    ;;
    clear_all)
        while read var; do  
            unset $var
        done < <(compgen -v ${FUNCNAME}_DATA_)
    ;;
    remove)
        local KEY="$2"
        unset ${FUNCNAME}_DATA_${KEY}
    ;;
    size)
        compgen -v ${FUNCNAME}_DATA_${KEY} | wc -l
    ;;
    *)
        echo "unsupported operation '$1'."
        return 1
    ;;
    esac
}

Usage:

shell_map new credit
credit put Mary 100
credit put John 200
for customer in `credit keys`; do 
    value=`credit get $customer`       
    echo "customer $customer has $value"
done
credit contains_key "Mary" && echo "Mary has credit!"
1

I've found it true, as already mentioned, that the best performing method is to write out key/vals to a file, and then use grep/awk to retrieve them. It sounds like all sorts of unnecessary IO, but disk cache kicks in and makes it extremely efficient -- much faster than trying to store them in memory using one of the above methods (as the benchmarks show).

Here's a quick, clean method I like:

hinit() {
    rm -f /tmp/hashmap.$1
}

hput() {
    echo "$2 $3" >> /tmp/hashmap.$1
}

hget() {
    grep "^$2 " /tmp/hashmap.$1 | awk '{ print $2 };'
}

hinit capitols
hput capitols France Paris
hput capitols Netherlands Amsterdam
hput capitols Spain Madrid

echo `hget capitols France` and `hget capitols Netherlands` and `hget capitols Spain`

If you wanted to enforce single-value per key, you could also do a little grep/sed action in hput().

1

Adding another option, if jq is available:

export NAMES="{
  \"Mary\":\"100\",
  \"John\":\"200\",
  \"Mary\":\"50\",
  \"John\":\"300\",
  \"Paul\":\"100\",
  \"Paul\":\"400\",
  \"David\":\"100\"
}"
export NAME=David
echo $NAMES | jq --arg v "$NAME" '.[$v]' | tr -d '"' 
0

What a pity I did not see the question before - I've wrote library shell-framework which contains among others the maps(Associative arrays). The last version of it can be found here.

Example:

#!/bin/bash 
#include map library
shF_PATH_TO_LIB="/usr/lib/shell-framework"
source "${shF_PATH_TO_LIB}/map"

#simple example get/put
putMapValue "mapName" "mapKey1" "map Value 2"
echo "mapName[mapKey1]: $(getMapValue "mapName" "mapKey1")"

#redefine old value to new
putMapValue "mapName" "mapKey1" "map Value 1"
echo "after change mapName[mapKey1]: $(getMapValue "mapName" "mapKey1")"

#add two new pairs key/values and print all keys
putMapValue "mapName" "mapKey2" "map Value 2"
putMapValue "mapName" "mapKey3" "map Value 3"
echo -e "mapName keys are \n$(getMapKeys "mapName")"

#create new map
putMapValue "subMapName" "subMapKey1" "sub map Value 1"
putMapValue "subMapName" "subMapKey2" "sub map Value 2"

#and put it in mapName under key "mapKey4"
putMapValue "mapName" "mapKey4" "subMapName"

#check if under two key were placed maps
echo "is map mapName[mapKey3]? - $(if isMap "$(getMapValue "mapName" "mapKey3")" ; then echo Yes; else echo No; fi)"
echo "is map mapName[mapKey4]? - $(if isMap "$(getMapValue "mapName" "mapKey4")" ; then echo Yes; else echo No; fi)"

#print map with sub maps
printf "%s\n" "$(mapToString "mapName")"
1
0

several years ago I wrote script library for bash which supported associative arrays among other features (logging, configuration files, extended support for command line argument, generate help, unit testing, etc). The library contains a wrapper for associative arrays and automatically switches to appropriate model (internal for bash4 and emulate for previous versions). It was called shell-framework and hosted at origo.ethz.ch but today the resource is closed. If someone still needs it I can share it with you.

1
  • 1
    Might be worth sticking it up on github Aug 21, 2013 at 11:02
0

Shell have no built-in map like data structure, I use raw string to describe items like that:

ARRAY=(
    "item_A|attr1|attr2|attr3"
    "item_B|attr1|attr2|attr3"
    "..."
)

when extract items and its attributes:

for item in "${ARRAY[@]}"
do
    item_name=$(echo "${item}"|awk -F "|" '{print $1}')
    item_attr1=$(echo "${item}"|awk -F "|" '{print $2}')
    item_attr2=$(echo "${item}"|awk -F "|" '{print $3}')

    echo "${item_name}"
    echo "${item_attr1}"
    echo "${item_attr2}"
done

This seems like not clever than other people's answer, but easy to understand for new people to shell.

-1

I modified Vadim's solution with the following:

####################################################################
# Bash v3 does not support associative arrays
# and we cannot use ksh since all generic scripts are on bash
# Usage: map_put map_name key value
#
function map_put
{
    alias "${1}$2"="$3"
}

# map_get map_name key
# @return value
#
function map_get {
    if type -p "${1}$2"
        then
            alias "${1}$2" | awk -F "'" '{ print $2; }';
    fi
}

# map_keys map_name 
# @return map keys
#
function map_keys
{
    alias -p | grep $1 | cut -d'=' -f1 | awk -F"$1" '{print $2; }'
}

The change is to map_get in order to prevent it from returning errors if you request a key that doesn't exist, though the side-effect is that it will also silently ignore missing maps, but it suited my use-case better since I just wanted to check for a key in order to skip items in a loop.

-1

Late reply, but consider addressing the problem in this way, using the bash builtin read as illustrated within the code snippet from a ufw firewall script that follows. This approach has the advantage of using as many delimited field sets (not just 2) as are desired. We have used the | delimiter because port range specifiers may require a colon, ie 6001:6010.

#!/usr/bin/env bash

readonly connections=(       
                            '192.168.1.4/24|tcp|22'
                            '192.168.1.4/24|tcp|53'
                            '192.168.1.4/24|tcp|80'
                            '192.168.1.4/24|tcp|139'
                            '192.168.1.4/24|tcp|443'
                            '192.168.1.4/24|tcp|445'
                            '192.168.1.4/24|tcp|631'
                            '192.168.1.4/24|tcp|5901'
                            '192.168.1.4/24|tcp|6566'
)

function set_connections(){
    local range proto port
    for fields in ${connections[@]}
    do
            IFS=$'|' read -r range proto port <<< "$fields"
            ufw allow from "$range" proto "$proto" to any port "$port"
    done
}

set_connections

Not the answer you're looking for? Browse other questions tagged or ask your own question.