65

Is there any commonly used (or unjustly uncommonly used) utility "library" of bash functions? Something like Apache commons-lang for Java. Bash is so ubiquitous that it seems oddly neglected in the area of extension libraries.

If not, how would I make one?

1

6 Answers 6

43

Libraries for bash are out there, but not common. One of the reasons that bash libraries are scarce is due to the limitation of functions. I believe these limitations are best explained on "Greg's Bash Wiki":

Functions. Bash's "functions" have several issues:

  • Code reusability: Bash functions don't return anything; they only produce output streams. Every reasonable method of capturing that stream and either assigning it to a variable or passing it as an argument requires a SubShell, which breaks all assignments to outer scopes. (See also BashFAQ/084 for tricks to retrieve results from a function.) Thus, libraries of reusable functions are not feasible, as you can't ask a function to store its results in a variable whose name is passed as an argument (except by performing eval backflips).

  • Scope: Bash has a simple system of local scope which roughly resembles "dynamic scope" (e.g. Javascript, elisp). Functions see the locals of their callers (like Python's "nonlocal" keyword), but can't access a caller's positional parameters (except through BASH_ARGV if extdebug is enabled). Reusable functions can't be guaranteed free of namespace collisions unless you resort to weird naming rules to make conflicts sufficiently unlikely. This is particularly a problem if implementing functions that expect to be acting upon variable names from frame n-3 which may have been overwritten by your reusable function at n-2. Ksh93 can use the more common lexical scope rules by declaring functions with the "function name { ... }" syntax (Bash can't, but supports this syntax anyway).

  • Closures: In Bash, functions themselves are always global (have "file scope"), so no closures. Function definitions may be nested, but these are not closures, though they look very much the same. Functions are not "passable" (first-class), and there are no anonymous functions (lambdas). In fact, nothing is "passable", especially not arrays. Bash uses strictly call-by-value semantics (magic alias hack excepted).

  • There are many more complications involving: subshells; exported functions; "function collapsing" (functions that define or redefine other functions or themselves); traps (and their inheritance); and the way functions interact with stdio. Don't bite the newbie for not understanding all this. Shell functions are totally f***ed.

Source: http://mywiki.wooledge.org/BashWeaknesses

One example of a shell "library" is /etc/rc.d/functions on Redhat based system. This file contains functions commonly used in sysV init script.

13
  • 8
    while this is awesome information, it does not answer the question at all Jul 6, 2012 at 21:43
  • 4
    I wouldn't say "at all". The first sentence directly answers the question, although it is vague.
    – jordanm
    Jul 6, 2012 at 21:48
  • 6
    This answer is quite vexxing. Functions can be treated as command scripts defined in a centralized file which is very useful. All of the bullet points are quite easily addressed: namely $(namespaced_func). Probably a majority of languages do not encourage assignment of variables by value. There is also the paradigm of avoiding functional side effects. In addition, you can assign to global variables which accomplishes assignment by value for all intensive purposes. I think it would be productive to delete this answer and focus on evaluating function libraries
    – MetaChrome
    Nov 17, 2013 at 20:50
  • 12
    I think I can count the number of downvotes I've cast on one hand, but the generous acceptance of this answer compels me to make an exception here. As a retired Unix and Linux programmer and DBA of 30+ years, I couldn't disagree more with this answer. Bash, like most shells, is a higher level (more abstract), implementation of C in an interpreter. Bash, like C, uses a flat function scope. It's different from OOLs but not restrictive. Code reusability and returning values capabilities are excellent when variable scopes are kept local as much as possible. May 12, 2015 at 10:31
  • 6
    This answer is obviously an answer for why Bash isn't used for development projects (not its intention), and not why it doesn't have shared function libraries. I currently use function libraries as the Sr. Automation Engineer at my company. From everything to controlling forked processes, to file transfers to the cloud. There is so much above I disagree with from my personal and professional experience.
    – SaintHax
    Feb 7, 2017 at 15:42
29

I see some good info and bad info here. Let me share what I know since bash is the primary language I use at work (and we build libraries..). Google has a decent write up on bash scripts in general that I thought was a good read: https://google.github.io/styleguide/shell.xml.

Let me start by saying you should not think of a bash library as you do libraries in other languages. There are certain practices that must be enforced to keep a library in bash simple, organized, and most importantly, reusable.

There is no concept of returning anything from a bash function except for strings that it prints and the function's exit status (0-255). There are expected limitations here and a learning curve especially if you're accustomed to functions of higher-level languages. It can be weird at first, and if you find yourself in a situation where strings just aren't cutting it, you'll want to leverage an external tool such as jq. If jq (or something like it) is available, you can start having your functions print formatted output to be parsed & utilized as you would an object, array, etc.

Function Declarations

There are two ways to declare a function in bash. One operates within your current shell, we'll call is Fx0. And one spawns a subshell to operate in, we'll call that Fx1. Here are examples of how they're declared:

Fx0(){ echo "Hello from $FUNCNAME"; }
Fx1()( echo "Hello from $FUNCNAME" )

These 2 functions perform the same operation - indeed. However, there is a key difference here. Fx1 cannot perform any action that alters the current shell. That means modifying variables, changing shell options and declaring other functions. The latter is what can be exploited to prevent name spacing issues that can easily creep up on you.

# Fx1 cannot change the variable from a subshell
Fx0(){ Fx=0; }
Fx1()( Fx=1 )
Fx=foo; Fx0; echo $Fx
# 0
Fx=foo; Fx1; echo $Fx
# foo

That being said, The only time you should use an "Fx0" kind of function is when you're wanting to redeclare something in the current shell. Always use "Fx1" functions because they are safer and you you don't have to worry about the naming of any functions declared within it. As you can see below, the innocent function is overwritten inside of Fx1, however, it remains unscathed after the execution of Fx1.

innocent_function()(
    echo ":)"
)
Fx1()(
    innocent_function()( true )
    innocent_function
)
Fx1 #prints nothing, just returns true
innocent_function
# :)

This would have (likely) unintended consequences if you had used curly braces. Examples of useful "Fx0" type functions would be specifically for changing the current shell, like so:

use_strict(){
    set -eEu -o pipefail
}
enable_debug(){
    set -Tx
}
disable_debug(){
    set +Tx
}

Regarding Declarations

The use of global variables, or at least those expected to have a value, is bad practice all the way around. As you're building a library in bash, you don't ever want a function to rely on an external variable already being set. Anything the function needs should be supplied to it via the positional parameters. This is the main problem I see in libraries other folks try to build in bash. Even if I find something cool, I can't use it because I don't know the names of the variables I need to have set ahead of time. It leads to digging through all of the code and ultimately just picking out the useful pieces for myself. By far, the best functions to create for a library are extremely small and don't utilize named variables at all, even locally. Take the following for example:

serviceClient()(
    showUsage()(
        echo "This should be a help page"
    ) >&2
    isValidArg()(
        test "$(type -t "$1")" = "function"
    )
    isRunning()(
        nc -zw1 "$(getHostname)" "$(getPortNumber)"
    ) &>/dev/null
    getHostname()(
        echo localhost
    )
    getPortNumber()(
        echo 80
    )
    getStatus()(
        if isRunning
        then echo OK
        else echo DOWN
        fi
    )
    getErrorCount()(
        grep -c "ERROR" /var/log/apache2/error.log
    )
    printDetails()(
        echo "Service status: $(getStatus)"
        echo "Errors logged: $(getErrorCount)"
    )
    if isValidArg "$1"
    then "$1"
    else showUsage
    fi
)

Typically, what you would see near the top is local hostname=localhost and local port_number=80 which is fine, but it is not necessary. It is my opinion that these things should be functional-ized as you're building to prevent future pain when all of a sudden some logic needs to be introduced for getting a value, like: if isHttps; then echo 443; else echo 80; fi. You don't want that kind of logic placed in your main function or else you'll quickly make it ugly and unmanageable. Now, serviceClient has internal functions that get declared upon invocation which adds an unnoticeable amount of overhead to each run. The benefit is now you can have service2Client with functions (or external functions) that are named the same as what serviceClient has with absolutely no conflicts. Another important thing to keep in mind is that redirections can be applied to an entire function upon declaring it. see: isRunning or showUsage This gets as close to object-oriented-ness as I think you should bother using bash.

. serviceClient.sh
serviceClient
# This should be a help page
if serviceClient isRunning
then serviceClient printDetails
fi
# Service status: OK
# Errors logged: 0

I hope this helps my fellow bash hackers out there.

5
  • 1. The psuedo class is very interesting, that deserves an upvote. 2. The discussion of FX0 vs FX1 somehow didn't mention using FX0() { local Fx=0 } to avoid changing the global Fx. Of course the local Fx is visible downwards (except in ksh?) to function called from within FX0, but that's another issue. 3. "...and if you find yourself in a situation where strings just aren't cutting it, you'll want to leverage an external tool such as jq....". read var1 var2 var3 other < <(func_that_returns_3_values <args in>). I think your words are too vague clarification needed before 3rd party lib Mar 24, 2018 at 1:48
  • 1
    I see what you're saying. My point with mentioning jq (just as an example data parser) is that you will find yourself in situations where even the absolute greatest read bash foo will not suffice. In these cases, controlled structures like xml or json or csv or whatever need to be leveraged to maintain the integrity and robustness of your function(s). Also, there's nothing wrong with declaring all of your variables "local" -- my points are #1 you can't declare functions "local" in that way and #2 sub-shelling your function forces protection of the environment (important for collaboration). Mar 24, 2018 at 4:43
  • 2
    Thanks for the clarification. You convinced me. Mar 24, 2018 at 5:42
  • 3
    Why can't I upvote this comment more than once? Genious idea about using a subshell function like a namespace. Two other benefits of using subshell functions that I like. 1) You don't have to use local for variables, which is less typing. 2) By not using local, command substituation when assigning to a variable preserves the exit/return status. ei. $? will always be 0 when you do local x=$(foo), even if foo fails. Whereas $? doesn't suffer from this problem when doing a plain x=$(foo)
    – jmrah
    Jun 28, 2020 at 1:10
  • 1
    Be aware that if you have code that includes an exit ... and you want to put that in a function for reuse, you can not use the Fx1() ( ... ) way... Sep 23, 2020 at 7:02
14

Here's a list of "worthy of your time" bash libraries that I found after spending an hour or so googling.

bashmenot is a library that is used by Halcyon and Haskell on Heroku. The above link points to a complete list of available functions with examples -- impressive quality, quantity and documentation.

MBFL offers a set of modules implementing common operations and a script template. Pretty mature project and still active on github

You need to look at the code for a brief description and examples. It has a few years of development in its back.

This has the fewer most basic functions. For documentation you also have to look at the code.

0
12

Variables declared inside a function but without the local keyword are global.

It's good practice to declare variables only needed inside a function with local to avoid conflicts with other functions and globally (see foo() below).

Bash function libraries need to always be 'sourced'. I prefer using the 'source' synonym instead of the more common dot(.) so I can see it better during debugging.

The following technique works in at least bash 3.00.16 and 4.1.5...

#!/bin/bash
#
# TECHNIQUES
#

source ./TECHNIQUES.source

echo
echo "Send user prompts inside a function to stderr..."
foo() {
    echo "  Function foo()..."              >&2 # send user prompts to stderr
    echo "    Echoing 'this is my data'..." >&2 # send user prompts to stderr
    echo "this is my data"                      # this will not be displayed yet
}
#
fnRESULT=$(foo)                       # prints: Function foo()...
echo "  foo() returned '$fnRESULT'"   # prints: foo() returned 'this is my data'

echo
echo "Passing global and local variables..."
#
GLOBALVAR="Reusing result of foo() which is '$fnRESULT'"
echo "  Outside function: GLOBALVAR=$GLOBALVAR"
#
function fn()
{
  local LOCALVAR="declared inside fn() with 'local' keyword is only visible in fn()"
  GLOBALinFN="declared inside fn() without 'local' keyword is visible globally"
  echo
  echo "  Inside function fn()..."
  echo "    GLOBALVAR=$GLOBALVAR"
  echo "    LOCALVAR=$LOCALVAR"
  echo "    GLOBALinFN=$GLOBALinFN"
}

# call fn()...
fn

# call fnX()...
fnX

echo
echo "  Outside function..."
echo "    GLOBALVAR=$GLOBALVAR"
echo
echo "    LOCALVAR=$LOCALVAR"
echo "    GLOBALinFN=$GLOBALinFN"
echo
echo "    LOCALVARx=$LOCALVARx"
echo "    GLOBALinFNx=$GLOBALinFNx"
echo

The sourced function library is represented by...

#!/bin/bash
#
# TECHNIQUES.source
#

function fnX()
{
  local LOCALVARx="declared inside fnX() with 'local' keyword is only visible in fnX()"
  GLOBALinFNx="declared inside fnX() without 'local' keyword is visible globally"
  echo
  echo "  Inside function fnX()..."
  echo "    GLOBALVAR=$GLOBALVAR"
  echo "    LOCALVARx=$LOCALVARx"
  echo "    GLOBALinFNx=$GLOBALinFNx"
}

Running TECHNIQUES produces the following output...

Send user prompts inside a function to stderr...
  Function foo()...
    Echoing 'this is my data'...
  foo() returned 'this is my data'

Passing global and local variables...
  Outside function: GLOBALVAR=Reusing result of foo() which is 'this is my data'

  Inside function fn()...
    GLOBALVAR=Reusing result of foo() which is 'this is my data'
    LOCALVAR=declared inside fn() with 'local' keyword is only visible in fn()
    GLOBALinFN=declared inside fn() without 'local' keyword is visible globally

  Inside function fnX()...
    GLOBALVAR=Reusing result of foo() which is 'this is my data'
    LOCALVARx=declared inside fnX() with 'local' keyword is only visible in fnX()
    GLOBALinFNx=declared inside fnX() without 'local' keyword is visible globally

  Outside function...
    GLOBALVAR=Reusing result of foo() which is 'this is my data'

    LOCALVAR=
    GLOBALinFN=declared inside fn() without 'local' keyword is visible globally

    LOCALVARx=
    GLOBALinFNx=declared inside fnX() without 'local' keyword is visible globally
11
  • 4
    Use cat <<-EOF instead of echo, echo, echo, echo…
    – tijagi
    Mar 12, 2018 at 11:18
  • 1
    I use nothing but spaces since that is the only way to preserve indentation as written when using multiple editors. For more text than what will comfortably fit on 1 line of code, I put the multiple lines inside quotes so the text in the code looks like the output. Like the Here Document, this too usually requires breaking the logic indentation. But often, the indentation can be included in the output text as well to set it off. Mar 12, 2018 at 21:43
  • 2
    @tajagi - I personally use heredoc all the time, BUT ... any worthy editor would be able to echo-ify a section of text so if you are including editor capabilities then the difference between heredoc and echo boils down to mere syntax - a tie to be decided by personal preference. Doc gave a complete explanation for his preference. Mar 21, 2018 at 23:20
  • 1
    @tijagi - If you want to be great aim high, not below the belt. Mar 23, 2018 at 15:19
  • 3
    "...every worthy editor can be configured to follow your coding standards. Implying that you have any in the first place… Heredoc can be used with align to your code, if you open man bash once in a decade." ---- Your words. The part in bold are your gratuitous putdowns and have nothing to do with any arguments. I am sure if you look for them there are guidelines on Stack Exchange which essentially say avoid gratuitous putdowns. In this case you aren't even addressing the OP's issue, just the fact that DocSalvager used 4 lines of echo instead of heredoc. Chill and refactor. :) Mar 24, 2018 at 1:01
6

I found a good but old article here that gave a comprehensive list of utility libraries:

http://dberkholz.com/2011/04/07/bash-shell-scripting-libraries/

3

I can tell you that the lack of available function libraries has nothing to do with Bash's limitations, but rather how Bash is used. Bash is a quick and dirty language made for automation, not development, so the need for a library is rare. Then, you start to have a fine line between a function that needs to be shared, and converting the function into a full fledged script to be called. This is from a coding perspective, to be loaded by a shell is another matter, but normally runs on personal taste, not need. So... again a lack of shared libraries.

Here are a few functions I use regularly In my .bashrc

cd () {
   local pwd="${PWD}/"; # we need a slash at the end so we can check for it, too
   if [[ "$1" == "-e" ]]; then
      shift
      # start from the end
      [[ "$2" ]] && builtin cd "${pwd%/$1/*}/${2:-$1}/${pwd##*/$1/}" || builtin cd "$@"
   else
      # start from the beginning
      if [[ "$2" ]]; then
         builtin cd "${pwd/$1/$2}"
         pwd
      else
         builtin cd "$@"
      fi
   fi
}

And a version of a log()/err() exists in a function library at work for coders-- mainly so we all use the same style.

log() {
   echo -e "$(date +%m.%d_%H:%M) $@"| tee -a $OUTPUT_LOG
}

err() {
   echo -e "$(date +%m.%d_%H:%M) $@" |tee -a $OUTPUT_LOG
}

As you can see, the above utilities we use here, are not that exciting to share. I have another library to do tricks around bash limitations, which I think is the best use for them and I recommend creating your own.

3
  • " I have another library to do tricks around bash limitations, which I think is the best use for them and I recommend creating your own." IDK about OP, but this is precisely what I'm trying to avoid. Bash is full of all kinds of awful "gotchas". To work around them, I'd like to have some library functions that do the "right" thing (e.g. source_relative, to allow more modular scripts, by splitting code into new files, without relying on any preconditions on the working directory, or the absolute location of the script). Sure I can probably do it myself, but I have very little confidence ...
    – Alexander
    Jul 25, 2019 at 20:47
  • that I could do it correctly, especially compared to the experts out there who have surely seen and solved this problem already. If everyone is reinventing the wheel, we never end up building a car.
    – Alexander
    Jul 25, 2019 at 20:48
  • 1
    "Bash is full of all kinds of awful "gotchas"" -- Which is why I've developed an extensive library/script/program/framework (whatever you want to call it) of solutions that I can just use in any other script. Basicly, each solution is a function and any function can be called as an argument of the script call as in 'kit dialog --title "kit - dialog" --text "Hello World"' calling the dialog() function which uses the best available dialog method (usually YAD). Nov 30, 2019 at 8:37

Not the answer you're looking for? Browse other questions tagged or ask your own question.