61

Given the following shell script, would someone be so kind as to explain the grep -Po regex please?

#!/bin/bash
# Issue the request for a bearer token, json is returned
raw_json=`curl -s -X POST -d "username=name&password=secret&client_id=security-admin-console" http://localhost:8081/auth/realms/master/tokens/grants/access`
# Strip away all but the "access_token" field's value using a Python regular expression
bearerToken=`echo $raw_json | grep -Po '"'"access_token"'"\s*:\s*"\K([^"]*)'`
echo "The bearer token is:"
echo $bearerToken

So specifically, I'm interested in understanding the parts of the regex

grep -Po '"'"access_token"'"\s*:\s*"\K([^"]*)'`

and how it works. Why so many quotes? What is the "K" for? I've some experience with grep regex but this confuses me.

This is the actual output of the curl command and the shell script (grep) works as desired returning just the contents of the "access_token" value.

{"access_token":"eyJhbGciOiJSandNoThisIsntRealndmbS1yZWFsbSI6eyJyb2xlcyI6WyJtYW5hZ2UtY2xpZW50cyIsInZpZXctcmVhbG0iLCJtYW5hZ2UtZXZlbnRzIiwidmlldy1ldmVudHMiLCJ2aWV3LWFwcGxpY2F0aW9ucyIsInZpZXctdXNlcnMiLCJ2aWV3LWNsaWVudHMiLCJtYW5hZ2UtdXNlcnMiLCJtYW5hZ2UtYXBwbGljYXRpb25zIiwibWFuYWdlLXJlYWxtIl19LCJtYXN0ZXItcmVhbG0iOnsicm9sZXMiOlsibWFuYWdlLWV2ZW50cyIsIm1hbmFnZS1jbGllbnRzIiwidmlldy1yZWFsbSIsInZpZXctZXZlbnRzIiwidmlldy1hcHBsaWNhdGlvbnMiLCJ2aWV3LXVzZXJzIiwidmlldy1jbGllbnRzIiwibWFuYWdlLXJlYWxtIiwibWFuYWdlLXVzZXJzIiwibWFuYWdlLWFwcGxpY2F0aW9ucyJdfX19.fQmQKn-xatvflHPAaxCfrrVow3ynpw0sREho7__jZo2d0g1SwZV7Lf4C26CcweNLlb3wmKHHo63HRz35qRxJ7BXyiZwHgXokvDJj13yuOb6Sirg9z02n6fwGy8Iog30pUvffnDaVnUWHfVL-h_R4-OZNf-_YUK5RcL2DHt0zUXI","expires_in":60,"refresh_expires_in":1800,"refresh_token":"eyJhbGciOiJSUzI1NiJ9.eyJqdGkiOiJlNWFmYTZiOC04ZjM5LTQ5MjUtOWZiMC00MmY3MTM4YzUzMGIiLCJleHAiOjE0NDY4Mjk3OTksIm5iZiI6MCwAreYouKiddingIwouldnotputSOmethigRealHereNpb25fc3RhdGUiOiI2MmVmYzA1Yy0xYmY1LTRmNTUtYjc0OS01ZTBlZmY5NDE1NWIiLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiYWRtaW4iLCJjcmVhdGUtcmVhbG0iXX0sInJlc291cmNlX2FjY2VzcyI6eyJ3Zm0tcmVhbG0iOnsicm9sZXMiOlsibWFuYWdlLWV2ZW50cyIsInZpZXctcmVhbG0iLCJtYW5hZ2UtY2xpZW50cyIsInZpZXctYXBwbGljYXRpb25zIiwidmlldy1ldmVudHMiLCJ2aWV3LXVzZXJzIiwidmlldy1jbGllbnRzIiwibWFuYWdlLXJlYWxtIiwibWFuYWdlLWFwcGxpY2F0aW9ucyIsIm1hbmFnZS11c2VycyJdfSwibWFzdGVyLXJlYWxtIjp7InJvbGVzIjpbInZpZXctcmVhbG0iLCJtYW5hZ2UtY2xpZW50cyIsIm1hbmFnZS1ldmVudHMiLCJ2aWV3LWFwcGxpY2F0aW9ucyIsInZpZXctZXZlbnRzIiwidmlldy11c2VycyIsInZpZXctY2xpZW50cyIsIm1hbmFnZS1hcHBsaWNhdGlvbnMiLCJtYW5hZ2UtdXNlcnMiLCJtYW5hZ2UtcmVhbG0iXX19fQ.WeiJOC1jQ52aKgnW8UN2Lv9rJ_yKZiOhijOYKLN2EEOkYF8rvRZsSKbTPFKTIUvjnwy2A7V_N-GhhJH4C-T7F5__QPNofSXbCNyvATj52jGLxk9V0Afvk-Z5QAWi55PJRTC0qteeMRcO2Frw-0KtKYe9o3UcGICJubxhZHsXBLA","token_type":"bearer","id_token":"eyJhbGciOiJSUzI1NiJ9.eyJuYW1lIjoiIiwianRpIjoiMGIyMGI0ODctOTI4OS00YTFhLTgyNmMtM2NiOTg0MDJkMzVkIiwiZXhwIjoxNDQ2ODI4MDU5LCJuYmYiOjAsImlhdCI6MTQ0NjgyNzk5OIwouldhaveToBeNutsUiLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJhZG1pbiIsImVtYWlsX3ZlcmlmaWVkIjpmYWxzZX0.DmG8Lm4niL1djzNrLsZ2CrsB1ZzUPnR2Nm7IZnrwrmkXsrPxjl6pyXKCWSj6pbk2sgVI8NNFqrGIJmEJ7gkTZWm328VGGpJsmMuJBki0KbqBRKORGQSgkas_34rwzhcTE3Iki8h_YVs2vvNIx_eZSOvIzyEcP3IGHuBoxcR6W3E","not-before-policy":0,"session-state":"62efc05c-1bf5-4f55-b749-5e0eff94155b"}

In case anyone finds this post, this is what I ended up using:

if hash jq 2>/dev/null; then
  # Use the jq command to safely parse json
  bearerToken=$(echo $raw_json | jq -r '.access_token')
else
  # Strip away all but the "access_token" field's value using a perl regular expression
  bearerToken=$(echo $raw_json | grep -Po '"'"access_token"'"\s*:\s*"\K([^"]*)')
fi
3
  • 7
    Note that grep is not the best (or even a good) tool for working with JSON. Get something like jq instead, which already knows how to parse JSON. bearerToken=$(echo "$raw_json" | jq '.accessToken') is far better.
    – chepner
    Nov 6, 2015 at 19:17
  • @chepner Thanks. I'm within a vagrant/puppet/centos environment. Perhaps I can yum install jq.
    – D-Klotz
    Nov 6, 2015 at 19:21
  • @chepner sudo yum install jq to the rescue. Thanks!
    – D-Klotz
    Nov 6, 2015 at 19:27

2 Answers 2

114

Since not all regex flavors support lookbehind, Perl introduced the \K. In general when you have:

a\Kb

When “b” is matched, \K tells the engine to pretend that the match attempt started at this position.

In your example, you want to pretend that the match attempt started at what appears after the "access_token":" text.

This example will better demonstrate the \K usage:

~$ echo 'hello world' | grep -oP 'hello \K(world)'
world
~$ echo 'hello world' | grep -oP 'hello (world)'
hello world

In addition, \K allows a variable-length look-behind:

$ echo foooooo bar | grep -oP "(?<=foo+) \Kbar"
grep: lookbehind assertion is not fixed length

$ echo foooooo bar | grep -oP "foo+ \Kbar"
bar
3
  • Thanks. I'm also curious about the spam of double quotes and single quotes. It does work, but I'm not sure how
    – D-Klotz
    Nov 6, 2015 at 19:20
  • 4
    The regular expression is a bit overquoted. It starts with '"', which is just a single double-quotation mark in a single-quoted string. Next comes a double-quoted string containing access_token; the two strings are simply concatenated together. Finally comes a single-quoted string that contains a few double-quotation marks. The shell concatenates the contents of three strings together; for example, 'foo'"bar"'baz' represents the same thing as "foobarbaz". The entire thing could be more simply written '"access_token"\s*:\s*"\K([^"]*)'.
    – chepner
    Nov 6, 2015 at 19:32
  • 2
    This seems like the same thing as \zs in vim
    – xdhmoore
    Jul 17, 2019 at 21:24
1

My solution was: sed -n 's/cut off this part \(display this part only\) cut off this part/\1/gp'

References:

  1. https://www.cyberciti.biz/faq/unix-linux-sed-print-only-matching-lines-command/
  2. info sed (texinfo package)
  3. man 1 sed

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.