18

How can I use shell one-liners and common GNU tools to concatenate lines in two files as in Cartesian product? What is the most succinct, beautiful and "linuxy" way?

For example, if I have two files:

$ cat file1
a
b
$ cat file2
c
d
e

The result should be

a, c
a, d
a, e
b, c
b, d
b, e
2
  • Oh no, it's turned into a competition ...
    – C. Ross
    Oct 26, 2009 at 20:31
  • 1
    @C. Ross , it didn't. I had a definite and expressed criterion of not using perl, python, etc. And the rest was only a usual battle for maintainability. simplicity and clarity.
    – P Shved
    Oct 26, 2009 at 21:10

14 Answers 14

16

Here's shell script to do it

while read a; do while read b; do echo "$a, $b"; done < file2; done < file1

Though that will be quite slow. I can't think of any precompiled logic to accomplish this. The next step for speed would be to do the above in awk/perl.

awk 'NR==FNR { a[$0]; next } { for (i in a) print i",", $0 }' file1 file2

Hmm, how about this hacky solution to use precompiled logic?

paste -d, <(sed -n "$(yes 'p;' | head -n $(wc -l < file2))" file1) \
          <(cat $(yes 'file2' | head -n $(wc -l < file1)))
3
  • 2
    @Pixelbeat: your first version needs to reverse the order of file1 and file2. (That is, it should be done < file2; done < file 1 to get the desired result.
    – Telemachus
    Oct 25, 2009 at 14:27
  • 3
    @Telemachus , the order is irrelevant: if I say "Cartesian product", I really mean it.
    – P Shved
    Oct 26, 2009 at 20:11
  • @HiteshPatel, I believe this may be useful to you. The only change you'll need to make the while read a-type answer work correctly is to add the -r argument, making it while read -r a; do while read -r b; do, since your content has literal backslashes. (@pixelbeat, you may want to edit said arguments into the answer proper). Feb 7, 2018 at 5:19
10

There won't be a comma to separate but using only join:

$ join -j 2 file1 file2
 a c
 a d
 a e
 b c
 b d
 b e
2
  • 1
    join -j 2 -o '1.1 2.1' -t ', ' file1 file2
    – Marcus
    May 31, 2017 at 1:30
  • @Marcus, it might be worth pointing out that if you downgrade this to a single separator, i.e. -t ,, it will also work with a number of non-GNU join implementations. The OP's proviso aside, the wider community appreciates portability in answers. We don't all run Linux. :)
    – ghoti
    Feb 24, 2018 at 21:50
7

The mechanical way to do it in shell, not using Perl or Python, is:

while read line1
do
    while read line2
    do echo "$line1, $line2"
    done < file2
done < file1

The join command can sometimes be used for these operations - however, I'm not clear that it can do cartesian product as a degenerate case.

One step up from the double loop would be:

while read line1
do
    sed "s/^/$line1, /" file2
done < file1
2
  • I'd go for the first solution because it doesn't make the files look like they're substantially different.
    – P Shved
    Oct 26, 2009 at 20:20
  • It (the first solution) would likely be substantially slower - but it would also be immune to odd characters (such as slashes) in the data. Fixing things so that is not a problem is a bit fiddlier, and at that point you start thinking about using Perl or Python after all. Oct 26, 2009 at 20:58
7

I'm not going to pretend this is pretty, but...

join -t, -j 9999 -o 2.1,1.1 /tmp/file1 /tmp/file2

(updated thanks to Iwan Aucamp below)

-- join (GNU coreutils) 8.4

1
  • you can eliminate use of cut by adding -o '2.1,1.1' (or whichever way round you like it) Mar 13, 2015 at 21:58
5

Edit:

DVK's attempt inspired me to do this with eval:

script='1{x;d};${H;x;s/\n/\,/g;p;q};H'
eval "echo {$(sed -n $script file1)}\,\ {$(sed -n $script file2)}$'\n'"|sed 's/^ //'

Or a simpler sed script:

script=':a;N;${s/\n/,/g;b};ba'

which you would use without the -n switch.

which gives:

a, c
a, d
a, e
b, c
b, d
b, e

Original answer:

In Bash, you can do this. It doesn't read from files, but it's a neat trick:

$ echo {a,b}\,\ {c,d,e}$'\n'
a, c
 a, d
 a, e
 b, c
 b, d
 b, e

More simply:

$ echo {a,b}{c,d,e}
ac ad ae bc bd be
2
  • nice. but i sure would not want to maintain this script. :)
    – ghostdog74
    Oct 25, 2009 at 23:49
  • Truly delightful, but unmaintainable. :)
    – P Shved
    Oct 26, 2009 at 20:18
3

a generic recursive BASH function could be something like this:

foreachline() {

    _foreachline() {

        if [ $#  -lt 2 ]; then
            printf "$1\n"
            return
        fi

        local prefix=$1
        local file=$2
        shift 2

        while read line; do
            _foreachline "$prefix$line, " $*
        done <$file
    }

    _foreachline "" $*
}

foreachline file1 file2 file3

Regards.

2
  • 2
    This solution is unique among these solutions in that it solves the more general case of an arbitrary set of cartesian product operations. Jun 4, 2015 at 13:39
  • 1
    The use of $* rather than "$@" is unfortunate, however; it means that any "*" as an argument will be replaced with a list of filenames, for example. Feb 7, 2018 at 5:20
2

Solution 1:

perl -e '{use File::Slurp; @f1 = read_file("file1"); @f2 = read_file("file2"); map { chomp; $v1 = $_; map { print "$v1,$_"; } @f2 } @f1;}'

6
  • Why did you use map here? Those should be for loops.
    – user181548
    Oct 25, 2009 at 14:05
  • @Kinopiko: Weren't you just complaining about "language police" on a different thread?
    – Telemachus
    Oct 25, 2009 at 14:24
  • The only thing I like to use more than maps is Regular Expressions. :)
    – DVK
    Oct 25, 2009 at 16:09
  • @Telemachus: If you can't beat them, join them.
    – user181548
    Oct 26, 2009 at 4:01
  • Language Police is right here: Language Cops are coming and busting you! :-)
    – P Shved
    Oct 26, 2009 at 20:16
2

Edit: Oops... Sorry, I thought this was tagged python...

If you have python 2.6:

from itertools import product
print('\n'.join((', '.join(elt) for elt in (product(*((line.strip() for line in fh) for fh in (open('file1','r'), open('file2','r'))))))))

a, c
a, d
a, e
b, c
b, d
b, e

If you have python pre-2.6:

def product(*args, **kwds):
    '''
    Source: http://docs.python.org/library/itertools.html#itertools.product
    '''
    # product('ABCD', 'xy') --> Ax Ay Bx By Cx Cy Dx Dy
    # product(range(2), repeat=3) --> 000 001 010 011 100 101 110 111
    pools = map(tuple, args) * kwds.get('repeat', 1)
    result = [[]]
    for pool in pools:
        result = [x+[y] for x in result for y in pool]
    for prod in result:
        yield tuple(prod)
print('\n'.join((', '.join(elt) for elt in (product(*((line.strip() for line in fh) for fh in (open('file1','r'), open('file2','r'))))))))
1
  • That would work, but python is not what I've been asking for.
    – P Shved
    Oct 25, 2009 at 13:52
2

A solution using join, awk and process substitution:

join <(xargs -I_ echo 1 _ < setA) <(xargs -I_ echo 1 _ < setB)
  | awk '{ printf("%s, %s\n", $2, $3) }'
3
  • What is the contents of the file "a"? Should one of them be a different file? The AWK could probably be replaced by cut -f2- -d' '. Jul 31, 2010 at 19:20
  • The "a" file contains the set. They may be different if wanted. I'll correct it!
    – yassin
    Jul 31, 2010 at 19:36
  • @Dennis, cut is probably better, since it works even if setB contains lines with whitespaces.
    – P Shved
    Aug 1, 2010 at 6:12
1
awk 'FNR==NR{ a[++d]=$1; next}
{
  for ( i=1;i<=d;i++){
    print $1","a[i]
  }
}' file2 file1

# ./shell.sh
a,c
a,d
a,e
b,c
b,d
b,e
1

OK, this is derivation of Dennis Williamson's solution above since he noted that his does not read from file:

$ echo {`cat a | tr "\012" ","`}\,\ {`cat b | tr "\012" ","`}$'\n'
a, c
 a, d
 a, e
 b, c
 b, d
 b, e
1
  • 1
    This is what that gives me: {a,b,}, {c,d,e,} as a literal string. Oct 25, 2009 at 20:17
1

GNU Parallel:

parallel echo "{1}, {2}" :::: file1 :::: file2

Output:

a, c
a, d
a, e
b, c
b, d
b, e
0

Of course perl has a module for that:

#!/usr/bin/perl

use File::Slurp;
use Math::Cartesian::Product;

use v5.10;
$, = ", ";

@file1 = read_file("file1", chomp => 1);
@file2 = read_file("file2", chomp => 1);

cartesian { say @_ } \@file1, \@file2;

Output:

a, c
a, d
a, e
b, c
b, d
b, e
0

In it's a one-liner

printf '%s\n' (cat file1)", "(cat file2)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.