3

Given the following command:

git ls-files | xargs perl -i -pe 's/SEARCHTERM/REPLACETERM/g'

All outputs to perl (from git ls-files) that were symbolic links are now copies of the target files.

I have two questions:

1) I think I vaguely understand why this happened, but only vaguely. Can someone explain in detail? And suggest the best mechanism for avoiding this? Expected behavior would be that the symbolic link targets would be the target of the read AND the write--not just the read.

2) Is there a better general approach for doing a search and replace on a local git branch?

Might be worth noting that my bash is pretty rudimentary and xargs is just my default goto when I want to ensure the contents of files are processed rather than a list of files. Not used for any reason other than that.

0

5 Answers 5

4

It says in perlrun

Note that because -i renames or deletes the original file before creating a new file of the same name, Unix-style soft and hard links will not be preserved.

So one cannot do that with -i.

Here is another way with Perl (as tagged -- even as there is a clean solution with sed)

I use files a.txt, b.txt, their symlinks (ln -s a.txt ln_a.txt etc), and c.txt (and any content is fine for this test), and list names of links and c.txt in a file

ln_a.txt ln_b.txt c.txt    # file "input_list.txt"

where file/link names in input_list.txt may be separated by spaces or newlines.

Then, open a temporary output file on the first line of each input file, and write each processed line to it. Once the end of the input file is reached rename that temporary output to its input file, or to its target if it is a link. So for each input file overwrite the file, or its target if link, with the output file.

cat input_list.txt | xargs perl -MPath::Tiny -ne'
    if ($.==1) { $tf = $ARGV."_tmp.$$"; $fh = path($tf)->openw };
    s/(\w+)/$1-NEW/; 
    print $fh $_; 
    if (eof) { close ARGV; rename $tf, (-l $ARGV ? readlink $ARGV : $ARGV) }
'

This changes the content of targets and leaves the links alone. It works for regular files as well.

The makeshift output file name (filename_tmp.$$) can be made properly with File::Temp, or rather with Path::Tiny::tempfile since that module is being used already.

The rename should probably be changed to move from File::Copy, for portability.

The eof as used checks whether the file is exhausted for each input file, at which point the output file is renamed to the input file or to its target. The -l is a file-test operator that tests whether the file on hand is a symbolic link, and if it is then readlink resolves the link.

It is safe to rename the input file or target at that time since it has been read and processed.

The $ARGV is the name of the currently processed file and ARGV is the filehandle for it.

The explicit close ARGV resets the line counter so we can open the temporary output at the beginning of each new input file by testing the line number counter $. against 1.

2
  • Thanks for this. I hope I don't ever have to use it, but if I do, it'll be invaluable.
    – zzxyz
    Nov 2, 2018 at 18:58
  • 1
    @zzxyz Yeah, good that there is that sed feature and the answer that unveiled it :). This is just a simple-minded (kinda) implementation of -i which honors symlinks. It contains a bunch of tidbits which I hope are generally useful.
    – zdim
    Nov 2, 2018 at 22:23
2

If you're not opposed to using sed, give this a whirl:

git ls-files | xargs -I{} -P4 sed --follow-symlinks -i'' 's/SEARCHTERM/REPLACETERM/g' {} 
3
  • Same result, I'm afraid. (Although I had to remove the '' fwiw)
    – zzxyz
    Nov 2, 2018 at 1:29
  • Yeah that worked. Thanks! Also "follow-symlinks" was a useful google search and led me to this: unix.stackexchange.com/questions/9318/…
    – zzxyz
    Nov 2, 2018 at 1:33
  • @zzxyz That's pretty boss! Def going to read more about moreutils.
    – Rafael
    Nov 2, 2018 at 1:37
2

Another Perl option:

git ls-files | xargs perl -MPath::Tiny -Mutf8 -E'
  path($_)->realpath->edit_lines_utf8(sub { s/SEARCHTERM/REPLACETERM/g })
  for @ARGV'

Or without xargs, just read filenames from STDIN:

git ls-files | perl -MPath::Tiny -Mutf8 -E'
  path($_)->realpath->edit_lines_utf8(sub { s/SEARCHTERM/REPLACETERM/g })
  for map { chomp; $_ } readline'

realpath ensures you are always working with the symlink target, and edit_lines is essentially a Path::Tiny implementation of the -i option. Using edit_lines_utf8 and -Mutf8 means that your source code (the search and replace terms) and the file contents will be decoded from UTF-8 while running the search/replace, which is usually helpful (but if your files aren't UTF-8 encoded, remove -Mutf8 and use edit_lines). You could also filter out non-text files based on the -T heuristic by adding grep { -T } after for.

As a bonus, edit_lines will always output to a new file then rename it over the original which is safer than clobbering the original file, which -i didn't do until Perl 5.28.

CAVEAT: Path::Tiny will set the permissions of all the files it edits based on your current umask, not the existing permissions; I opened an issue to see if it's considered a bug.

2

All you need is

git ls-files | xargs readlink -e | xargs perl -i -pe'...'

You could also expand the links in Perl too.

git ls-files | xargs perl -i -pe'BEGIN { @ARGV = map readlink($_) // $_, @ARGV } ...'
0
1

My initial solution to this was just letting the symlinks get clobbered, and then running:

git status --porcelain | awk '{if ($1 == "T"){print $2}}' | xargs git checkout

This is really good for various tools that might clobber the symlinks if they happen to be incidental and not really relevant to the search.

The link here has some interesting ideas: https://unix.stackexchange.com/questions/9318/is-there-a-way-to-make-perl-i-not-clobber-symlinks

zdim's answer is the best implementation of one of them, I think. A little hairy, though.

The other answer at that link, using sponge, works, but creates a very complicated xargs pipe. It also, I believe requires the use of xargs -n1, which is a catastrophically slow way to run perl on large file systems. So that's a no-go.

And Rafael's answer (just use sed) is probably...the sanest way to approach this problem :)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.