4

I'm using Git for Windows (and TortoiseGit).

My goal is to prevent commits which have at least one non-UTF-8 file among modified/added.

  • Enumerating modified/added files: I've found the following code

    { git diff --name-only ; git diff --name-only --staged ; }
    

    Is this the best (correct and most concise) approach?

  • Searching for non-UTF-8 files: I've found the following code

    { git diff --name-only ; git diff --name-only --staged ; } | xargs -I {} bash -c "iconv -f utf-8 -t utf-16 {} &>/dev/null || echo {} - is non-UTF8!"
    

    If I start Git Bash at my repository root folder - it works (each non-UTF-8 file is displayed). So I've renamed .git/hooks/pre-commit.sample to .git/hooks/pre-commit and copy-pasted the code above. After committing changes nothing special displays inside TortoiseGit commit gui window. So looks like pre-commit hook is not working correctly.

  • Rejecting commit if there is any non-UTF-8 file: After displaying all non-UTP-8 files commit should be rejected. But I have no idea how to do this (show some exit code - but how?).

So any help is appreciated.

2
  • pre-commit: Exiting with a non-zero status from this script causes the git commit command to abort before creating a commit. So: { echo {} - is non-UTF8! >&2; exit 1; }.
    – phd
    Apr 12, 2019 at 10:10
  • @phd >&2 part shows an error for me. Anyway { echo {} - is non-UTF8!; exit 1; } works perfect (shows all info that I need and aborts commit) inside TortoiseGit commit gui window. Thank you.
    – bairog
    Apr 12, 2019 at 11:48

2 Answers 2

5

So the answer is (thanks to phd and great thanks to torek for their useful notes):

    git diff --name-only --staged --diff-filter d | xargs -I {} bash -c 
 "iconv -f utf-8 -t utf-16 {} &>/dev/null || { echo {} - is non-UTF8!; exit 1; }"

This code iterates through all files, that changed in commit (except for deleted - i.e. added, modified, copied and renamed) and checks if there is any non-UTF8 file. All found files are listed and commit is aborted.

2

Your existing solution is probably sufficient. It's not 100% correct though: here are the remaining issues, all of which are minor ones that you can fix later (if ever) at your leisure:

  • You need only the git diff ... --staged (or --cached), as what Git will commit is whatever files are in the index/staging-area, and git diff compares that with what's in the HEAD commit and tells you what's different there. If a copy of a file in the index differs from the copy of the file in HEAD, you should examine the index copy.

  • Technically it would be better to use git diff-index --cached here so as to not obey any of the user's git diff configuration. That is, git diff-index is a plumbing command in Git, which means it's aimed at being used from other computer programs: it runs in a completely predictable manner based on arguments only, not on any git config settings. But if you're doing this for yourself, and you configure git diff such that it breaks your own use of git diff, well, that's your own fault. :-)

  • You might also consider using a --diff-filter to exclude deleted files here. Otherwise your checker will always fail on deletion (as iconv won't be able to read the deleted file).

  • Most significant: iconv will be reading the file from the work-tree. As I noted in the first bullet point, Git is going to commit what's staged, not what's in the work-tree.

As an example—which may or may not be possible from within TortoiseGit—consider what happens if you do this:

$ git checkout master
$ printf '\300\300\300' > badfile    # put bad non-UTF-8 crud into file
$ git add badfile                    # copy file into index
$ echo 'good data' > badfile         # replace work-tree contents
$ git commit

This commit is going to commit the bad contents—the three bytes of \300 with no newline—that are in the index, but your pre-commit hook is going to run iconv -f utf-8 -t utf-16 over the contents of the good file, reading good data, that is of course good.

To fix this, your pre-commit filter must extract the data from the index for each file that is to be committed. How you go about doing that is up to you. The simplest (but perhaps slowest) method is to just extract the entire index contents to a temporary work area using git checkout-index. A better method might be to turn each in-index (in-staging-area) path name to valid index specifier (that is, path/to/file becomes :path/to/file) and use git cat-file -p $specifier | iconv ... to scan each. But all of these will be fairly inefficient, especially on Windows. For efficiency, you might want to write a Python script that uses git cat-file --batch to extract them all in one pass, and do the format-checking there.

3
  • Best might be to just cat-file --batch them, git diff-index --cached @ | awk '$4~/[^0]/{print $4}'|git cat-file --batch|iconv -f utf-8 -t utf-8 -o /dev/null.
    – jthill
    Apr 12, 2019 at 19:46
  • I confess I didn't read through to your last sentence. I think this should be the first thing done, because it'll be fastest and 99+% of the time it'll be enough. Going back and digging out exactly which file can be done separately, if needed.
    – jthill
    Apr 12, 2019 at 20:53
  • @jthill: That's a valid point. Using iconv is probably faster than doing this in Python, too, although I'd have to measure to be sure.
    – torek
    Apr 12, 2019 at 21:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.