5

This question is in continuation to another question about selectively appending lines from one file to another.

The regex that I'm using works just fine at matching the lines to keep/to discard. The problem is that the file was composed from a bunch of other files, and sometimes the line I want to keep started out as the first line of a UTF-8 encoded file. This means that the findstr command returns something like:

LineToKeep that started out as the first line in its file
LineToKeep another
LineToKeep more lines
LineToKeep that started out as the first line in its file
LineToKeep more

It's guaranteed that excepting the BOM bytes, the line will always begin with "LineToKeep". How can I get rid of those three UTF-8 BOM bytes, since these windows shell commands can't properly handle them?

I'm hoping for a way to remove them in place, or perhaps a modification to the findstr command from that previous question.

Since I know each line must begin with "LineToKeep" or "LineToKeep", I figure there's a way to compute something like if (Line[3:10] == "LineToKeep") { Line = Line[3:]; } for every line.

3
  • 1
    ..and this is why perl should ship with every OS ;-)
    – user166390
    Commented Oct 12, 2012 at 1:32
  • Have you thought about using Powershell instead of the vanilla command interpreter? Doing something like this would be trivial. The Windows shell doesn't like Unicode much.
    – kprobst
    Commented Oct 12, 2012 at 1:39
  • I'm going to run the script in Visual Studio's pre/post-build events. According to stackoverflow.com/questions/6500320/…, running a PowerShell script may be an option. It would feel kind of like an external solution though.
    – Kache
    Commented Oct 12, 2012 at 5:28

2 Answers 2

2

Another alternative from unix world that removes the BOM in file in-place:

sed -zbi "1s/^\xEF\xBB\xBF//" filepath

This requires to download sed 4.4 for windows from https://github.com/mbuilov/sed-windows which offers working -z and -b options which prevent corruption of line endings.

1

I ended up calling PowerShell in windows cmd:

powershell . "Get-ChildItem . | Select-String '^LineToKeep' | foreach {$_.Line}"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.