I'm trying to create a PowerShell script that would, for each binary file in a folder : 1) read a short string at a fixed offset near the beginning (for instance 16 bytes at offset 1024) ; 2) copy that string as hex values to a text file (preferrably in the form “\xFC\x70\x28\x4C\x00”, or at least in a form that can be easily edited as such), with a line break. The goal is to create a list of search terms for WinHex, to find matches between two groups of files from a data recovery made with R-Studio.
I did the same request (with some added context) at SuperUser, but couldn't get quite what I need. Based on someone's suggestion I tried this :
foreach ($file in gci *.mts, *.vob, *.mpg) {
$16Bytes = [System.Text.Encoding]::Default.GetString([System.IO.File]::ReadAllBytes("$file"), 1024, 16)
Add-Content -path "G:\search terms.txt" -value $16Bytes
}
It works but that was before I realized that a list of search terms in ASCII characters wouldn't be reliable (issues with null bytes), so I need to convert the output to hexadecimal values. Someone linked this article, but I got errors when trying to add the [System.BitConverter]::ToString
command to the script above, and couldn't get further. On this page I found a way to convert a byte array to a hexadecimal string formatted like I need : [bitconverter]::ToString($16Bytes).Replace("-", "\x");
— but again I fail to combine this with the other commands.
The other issue is that the above script seems to be reading each file in its entirety before extracting the requested string, making it very inefficient, as there's about 1TB of files to process. I would need a way to parse only the requested portion of each file, which should be very quick. Based on this thread, Get-Content
would have the same drawback, but no better approach was provided there, so again I got stuck. Here I found more specific information but it's way too vague to be of any practical use.
For extremely large files, though, it may be unacceptably slow to load the entire file into memory when you work with it. If you begin to run against this limit, the solution is to use file management classes from the .NET Framework. These classes include BinaryReader, StreamReader, and others.
My current knowledge of PowerShell is very limited, my understanding of each part of each command is very fuzzy, which makes it very painstaking to assemble these snippets of code gathererd here and there in a way that makes sense, and most of the time when I attempt to run a command I get a big dump of red lines, error warnings that I don't understand, and since they're in french I can't even use them as search terms to find a possible solution since most resources about PowerShell are in english. This is all very frustrating, especially knowing that it drives me further and further away from the task I want to accomplish, which may not even deserve that kind of effort in the first place...
Then, if I manage to accomplish this first part, and if the search with WinHex works as intended, an extra step would be to perform automated checksum comparisons. WinHex can do a “logical search” within a whole volume, meaning that for each search hit, it can report the absolute offset (relative to the start of the partition) as well as the file offset (where the searched string was found within a file identified through that partition's filesystem, even if it is fragmented or NTFS-compressed). So, once I have the list of search hits, with the path / names of the files and the offsets where the string was found, what I would like is to : 1) compute the MD5 checksum of file “A” (the one from which the search term was copied) ; 2) compute the MD5 checksum of a block in file “B” (the one where a hit for the search term was found) which supposedly coincides with file “A” ; 3) print the result to a report file, and indicate if both values match or not. If the MD5 checksums match, it means that file “A” is totally and exactly included in file “B”, and can therefore be deleted ; if not, either it's a false positive (the search term wasn't specific enough, or the original file was fragmented so the recovered file may contain foreign data), in which case it has to be manually checked. [EDIT] To do that I would have to define, for each pair of files, in a loop, a block within file B starting at {offset where the hit was found in file B} - {offset where the search string was copied from file A}
, and ending at {starting offset} + {size of file A} - 1
. Then calculate the MD5 checksum of this block in file B, the MD5 of the whole file A, and report if both values match or not. For instance : if WinHex finds a hit for the search term obtained at offset 1024 from file 12345.mpg inside VTS_01_1.VOB at offset 1049600, and the size of 12345.mpg is 20971520, then I would need to calculate the MD5 of a block in VTS_01_1.VOB starting at 1049600 - 1024 = 1048576
and ending at 1048576 + 20971520 - 1 = 22020095
; and compare the result with the MD5 of the whole 12345.mpg file. [/EDIT] I already know a command line tool called dsfo which can calculate the MD5 checksum of a block within a file, so I could use that as a workaround, but it would be more streamlined to do it all in PowerShell.
Thanks.
EDIT :
When I try to add the ToString command :
foreach ($file in gci *.wmv) {
$16Bytes = [System.Text.Encoding]::Default.GetString([System.IO.File]::ReadAllBytes("$file"), 1024, 16)
$var = [bitconverter]::ToString($16Bytes).Replace("-", "\x"); Add-Content -path "G:\search terms.txt" -value $16var
}
I get this error for each file :
Impossible de convertir l'argument « 0 » (valeur « ») de « ToString » en type « System.Byte[] » : « Impossible de convertir la valeur « » en type « System.Byte[] ». Erreur : « Impossible de convertir la valeur « » en type « System.Byte ». Erreur :
« Le format de la chaîne d'entrée est incorrect. » » »
Au niveau de ligne : 3 Caractère : 32
+ $var = [bitconverter]::ToString <<<< ($16Bytes).Replace("-", "\x"); Add-Content -path "G:\search terms.txt" -value $16var
+ CategoryInfo : NotSpecified: (:) [], MethodException
+ FullyQualifiedErrorId : MethodArgumentConversionInvalidCastArgument
UPDATE 20200628 #1 :
Based on the suggestions by Theo, after installing PowerShell 5.1, I tested this script on a folder containing 8 MKV video files :
foreach ($file in gci *.mkv) {
$buffer = [Byte[]]::new(16)
$stream = [System.IO.FileStream]::new($file.FullName, 'Open', 'Read')
$stream.Position = 65536
$readSize = $stream.Read($buffer, 0, 16)
$stream.Dispose()
if ($readSize) {
# create a hex string and write to file
$hex = for ($i = 0; $i -lt $readSize; $i++) { '\x{0:X2}' -f $buffer[$i] } -join ''
Add-Content -Path "G:\search terms.txt" -Value $hex
}
}
$buffer = $null
It works almost as intended, the values are correct (verified with WinHex), and it's very quick (took a split second to process all 8 files with a total of 2.64GB), but in the resulting text file there is only one byte per line, so it has 128 lines instead of 8. It looks like this :
\xE9
\xF6
\x4B
\x72
\x6C
\x6B
\x47
\xBC
\x48
\xD5
...
It should look like this :
\xE9\xF6\x4B\x72\x6C\x6B\x47\xBC\x48\xD5\x4A\x2B\x6E\x12\x8F\xDB
\x3C\x20\xD7\xFD\xB4\x63\x55\x57\xB3\x0D\x95\x99\x24\xF5\xCB\x85
\x72\x88\x2B\xBF\x0B\xC6\xE2\x5F\x1F\xB2\x72\x3C\xD3\xBB\x21\xA3
\xA4\x6F\x78\x6F\xA1\x2C\x1C\xD6\x17\x84\x3D\xDC\xB8\xBB\x20\x54
...
So how can I tweak this script to get this output ?
Also it would be nice to have a brief explanation for each command, particularly the $hex = ...
line.
UPDATE 20200628 #2 :
I tried this :
$hex1 = [System.BitConverter]::ToString($buffer).Replace("-", "")
$hex2 = for ($i = 0; $i -lt $readSize; $i++) { '\x{0:X2}' -f $hex1[$i] } -join ''
Add-Content -Path "G:\search terms.txt" -Value $hex2
But the output is now half a byte per line, and each line stops at half the 16 bytes string :
\xE => beginning of string 1
\x9
\xF
\x6
\x4
\xB
\x7
\x2
\x6
\xC
\x6
\xB
\x4
\x7
\xB
\xC
\x3
\xC => stop halfway through string 1
\x2 => beginning of string 2
\x0
...
Then if I do this :
$hex = [System.BitConverter]::ToString($buffer).Replace("-", "\x")
Add-Content -Path "G:\search terms.txt" -Value $hex
It works with PowerShell 5.1, the output is almost what's expected, there's only one “\x” missing at the beginning, I could work with that, and add the missing character to each line with a text editor. But it would be better to get the intended output right away, at this point it shouldn't be difficult.
E9\xF6\x4B\x72\x6C\x6B\x47\xBC\x48\xD5\x4A\x2B\x6E\x12\x8F\xDB
3C\x20\xD7\xFD\xB4\x63\x55\x57\xB3\x0D\x95\x99\x24\xF5\xCB\x85
72\x88\x2B\xBF\x0B\xC6\xE2\x5F\x1F\xB2\x72\x3C\xD3\xBB\x21\xA3
A4\x6F\x78\x6F\xA1\x2C\x1C\xD6\x17\x84\x3D\xDC\xB8\xBB\x20\x54
E4\x76\x1A\x79\x7C\x77\xA0\x83\x80\xCD\x46\x80\x78\x25\x64\x07
35\xD3\xBD\xDD\xD9\xA2\x5D\xF2\x70\x1A\x07\x72\xEE\xB4\xE0\x24
5A\x25\xA5\x48\x6A\x61\x48\xF5\xA3\xFD\x14\xED\x09\x84\xF9\xB4
47\x02\xF5\x81\x81\x6B\x54\xB0\xB1\x6F\xCD\x90\x11\xAB\x09\xA2
$var = [bitconverter]::ToString($16Bytes).Replace("-", "\x"); Add-Content -path "G:\search terms.txt" -value $var
. Is this what you want? I don't see an issue of the code that you've provided from superuser.echo text (or output of any command) | bash -c "md5sum"
or fullbash -c "echo test|md5sum"
and with variablebash -c "echo $var |md5sum"
and to perform operations on the variablebash -c "$($var.tolower()) | md5sum"
[byte[]]$Bytes1 = Get-Content "E:\ST9500325AS.txt" -Encoding byte
//[bitconverter]::ToString($Bytes1).Replace("-", "\x");
-- it works. I don't know what's wrong.