So, this is slightly different from the other answers. I can't say that a C++ compiler is exactly a "Linux CLI tool", but running g++ -O3 -march=native -o set_diff main.cpp
(with the below code in main.cpp
can do the trick):
#include<algorithm>
#include<iostream>
#include<iterator>
#include<fstream>
#include<string>
#include<unordered_set>
using namespace std;
int main(int argc, char** argv) {
ifstream keep_file(argv[1]), del_file(argv[2]);
unordered_multiset<string> init_lines{istream_iterator<string>(keep_file), istream_iterator<string>()};
string line;
while (getline(del_file, line)) {
init_lines.erase(line);
}
copy(init_lines.begin(),init_lines.end(), ostream_iterator<string>(cout, "\n"));
}
To use, simply run set_diff B A
(not A B
, since B
is nodes_to_keep
) and the resulting difference will be printed to stdout.
Note that I've forgone a few C++ best practices to keep the code simpler.
Many additional speed optimizations could be made (at the price of more memory). mmap
would also be particularly useful for large data sets, but that'd make the code much more involved.
Since you mentioned that the data sets are large, I thought that reading nodes_to_delete
a line at a time might be a good idea to reduce memory consumption. The approach taken in the code above isn't particularly efficient if there are lots of dupes in your nodes_to_delete
. Also, order is not preserved.
Something easier to copy and paste into bash
(i.e. skipping creation of main.cpp
):
g++ -O3 -march=native -xc++ -o set_diff - <<EOF
#include<algorithm>
#include<iostream>
#include<iterator>
#include<fstream>
#include<string>
#include<unordered_set>
using namespace std;
int main(int argc, char** argv) {
ifstream keep_file(argv[1]), del_file(argv[2]);
unordered_multiset<string> init_lines{istream_iterator<string>(keep_file), istream_iterator<string>()};
string line;
while (getline(del_file, line)) {
init_lines.erase(line);
}
copy(init_lines.begin(),init_lines.end(), ostream_iterator<string>(cout, "\n"));
}
EOF