0

I went through enough of threads and post on this topic but somehow its not helping me add unicode support to my code. I have very simple task to do - read the Unicode file (.txt and csv) - Parse it and store the word as tokens in 2D array using some delimiters (, or " separated words) - Perform some operations on it - store these strings text file

Problem i am facing is some of my older code functions are not compatible i guess as i don't find substitute or i am able to compile them but no out put generated. This code works perfectly fine with ASCII but now i need unicode support for it.

It would be great if i get sample source code ,does not need to be whole big code but at least like how to get Unicode file parse it and store it in token and which functions to use for comparison etc,

I am pasting some part of code below , i did modify few things so may not compile in first go.

get the text file as input e.g. profiles.txt which is in unicode (UTF 16 - basically Chinese and Korean words in it)


// adding all std headers here


const int MAX_CHARS_PER_LINE = 4072;  
const int MAX_TOKENS_PER_LINE = 1;      
const wchar_t* const DELIMITER = L"\"";

class IntegrityCheck
{
    public:
        std::wstring Profile_Container[5000][4];
        void Profile_PRD_Parser();
};

 void IntegrityCheck::Profile_PRD_Parser()
{

std::wstring skip (L".exe");
std::wstring databoxtemp[1][1];
int a=-1;

// create a file-reading object
wifstream fin.open("profiles.txt");  //open a file
wofstream fout("out.txt");  // this dumps the parsing ouput 

// read each line of the file
while (!fin.eof())
{
    // read an entire line into memory
    wchar_t buf[MAX_CHARS_PER_LINE];

    fin.getline(buf, MAX_CHARS_PER_LINE);

    // parse the line into blank-delimited tokens
    int n = 0; // a for-loop index

    // array to store memory addresses of the tokens in buf
    const wchar_t* token[MAX_TOKENS_PER_LINE] = {}; // initialize to 0

    // parse the line
    token[0] = wcstok(buf, DELIMITER); // first token

    if (token[0]) // zero if line is blank
    {

        for (n = 0; n < MAX_TOKENS_PER_LINE; n++)   // setting n=0 as we want to ignore the first token
        {
            oken[n] = wcstok(0, DELIMITER); // subsequent tokens

            if (!token[n]) break; // no more tokens

            std::wstring str2 =token[n];

            std::size_t found = str2.find(str);  //substring comparison

            if (found!=std::string::npos)   // if its exe then it writes in Dxout for same app name on new line
            {
                a++;
                Profile_Container[a][0]=token[n];
                std::transform(Profile_Container[a][2].begin(), Profile_Container[a][2].end(), Profile_Container[a][2].begin(), ::tolower);  //convert all data to lower 

                fout<<Profile_Container[a][0]<<"\t"<<Profile_Container[a][1]<<"\t"<<Profile_Container[a][2]<<"\n"; //write to file
            }

        }
    }

}

fout.close();
fin.close();
}

int main()
{
IntegrityCheck p1;
p1.Profile_PRD_Parser();
}     
13
  • There's a typo, the word is spelled "Integrity", not "Intigrity". Dec 2, 2013 at 9:25
  • If you use already using namespace std; then theres no reason to also write using std::cout; and so on. You're using already the whole std namespace.
    – Constantin
    Dec 2, 2013 at 9:28
  • Just remove the using namespace std line. It does not "add all std headers". I would not recommend using it if you know what it does, but that comment betrays that you don't know what it does, so I I have to make an even stronger recommendation to not use that. Dec 2, 2013 at 9:39
  • 1
    First thing is to remove every mention of char. Don't cast to char when calling getline, use wcstok not strtok.
    – john
    Dec 2, 2013 at 9:41
  • "now i need unicode support for it." is not a good description of a problem. What do you want to do with the data? How do you expect the input to be encoded? What platform is this? (wsomething does not magically make things "support Unicode") Dec 2, 2013 at 9:43

2 Answers 2

0

Looking quickly over your code the only changes I see are

const wchar_t* const DELIMITER = L"\"";

fin.getline(buf, MAX_CHARS_PER_LINE);

token[0] = wcstok(buf, DELIMITER);

std::transform(Profile_Container[a][2].begin(), Profile_Container[a][2].end(), Profile_Container[a][2].begin(), ::towlower); 

Not sure that towlower will be able to convert every Unicode character to lower case, but if your text is Chinese and Korean I guess that's not so much of an issue.

EDIT

The following is necessary on Windows with Visual Studio 2010

#include <codecvt>
#include <locale>

wifstream fin("profiles.txt", ios_base::binary);  //open a file
fin.imbue(std::locale(fin.getloc(),
   new std::codecvt_utf16<wchar_t, 0x10ffff, std::consume_header>));

This worked for me with a file encoded in UTF-16 'big endian' (but not in little endian).

The only problem with your current code is the file reading (and maybe writing I haven't looked at that). Once you can get your characters into strings from the file it should be OK.

If the above doesn't work for you then I'm not sure. This page has the gory details.

26
  • ,Thanks John for being more specific, I would make changes and see how it goes and update to thread
    – NxC
    Dec 2, 2013 at 16:42
  • i made corrections in code, its comipling but i am missing something, basically getline gets the line from the unicode file and then i try to break it in tokens (using delimiters) but i see that getline gets everything in binary and henc it fails to break the buffer into tokens, and also the comparison fails and i get blank output, do i need toconvert it back to ASCII then?but it will loose data right? so how should i approach this? i wrote all logic keeping simple ASCII strings in mind and now its making difficult. any suggestions to make it working more than welcome
    – NxC
    Dec 3, 2013 at 17:47
  • Or do you suggest alternate way to do this? i searched but could not find related article or sample code on unicode text file parsing
    – NxC
    Dec 3, 2013 at 18:12
  • I am really stuck with this unicode now, can some one please help me with it?
    – NxC
    Dec 3, 2013 at 23:47
  • @NeileshC I tried your code and was surprised that it didn't work (for me). The problem is that just using wchar_t is not enough to tell the compiler that your file is UTF-16. There doesn't seem to be any completely platform independent way to do this so the way to proceed depends on your compiler etc. I've updated the answer above with some code that worked for me.
    – john
    Dec 4, 2013 at 8:06
0

Final code that comipled and ran:

fin.imbue(std::locale(fin.getloc(), new std::codecvt_utf16<wchar_t, 0x10ffff,std::codecvt_mode(std::little_endian|std::consume_header)>));
fout.imbue(std::locale(fin.getloc(), new std::codecvt_utf16<wchar_t, 0x10ffff,std::codecvt_mode(std::little_endian|std::consume_header)>));


while (!fin.eof())
{

 wchar_t buf[MAX_CHARS_PER_LINE];

 fin.getline(buf, MAX_CHARS_PER_LINE);

 wchar_t* token[MAX_TOKENS_PER_LINE] = {};
token[0] = wcstok(buf, DELIMITER);


if (token[0]) // zero if line is blank
{
    int n = 0; 
    for (n = 0; n < MAX_TOKENS_PER_LINE; n++)   // setting n=0 as we want to ignore the first token
    {
        token[n] = wcstok(0, DELIMITER); // subsequent tokens


        if (!token[n]) break; // no more tokens

        std::wstring str2 =token[n];

        std::size_t found = str2.find(str);  //substring comparison

        if (found!=std::string::npos)   // if its exe then it writes in Dxout for same app name on new line
        {  
            a++;
            Profile_Container[a][0]=token[n];
            fout<<Profile_Container[a][0];
        }
    }
}

}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.