0

i am dealing with large dataset. May i ask you how it is possible to store strings in the classes i want to use with stxxl? I have read several discussions and everywhere was said that string is not POD type therefore it cannot be stored in the stxxl::vector, but i am not sure,because i tried it and i checked the data and everything seems to be fine. i have also saw an approach here https://github.com/Project-OSRM/osrm-backend/blob/725b93a961625a7b04d54806d7e0f80252a6bcd0/extractor/extraction_containers.hpp and they use stxxl::vector, so maybe the library got updated to support std::string?

class HighWay
{
private:
    uint64_t id; 
    string name;
    int speed;
    string attributes; //additional attributes of way
    string edges; //written uint64_t from,uint64_t to, int distance  written as string
    string nodes; //vector<uint64_t> written as string
public:
    HighWay() = default;
    void setId(uint64_t _id) {
        id = boost::lexical_cast<string>(_id);
    }
void setName(string _name) {
    name = _name;
}
void setSpeed(int _speed) {
    speed = _speed;
}
void setAttributes(string _attributes) {
    attributes = _attributes;
}
void setEdges(string _edges) {
    edges = _edges;
}

void setNodes(vector<uint64_t>refs) {
    stringstream s;
    uint64_t i = 0;
    for (; i < refs.size()-1;i++) {
        s << boost::lexical_cast<uint64_t>(refs[i]) << " ";
    }
    s << boost::lexical_cast<uint64_t>(refs[i]);

    nodes = s.str();
}

uint64_t getId() {
    return id;
}
string getName() {
    return name;
}
int getSpeed() {
    return speed;
}
string getAttributes() {
    return attributes;
}
string getEdges() {
    return edges;
}

std::vector<int64_t> getNodes() {
    stringstream s(nodes);
    uint64_t node;
    std::vector<int64_t> result;
    while (s >> node) {
        result.push_back(static_cast<int64_t>(node));
    }

    return result;
}
};

I have also created code which stores the strings as POD,storing the string in vector of char and in map remembering lower and upper bound index in the array. But this approach leads to many std::maps used in the application.

//class to store in map
struct TypeName{
    uint64_t start;
    uint64_t end;
};

std::istream& operator >> (std::istream& i, TypeName& entry)
{
    i >> entry.start;
    i >> entry.end;
    return i;
}
std::ostream& operator << (std::ostream& i, const TypeName& entry)
{
    i << entry.start << " ";
    i << entry.end;
    return i;
}

struct PoiCategories{
   uint64_t start;
   uint64_t end;
};

std::istream& operator >> (std::istream& i,PoiCategories& entry)
{
    i >> entry.start;
    i >> entry.end;
    return i;
}

std::ostream& operator << (std::ostream& i, const PoiCategories& entry)
{
    i << entry.start << " ";
    i << entry.end;
    return i;
}

 //object i want to store
struct Poi {
    Poi() = default;
    uint64_t id;
    char type;
    uint64_t id_in_pois; //id in vector pois

void addCategories(
    vector<int> &kats, //categories to insert
    stxxl::vector<uint64_t> &categories, //vector to store category
    std::unordered_map <uint64_t, PoiCategories> &idPoi_categories //index to vector categories to retrieve all categories for Poi
    )
{
    size_t  start = categories.size();
    for (auto & kat : kats) {
        categories.push_back(kat);
    }
    size_t end = categories.size() - 1;
    idPoi_categories.insert(make_pair(id, PoiCategories{start, end }));


}

vector<int> getCategories(
    stxxl::vector<uint64_t> &categories,
    std::unordered_map <uint64_t, PoiKategorie> &idPoi_categories
    )
{
    std::vector<int> result;
    PoiCategories bounds = idPoi_categories.find(id)->second;
    for (size_t i = bounds.start; i <= bounds.end; i++) {
        result.push_back(categories[i]);
    }

    return result;
}

};

Problem in my application is that i am storing a few string data, which are mainly names of streets and POIs. Maybe i am using wrong library. If so,can you recommend me a better approach to store data while preprocessing?

1 Answer 1

0

It's indeed banned, but the symptoms of violating the no-POD rule are generally unpredictable. It may very well appear to work as long as the strings all fit in memory, but in that case you didn't need the STXXL anyway.

3
  • I understand, is there any better way to solve this? Stxxl library doesn´t like a viable solution to my problem anymore. Do you have any experience with this DB for instance? ssdb.io They say its something like redis, so maybe it can be used for the preprocessing or is there any better approach that i don´t know about? My problem is i need to store the data temporarily for preprocessing before i insert the final data into DB and its way larger dataset than amount of installed RAM can handle.
    – brano199
    May 11, 2015 at 20:47
  • No experience with those. When I had such a problem, I very aggressively optimized the problem to make it fit in RAM. That included loading all strings which I didn't need to edit into a single block of memory, and storing char const* const into that block. Also, I split my algorithm into multiple passes each of whioch did fit into memory.
    – MSalters
    May 11, 2015 at 20:51
  • Yeah, i am also using multiple passes to get what i need, in fact i need to read the entire file two times. The problem is that if i store the part of a preprocessed data in database, the update operation for such a large dataset is time expensive. So because of the speed i had to keep in memory even the preprocessed files so i could add additional data which i discovered later to them.
    – brano199
    May 12, 2015 at 11:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.