Questions tagged [stxxl]
STXXL is an open source out-of-core implementation of the STL, that is available for all major desktop operating systems under the Boost Software License 1.0.
52
questions
2
votes
0
answers
176
views
How to use a C++ 14 library in a C++ 23 project (MSVC)?
I'm writing a program that makes heavy use of C++ 20 and newer features. The problem is that in the project I need the stxxl library, which is compiled exclusively on C++ 14 . If I try to include it ...
0
votes
0
answers
103
views
stxxl exceeding OS limit of nr of threads
STXXL 1.4.1 on Windows 10 / VS2015.
I have some code in which I essentially use this:
typedef stxxl::vector<RC, 4, stxxl::lru_pager<8>, 2 * 1024> TVector;
typedef stxxl::deque<RC, ...
2
votes
0
answers
177
views
STXXL underperforms when using very fast SSDs
I'm currently writing a tool which uses the STXXL to find similarities between a large file on the hard drive and vectors residing in RAM. I wrote an example(for Windows) below to show what I mean. ...
0
votes
1
answer
202
views
Including STXXL library in the MakeFile
I was trying to install STXXL library to a custom path following this answer supplying prefix to cmake this way:
cmake -DCMAKE_INSTALL_PREFIX=/usr . && make all install
When I run the tests,...
1
vote
1
answer
711
views
OSRM extract and best swap stxxl location on 2 HDs
I want to set up the car profile data, but it is now taking 3 days and still at 100% Graphs (so way more to go) and I am thinking, maybe my swap and stxxl is just set up inappropriately.
I have the ...
0
votes
1
answer
140
views
STXXL fails at creating a map of strings in long numeric
With the code that follows, I want to create a map of type std::map. By using a debugger, I got that the exception was raised by stl_pair.h at line 368 (pair& operator=(typename conditional<...
0
votes
1
answer
457
views
Any STXXL alternatives to STXXL to handle unknown size at compile time, or any tricks?
This question and answers to it reassure me that STXXL handles only fixed data length. Are there any alternatives to STXXL to handle dynamic data size (data size unknown at compile time)?
Or are ...
1
vote
1
answer
318
views
Operating with huge array with random access to its elements
I have to work with very large array (about 2000^3) of float, moreover, sometimes I have to route through array elements randomly, or by another non-sequential way.
I use stxxl library and store my ...
0
votes
1
answer
313
views
Stxxl Vector as dropin replacement for std::vector
This is a followup of Vector of pairs with generic vector and pair type, template of template.
I'd like to be able to call a method with a std::vector or stxxl:vector, while template arguments of ...
1
vote
0
answers
187
views
Compiling STXXL on visual studio 2015
I am trying to use STXXL map in my c++ code in visual studio 2015.
I followed the instruction on http://stxxl.sourceforge.net/tags/master/install_windows.html'
and when I try to build the project in ...
1
vote
1
answer
117
views
C++: STXXL and VS runtime error in simple code
I have the following code which is a pretty simple test, but the VS refuses to run it:
stxxl::syscall_file OutputFile("Data/test.bin", stxxl::file::RDWR | stxxl::file::CREAT | stxxl::file::DIRECT);
...
1
vote
0
answers
249
views
Specify location to store stxxl::map/persist its data?
stxxl::vector has a constructor where you can provide a stxxl::file *. This ties the vector to that specific file:
typedef stxxl::VECTOR_GENERATOR<int>::result stxxl_vector_int;
{
stxxl::...
0
votes
1
answer
95
views
Combining Streams and Iteration in STXXL
STXXL supports a streaming model of data processing. STXXL data can either be processed by iterating through it (e.g. using stxxl::for_each), or by converting to a stream mode (e.g. using streamify() ...
1
vote
1
answer
290
views
Parallel STXXL Vector Initialization
The following minimal example illustrates the behaviour of stxxl when initializing containers in parallel (using openMP):
#include <omp.h>
#include <stdio.h>
#include <stxxl.h>
...
0
votes
0
answers
457
views
Stxxl map with std::string as key and std::vector as value
I need a large map of sets of numbers.
So i'm using stxxl and it's basically something like map<std::string, std::vector<int>>.
The key is 3-letters identifier.
There are about 3k enteries ...
0
votes
1
answer
163
views
Mapping an stxxl::vector to a File with Header
I have a large binary (STL) file that begins with a known and fixed size header and then continues with a bunch of fixed size buffers (PODs) representing consecutive records.
I want to map the ...
0
votes
1
answer
145
views
Changing the size of stxxl:vector cache
In my algorithm I have two stages:
1) Populating a file with records using a big data structure X that takes as much RAM as possible
2) Sorting the file. Now there is lot of memory since X is ...
1
vote
1
answer
244
views
STXXL: How to sort Vector of pairs on second element?
Similar Question is available here: How do I sort a vector of pairs based on the second element of the pair? but I am interested in External Memory Sorting.
I have tried using the analogies from ...
2
votes
2
answers
222
views
C++: Which Data Type in STXXL is suitable to create External Memory Binary Search Tree?
I want to create an External Memory Binary Search Tree Data Structure whose data sits in the external memory using stxxl as a library.
For this purpose, which Data Type in STXXL is suitable to use ...
0
votes
1
answer
90
views
Why do creating pointer to instances beyond certain number(30) of a 'Structure' with 'stxxl:Vector' as one of its DataType fails?
I am using STXXL Library's stxxl::vector in my code as :
struct B
{
typedef stxxl::VECTOR_GENERATOR<float>::result vector;
vector content;
};
And then creating many instances of the ...
0
votes
1
answer
294
views
STXXL Dealing with strings
i am dealing with large dataset. May i ask you how it is possible to store strings in the classes i want to use with stxxl? I have read several discussions and everywhere was said that string is not ...
0
votes
1
answer
512
views
stxxl save and read vector from disk file
I'm struggling in trying to use the stxxl library in a way, that I cannot only store the data from their vector structure into a file but also recover it from that file in a rerun of my program.
I ...
0
votes
1
answer
225
views
Parallelizing STXXL Container
I am trying to use STXXL to build a large matrix and be able to perform several different operations on the matrix. Currently, I'm using stxxl::vector as a way of storing submatrices of my large ...
3
votes
1
answer
657
views
stxxl sorting of very large file (ubuntu)
I am trying to sort a large file with about billion records (each containing four integers). The size of the file would shoot up beyond 50GB.
I am testing my code with 400 million records (about 6 GB ...
0
votes
1
answer
117
views
How to delete a STXXL priority queue?
I created a STXXL priority queue in my program.
If I do not need the PQ any more, how to delete it? Just like delete[] buffer in C++.
I want to free the memory used by it.
I looked through their ...
2
votes
3
answers
330
views
How to append a large number of elements to stxxl vector efficiently?
I need to append a large number of elements to a stxxl vector. What is the most efficient way of adding elements to a stxxl vector? Right now, I'm using push_back of the stxxl vector, but it doesn't ...
2
votes
2
answers
190
views
STXXL multidimensional vector makes program freeze
Trying to deal with huge data sets, I have been using the convenient STXXL library.Though I have run into a slight problem when trying to use/generate a multidimensional vector.
The following ...
1
vote
3
answers
842
views
How to use std::string as key in stxxl::map
I am trying to use std::string as a key in the stxxl::map
The insertion was fine for small number of strings about 10-100.
But while trying to insert large number of strings about 100000 in it, I am ...
0
votes
5
answers
1k
views
Handling large amounts of data in C++, need approach
So I have a 1GB file in a CSV format like so, that I converted to a SQLite3 database
column1;column2;column3
1212;abcd;20090909
1543;efgh;20120120
Except that I have 12 columns. Now, I need to read ...
2
votes
3
answers
339
views
Why cannot structs storing plain pointers to internal memory be stored in stxxl containers?
In the stxxl FAQ, I found this:
Parameterizing STXXL Containers
STXXL container types like stxxl::vector can be parameterized only
with a value type that is a POD (i.e. no virtual functions, ...
0
votes
0
answers
505
views
STXXL disk allocation error Windows7
I am trying to use stxxl in windows 7 - MSVC12 Express.
If I do not give the config file (.stxxl.txt): the program initiates by
[STXXL-MSG] STXXL v1.4.0 (prerelease/)
[STXXL-ERRMSG] Warning: no ...
2
votes
1
answer
266
views
STXXL: limited parallelism during sorting?
I populate a very large array using a stxxl::VECTOR_GENERATOR<MyData>::result::bufwriter_type (something like 100M entries) which I need to sort in parallel.
I use the stxxl::sort(vector->...
0
votes
2
answers
425
views
Working with stxxl in multiple threads
The following program, crashes with
libc++abi.dylib: terminating with uncaught exception of type stxxl::io_error: Error in virtual void stxxl::ufs_file_base::lock() : fcntl(,F_SETLK,) path=/var/tmp/...
0
votes
2
answers
269
views
Is it possible to define the length of the type to sort in STXXL at run time?
I have an application that requires a built-in sort and I'm hoping to replace the existing sort mechanism with the sort provided by STXXL. I have successfully tested it using STXXL, but my problem is ...
0
votes
1
answer
315
views
Cannot build STXXL with CMake
It seems that I cannot build the solution for STXXL with CMake 2.8.12.1. I looked at all the articles and followed their instructions but no luck. I have VS2010 Professional with SP1 and Windows 7. I ...
3
votes
2
answers
2k
views
High memory usage with STXXL
I'm working on a project using STXXL, which I understand to be an out of core version of the C++ STL library. Currently, my program is running fine with it, but the problem I'm facing now is that when ...
0
votes
1
answer
136
views
STXXL vector_type does not seem to work
I wanted to try STXXL to find how efficient it is in reading a big data file from the disk.
So i setup the enviornment for using it.
Then i ran this program http://algo2.iti.kit.edu/dementiev/stxxl/...
0
votes
2
answers
87
views
How to find no of I/Os of a program in stxxl?
I am using STXXL, can somebody help me in finding the no. of I/O's(or blocks transferred) done by my program(or algorithm or process)? I know how to restrict the memory usage by any particular process,...
0
votes
1
answer
126
views
Error when using STXXL Autogrow
I'm currently working on a project that requires about 20 vectors to be written to individual files. I also need my STXXL disk file to grow automatically to account for very large vectors. I ...
0
votes
1
answer
167
views
Boost noncopyable error when creating STXXL maps
I have been doing some work with STXXL, and I've kind of encountered a problem with the maps inheriting from boost::noncopyable... For this project, I create several maps with the statement:
stxxl::...
0
votes
1
answer
80
views
Error in a recursive template function using STXXL templates in C++
I need to optimize the block size BlkSize_ parameter of the stxxl vector for partial sums finding using a simple grid search. As the only way to specify it for a stxxl vector seems to use it as a ...
2
votes
0
answers
78
views
Does STXXL save the state after crash?
I'm writing a webapp in C++ and would like to use a fast database on SSD.
Seems like STXXL would be faster than other choices (slite,mongodb,...)
Do STXXL's containers keep the data after accidental ...
9
votes
1
answer
2k
views
Compiling STXXL under Android NDK r8b
I'm trying to compile STXXL under Android NDK r8b (I have the same problem under the newer r8c btw).
I'm compiling using gnustl_static.
I need C++11 support so initially I tried setting
...
3
votes
1
answer
364
views
Using STXXL in Qt project
How do I get qmake to include the stxxl.mk file into the generated makefiles?
I have a Qt project, which deals with large files (>RAM) and therefore want to employ STXXL. The STXXL documentation says:...
0
votes
1
answer
277
views
stxxl map<int, string>
I am looking for a storage to index points in 2 dimensions. To be more
specific, I would like to store the geometry of ways (or edges) in OpenStreetMap
and have it searchable. Queries to the storage ...
3
votes
1
answer
257
views
`pointer being freed was not allocated` when using stxxl queue
My code seems to work(I haven't tried it with large datasets because of the above error).
Code:
#include <iostream>
#include <queue>
#include <stxxl/queue>
int main()
{
//queue&...
0
votes
1
answer
150
views
stxxl Assertion `it != root_node_.end()' failed
I am receiving this assertion failed error when trying to insert an element in a stxxl map.
The entire assertion error is the following:
resCache: /usr/include/stxxl/bits/containers/btree/btree.h:...
-1
votes
1
answer
345
views
Using const char* in a STXXL map
I've looked for it but I've not been able to find a solution. Sorry it this has been already posted.
I have to create a stxxl::map structure mapping const char* into const char* (the optimum would be ...
0
votes
1
answer
240
views
Makefile with STXXL
I need to use the STXXL library for a software project I am working on, but for some reason, I am having trouble compiling a test file. I am not very familiar with makefiles, so I might have mixed up ...
0
votes
1
answer
177
views
stxxl and overriding cxx_constructor
I'm trying to use stxxl as a backing store to a vector. If I do nothing it will automatically allocate 1 gig of disk space and everything works perfectly. However I don't need that much space ...