Questions tagged [stxxl]

STXXL is an open source out-of-core implementation of the STL, that is available for all major desktop operating systems under the Boost Software License 1.0.

Filter by
Sorted by
Tagged with
2 votes
0 answers
176 views

How to use a C++ 14 library in a C++ 23 project (MSVC)?

I'm writing a program that makes heavy use of C++ 20 and newer features. The problem is that in the project I need the stxxl library, which is compiled exclusively on C++ 14 . If I try to include it ...
Chase's user avatar
  • 43
0 votes
0 answers
103 views

stxxl exceeding OS limit of nr of threads

STXXL 1.4.1 on Windows 10 / VS2015. I have some code in which I essentially use this: typedef stxxl::vector<RC, 4, stxxl::lru_pager<8>, 2 * 1024> TVector; typedef stxxl::deque<RC, ...
Roel's user avatar
  • 19.4k
2 votes
0 answers
177 views

STXXL underperforms when using very fast SSDs

I'm currently writing a tool which uses the STXXL to find similarities between a large file on the hard drive and vectors residing in RAM. I wrote an example(for Windows) below to show what I mean. ...
Kasastran's user avatar
0 votes
1 answer
202 views

Including STXXL library in the MakeFile

I was trying to install STXXL library to a custom path following this answer supplying prefix to cmake this way: cmake -DCMAKE_INSTALL_PREFIX=/usr . && make all install When I run the tests,...
Unni's user avatar
  • 5,486
1 vote
1 answer
711 views

OSRM extract and best swap stxxl location on 2 HDs

I want to set up the car profile data, but it is now taking 3 days and still at 100% Graphs (so way more to go) and I am thinking, maybe my swap and stxxl is just set up inappropriately. I have the ...
TheRealPir's user avatar
0 votes
1 answer
140 views

STXXL fails at creating a map of strings in long numeric

With the code that follows, I want to create a map of type std::map. By using a debugger, I got that the exception was raised by stl_pair.h at line 368 (pair& operator=(typename conditional<...
jackb's user avatar
  • 694
0 votes
1 answer
457 views

Any STXXL alternatives to STXXL to handle unknown size at compile time, or any tricks?

This question and answers to it reassure me that STXXL handles only fixed data length. Are there any alternatives to STXXL to handle dynamic data size (data size unknown at compile time)? Or are ...
sofname's user avatar
  • 439
1 vote
1 answer
318 views

Operating with huge array with random access to its elements

I have to work with very large array (about 2000^3) of float, moreover, sometimes I have to route through array elements randomly, or by another non-sequential way. I use stxxl library and store my ...
user2760303's user avatar
0 votes
1 answer
313 views

Stxxl Vector as dropin replacement for std::vector

This is a followup of Vector of pairs with generic vector and pair type, template of template. I'd like to be able to call a method with a std::vector or stxxl:vector, while template arguments of ...
Jan B's user avatar
  • 518
1 vote
0 answers
187 views

Compiling STXXL on visual studio 2015

I am trying to use STXXL map in my c++ code in visual studio 2015. I followed the instruction on http://stxxl.sourceforge.net/tags/master/install_windows.html' and when I try to build the project in ...
Rachel's user avatar
  • 11
1 vote
1 answer
117 views

C++: STXXL and VS runtime error in simple code

I have the following code which is a pretty simple test, but the VS refuses to run it: stxxl::syscall_file OutputFile("Data/test.bin", stxxl::file::RDWR | stxxl::file::CREAT | stxxl::file::DIRECT); ...
TheShadow's user avatar
1 vote
0 answers
249 views

Specify location to store stxxl::map/persist its data?

stxxl::vector has a constructor where you can provide a stxxl::file *. This ties the vector to that specific file: typedef stxxl::VECTOR_GENERATOR<int>::result stxxl_vector_int; { stxxl::...
Claudiu's user avatar
  • 225k
0 votes
1 answer
95 views

Combining Streams and Iteration in STXXL

STXXL supports a streaming model of data processing. STXXL data can either be processed by iterating through it (e.g. using stxxl::for_each), or by converting to a stream mode (e.g. using streamify() ...
Adi Shavit's user avatar
  • 16.9k
1 vote
1 answer
290 views

Parallel STXXL Vector Initialization

The following minimal example illustrates the behaviour of stxxl when initializing containers in parallel (using openMP): #include <omp.h> #include <stdio.h> #include <stxxl.h> ...
Lars Hadidi's user avatar
0 votes
0 answers
457 views

Stxxl map with std::string as key and std::vector as value

I need a large map of sets of numbers. So i'm using stxxl and it's basically something like map<std::string, std::vector<int>>. The key is 3-letters identifier. There are about 3k enteries ...
flyman's user avatar
  • 210
0 votes
1 answer
163 views

Mapping an stxxl::vector to a File with Header

I have a large binary (STL) file that begins with a known and fixed size header and then continues with a bunch of fixed size buffers (PODs) representing consecutive records. I want to map the ...
Adi Shavit's user avatar
  • 16.9k
0 votes
1 answer
145 views

Changing the size of stxxl:vector cache

In my algorithm I have two stages: 1) Populating a file with records using a big data structure X that takes as much RAM as possible 2) Sorting the file. Now there is lot of memory since X is ...
Ilia Minkin's user avatar
1 vote
1 answer
244 views

STXXL: How to sort Vector of pairs on second element?

Similar Question is available here: How do I sort a vector of pairs based on the second element of the pair? but I am interested in External Memory Sorting. I have tried using the analogies from ...
Akif Khan's user avatar
2 votes
2 answers
222 views

C++: Which Data Type in STXXL is suitable to create External Memory Binary Search Tree?

I want to create an External Memory Binary Search Tree Data Structure whose data sits in the external memory using stxxl as a library. For this purpose, which Data Type in STXXL is suitable to use ...
Akif Khan's user avatar
0 votes
1 answer
90 views

Why do creating pointer to instances beyond certain number(30) of a 'Structure' with 'stxxl:Vector' as one of its DataType fails?

I am using STXXL Library's stxxl::vector in my code as : struct B { typedef stxxl::VECTOR_GENERATOR<float>::result vector; vector content; }; And then creating many instances of the ...
Akif Khan's user avatar
0 votes
1 answer
294 views

STXXL Dealing with strings

i am dealing with large dataset. May i ask you how it is possible to store strings in the classes i want to use with stxxl? I have read several discussions and everywhere was said that string is not ...
brano199's user avatar
0 votes
1 answer
512 views

stxxl save and read vector from disk file

I'm struggling in trying to use the stxxl library in a way, that I cannot only store the data from their vector structure into a file but also recover it from that file in a rerun of my program. I ...
Kasastran's user avatar
0 votes
1 answer
225 views

Parallelizing STXXL Container

I am trying to use STXXL to build a large matrix and be able to perform several different operations on the matrix. Currently, I'm using stxxl::vector as a way of storing submatrices of my large ...
mculp's user avatar
  • 59
3 votes
1 answer
657 views

stxxl sorting of very large file (ubuntu)

I am trying to sort a large file with about billion records (each containing four integers). The size of the file would shoot up beyond 50GB. I am testing my code with 400 million records (about 6 GB ...
Chirag Jain's user avatar
0 votes
1 answer
117 views

How to delete a STXXL priority queue?

I created a STXXL priority queue in my program. If I do not need the PQ any more, how to delete it? Just like delete[] buffer in C++. I want to free the memory used by it. I looked through their ...
Charlie's user avatar
2 votes
3 answers
330 views

How to append a large number of elements to stxxl vector efficiently?

I need to append a large number of elements to a stxxl vector. What is the most efficient way of adding elements to a stxxl vector? Right now, I'm using push_back of the stxxl vector, but it doesn't ...
Da Zheng's user avatar
  • 121
2 votes
2 answers
190 views

STXXL multidimensional vector makes program freeze

Trying to deal with huge data sets, I have been using the convenient STXXL library.Though I have run into a slight problem when trying to use/generate a multidimensional vector. The following ...
Morgan Bengtsson's user avatar
1 vote
3 answers
842 views

How to use std::string as key in stxxl::map

I am trying to use std::string as a key in the stxxl::map The insertion was fine for small number of strings about 10-100. But while trying to insert large number of strings about 100000 in it, I am ...
Sriram Mahavadi's user avatar
0 votes
5 answers
1k views

Handling large amounts of data in C++, need approach

So I have a 1GB file in a CSV format like so, that I converted to a SQLite3 database column1;column2;column3 1212;abcd;20090909 1543;efgh;20120120 Except that I have 12 columns. Now, I need to read ...
user3195734's user avatar
2 votes
3 answers
339 views

Why cannot structs storing plain pointers to internal memory be stored in stxxl containers?

In the stxxl FAQ, I found this: Parameterizing STXXL Containers STXXL container types like stxxl::vector can be parameterized only with a value type that is a POD (i.e. no virtual functions, ...
gd1's user avatar
  • 11.3k
0 votes
0 answers
505 views

STXXL disk allocation error Windows7

I am trying to use stxxl in windows 7 - MSVC12 Express. If I do not give the config file (.stxxl.txt): the program initiates by [STXXL-MSG] STXXL v1.4.0 (prerelease/) [STXXL-ERRMSG] Warning: no ...
user3319630's user avatar
2 votes
1 answer
266 views

STXXL: limited parallelism during sorting?

I populate a very large array using a stxxl::VECTOR_GENERATOR<MyData>::result::bufwriter_type (something like 100M entries) which I need to sort in parallel. I use the stxxl::sort(vector->...
jrbn's user avatar
  • 303
0 votes
2 answers
425 views

Working with stxxl in multiple threads

The following program, crashes with libc++abi.dylib: terminating with uncaught exception of type stxxl::io_error: Error in virtual void stxxl::ufs_file_base::lock() : fcntl(,F_SETLK,) path=/var/tmp/...
Martin Kristiansen's user avatar
0 votes
2 answers
269 views

Is it possible to define the length of the type to sort in STXXL at run time?

I have an application that requires a built-in sort and I'm hoping to replace the existing sort mechanism with the sort provided by STXXL. I have successfully tested it using STXXL, but my problem is ...
MarkI's user avatar
  • 3
0 votes
1 answer
315 views

Cannot build STXXL with CMake

It seems that I cannot build the solution for STXXL with CMake 2.8.12.1. I looked at all the articles and followed their instructions but no luck. I have VS2010 Professional with SP1 and Windows 7. I ...
Dimitris's user avatar
  • 2,040
3 votes
2 answers
2k views

High memory usage with STXXL

I'm working on a project using STXXL, which I understand to be an out of core version of the C++ STL library. Currently, my program is running fine with it, but the problem I'm facing now is that when ...
Andrewziac's user avatar
0 votes
1 answer
136 views

STXXL vector_type does not seem to work

I wanted to try STXXL to find how efficient it is in reading a big data file from the disk. So i setup the enviornment for using it. Then i ran this program http://algo2.iti.kit.edu/dementiev/stxxl/...
user2911470's user avatar
0 votes
2 answers
87 views

How to find no of I/Os of a program in stxxl?

I am using STXXL, can somebody help me in finding the no. of I/O's(or blocks transferred) done by my program(or algorithm or process)? I know how to restrict the memory usage by any particular process,...
user2890493's user avatar
0 votes
1 answer
126 views

Error when using STXXL Autogrow

I'm currently working on a project that requires about 20 vectors to be written to individual files. I also need my STXXL disk file to grow automatically to account for very large vectors. I ...
Andrewziac's user avatar
0 votes
1 answer
167 views

Boost noncopyable error when creating STXXL maps

I have been doing some work with STXXL, and I've kind of encountered a problem with the maps inheriting from boost::noncopyable... For this project, I create several maps with the statement: stxxl::...
Andrewziac's user avatar
0 votes
1 answer
80 views

Error in a recursive template function using STXXL templates in C++

I need to optimize the block size BlkSize_ parameter of the stxxl vector for partial sums finding using a simple grid search. As the only way to specify it for a stxxl vector seems to use it as a ...
FrauHahnhen's user avatar
2 votes
0 answers
78 views

Does STXXL save the state after crash?

I'm writing a webapp in C++ and would like to use a fast database on SSD. Seems like STXXL would be faster than other choices (slite,mongodb,...) Do STXXL's containers keep the data after accidental ...
kirill_igum's user avatar
  • 3,953
9 votes
1 answer
2k views

Compiling STXXL under Android NDK r8b

I'm trying to compile STXXL under Android NDK r8b (I have the same problem under the newer r8c btw). I'm compiling using gnustl_static. I need C++11 support so initially I tried setting ...
Goz's user avatar
  • 61.5k
3 votes
1 answer
364 views

Using STXXL in Qt project

How do I get qmake to include the stxxl.mk file into the generated makefiles? I have a Qt project, which deals with large files (>RAM) and therefore want to employ STXXL. The STXXL documentation says:...
Nobody moving away from SE's user avatar
0 votes
1 answer
277 views

stxxl map<int, string>

I am looking for a storage to index points in 2 dimensions. To be more specific, I would like to store the geometry of ways (or edges) in OpenStreetMap and have it searchable. Queries to the storage ...
wipkirill's user avatar
3 votes
1 answer
257 views

`pointer being freed was not allocated` when using stxxl queue

My code seems to work(I haven't tried it with large datasets because of the above error). Code: #include <iostream> #include <queue> #include <stxxl/queue> int main() { //queue&...
Lostsoul's user avatar
  • 25.2k
0 votes
1 answer
150 views

stxxl Assertion `it != root_node_.end()' failed

I am receiving this assertion failed error when trying to insert an element in a stxxl map. The entire assertion error is the following: resCache: /usr/include/stxxl/bits/containers/btree/btree.h:...
Fabrizio Silvestri's user avatar
-1 votes
1 answer
345 views

Using const char* in a STXXL map

I've looked for it but I've not been able to find a solution. Sorry it this has been already posted. I have to create a stxxl::map structure mapping const char* into const char* (the optimum would be ...
Fabrizio Silvestri's user avatar
0 votes
1 answer
240 views

Makefile with STXXL

I need to use the STXXL library for a software project I am working on, but for some reason, I am having trouble compiling a test file. I am not very familiar with makefiles, so I might have mixed up ...
alguru's user avatar
  • 404
0 votes
1 answer
177 views

stxxl and overriding cxx_constructor

I'm trying to use stxxl as a backing store to a vector. If I do nothing it will automatically allocate 1 gig of disk space and everything works perfectly. However I don't need that much space ...
Goz's user avatar
  • 61.5k