5

Everyone is concerned with protecting user privacy and their data these days. Being capable of performing statistical analysis on data without actually keeping data samples around longer than necessary is one firm step in the right direction.

The concept of accumulators in the boost library looks like ideal fit to make it happen.

The Accumulators Framework is framework for performing incremental calculations. Usage of the framework follows the following pattern:

Users push data into the accumulator_set<> object one sample at a time.

The accumulator_set<> computes the requested quantities in the most efficient method possible, resolving dependencies between requested calculations, possibly caching intermediate results.

There is just one big catch. I have no idea how to serialize the accumulator instance so that I can persist it without needing to keep the sample data around between app launches.

How to create an instance and restore all necessary parameters needed for it to apply new samples and continue its incremental computations? I do not want to start from scratch nor do I want to keep the previous samples around.

1 Answer 1

2
+200

I think you're misunderstanding how accumulator_set works internally.

It doesn't keep the samples. In fact, if you just parameterized the set for the 'min' and 'max' stats, then the only state that is going to be kept is 2 values of the result type.

Some other stats keep more state (e.g. histograms, percentiles etc.).

Inasmuch as accumulator_set<> supports serialization in the first place¹, it should be fine with respect to sample retention.

¹ I haven't checked


UPDATE

Okay. I've looked at things for about an hour now and I see no reason to think that serialization is supported or easy to implement.

The most I've seen is that features are copyable.

But there's no allocator that you can tweak from the outside so you can't leverage say memory mapped files either.

That leads me to conclude that what you want is not a feature of the library. So, where the documentation intro:

Boost.Accumulators is both a library for incremental statistical computation as well as an extensible framework for incremental calculation in general.

Then apparently we should take that to mean "incremental operations during the lifetime of an accumulator_set", not "incremental" as in resumable/persistable.

It does seem like a nice feature request that the library authors would like to help with?

6
  • Yes, it's good that it does not keep samples around. That is why i'd like to use it - it can perform incremental calculations based on previous samples having cached necessary intermediary results. But how to restore those intermediary results when constructing a new instance of the accumulator?
    – svena
    Aug 30, 2015 at 6:45
  • 1
    I don't think he wants to keep the samples at all, just the internal state necessary to "keep going". In other words, what he's actually asking is about the details to your footnote. Aug 31, 2015 at 21:06
  • @Nir Added some thoughts; we could plink Eric Niebler for an expert opinion :)
    – sehe
    Aug 31, 2015 at 22:01
  • @EricNiebler kind request for comment here (to make sure I'm not talking nonsense about "your" library). Cheers
    – sehe
    Aug 31, 2015 at 22:08
  • 1
    @Sehe I really appreciate you taking time for this. Judging from your update and what Nir said, we are totally on the same page here on what the issue's about.
    – svena
    Sep 1, 2015 at 13:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.