0

Calculate fft with 16GB memory,cause memory exhausted.

print(data_size)
freqs, times, spec_arr = signal.spectrogram(data, fs=samp_rate,nfft=1024,return_onesided=False,axis=0,scaling='spectrum',mode='magnitude')

Output as below:

537089518
Killed

How to calculate fft of large size data ,with existing python package?

1
  • signal.spectrogram doesn’t compute the FFT. It uses the FFT internally, but the result is not the same as the FFT. Mar 13, 2022 at 16:26

1 Answer 1

1

A more general solution is to do that yourself. 1D FFTs can be split in smaller ones thanks to the well-known Cooley–Tukey FFT algorithm and multidimentional decomposition. For more information about this strategy, please read The Design and Implementation of FFTW3. You can do the operation in virtually mapped memory so to do that more easily. Some library/package like the FFTW enable you to relatively-easily perform fast in-place FFTs. You may need to write your own Python package or to use Cython so not to allocate additional memory that is not memory mapped.

One alternative solution is to save your data in HDF5 (for example using h5py, and then use out_of_core_fft and then read again the file. But, be aware that this package is a bit old and appear not to be maintained anymore.

4
  • Big thanks.what I tried with torch:t=torch.from_numpy(data), f=torch.fft.fft(t).I am not sure whether that's a correct way,because the result has big difference.
    – kittygirl
    Mar 13, 2022 at 13:43
  • I can't tell for torch as I never used it. Note that some library renormalize results (AFAIK using a simple multiplication by a factor) while other do not. This could explain the difference. It appears torch does not normalize the results by default (while people often need it). Mar 13, 2022 at 13:49
  • If I don't care about time,how about enlarge swap to make fft consume?
    – kittygirl
    Mar 13, 2022 at 14:13
  • This can be a simple solution indeed, but this require root privilege. It also reserve a fixed amount of space on the disk that should be cleverly chosen. In addition, this is OS dependent. On Windows the swap-file already use a variable amount of space. AFAIK, on Linux, you should avoid allocating much more than the amount of RAM, otherwise the whole system can become very very slow (due to trashing). See: serverfault.com/questions/17718 . Note this only work on your target machine (you can hardly ask people to do this to run your program). Mar 13, 2022 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.