Questions tagged [pickle]

Pickle is an object serialization module for Python. Use this tag together with the Python tag for questions related to storing or loading objects with Pickle.

pickle
Filter by
Sorted by
Tagged with
28 votes
3 answers
9k views

Why can't I pickle an error's Traceback in Python?

I've since found a work around, but still want to know the answer.
Trindaz's user avatar
  • 17.4k
28 votes
8 answers
22k views

How can I speed up unpickling large objects if I have plenty of RAM?

It's taking me up to an hour to read a 1-gigabyte NetworkX graph data structure using cPickle (its 1-GB when stored on disk as a binary pickle file). Note that the file quickly loads into memory. In ...
conradlee's user avatar
  • 13.3k
27 votes
2 answers
42k views

Find pickle version installed in my system

I would like to know pickle version installed in my system.Can someone please tell me what should be the procedure to do this.
Japneesh's user avatar
  • 287
27 votes
7 answers
12k views

How to hash a large object (dataset) in Python?

I would like to calculate a hash of a Python class containing a dataset for Machine Learning. The hash is meant to be used for caching, so I was thinking of md5 or sha1. The problem is that most of ...
user avatar
27 votes
2 answers
23k views

Load pickled object in different file - Attribute error

I have some trouble with loading a pickled file in a module that is different from the module where I pickled the file. I am aware of the following thread: Unable to load files using pickle and ...
lmartens's user avatar
  • 1,472
27 votes
2 answers
42k views

Save and reuse TfidfVectorizer in scikit learn

I am using TfidfVectorizer in scikit learn to create a matrix from text data. Now I need to save this object to reuse it later. I tried to use pickle, but it gave the following error. loc=open('...
Joswin K J's user avatar
27 votes
6 answers
9k views

Python-like pickling of full Javascript objects

Is there any serialization framework for Javascript which would retain class and reference information like Python pickles? I.e. one could directly take a prototypial inherited class instance (not ...
Mikko Ohtamaa's user avatar
26 votes
2 answers
50k views

Python pickle/unpickle a list to/from a file

I have a list that looks like this: a = [['a string', [0, 0, 0], [22, 'bee sting']], ['see string', [0, 2, 0], [22, 'd string']]] and am having problems saving it and retrieving it. I can save ...
Lee's user avatar
  • 30.3k
26 votes
3 answers
3k views

Preserve custom attributes when pickling subclass of numpy array

I've created a subclass of numpy ndarray following the numpy documentation. In particular, I have added a custom attribute by modifying the code provided. I'm manipulating instances of this class ...
Gabriel's user avatar
  • 1,900
26 votes
2 answers
101k views

python 3.6 socket pickle data was truncated

I can not send my numpy array in socket. I use pickle but my client pickle crashes with this error: pickle data was truncated My server : I create a numpy array and I want to send in my client with ...
sazearte's user avatar
  • 369
26 votes
2 answers
21k views

Pickle versus shelve storing large dictionaries in Python

If I am storing a large directory as a pickle file, does loading it via cPickle mean that it will all be consumed into memory at once? If so, is there a cross platform way to get something like ...
user avatar
25 votes
2 answers
28k views

Pickle - Load variable if exists or create and save it

Is there a better way to load a variable with pickle if it already exists or create it and dump it if it doesn't? if os.path.isfile("var.pickle"): foo = pickle.load( open( "var.pickle", "rb" ) ) ...
Matteo's user avatar
  • 8,034
25 votes
2 answers
13k views

What is the difference between __reduce__ and __reduce_ex__?

I understand that these methods are for pickling/unpickling and have no relation to the reduce built-in function, but what's the difference between the 2 and why do we need both?
Moe's user avatar
  • 29.2k
25 votes
4 answers
10k views

dreaded "not the same object error" pickling a queryset.query object

I have a queryset that I need to pickle lazily and I am having some serious troubles. cPickle.dumps(queryset.query) throws the following error: Can't pickle <class 'myproject.myapp.models.myfile....
Bacon's user avatar
  • 2,165
24 votes
2 answers
25k views

Python: saving objects and using pickle. extension of filename

Hello I´m trying using the next piece of code: import pickle object = Object() filehandler = open(filename, 'w') pickle.dump(object, filehandler) I would like to know what should be the ...
Peterstone's user avatar
  • 7,259
24 votes
8 answers
56k views

FileNotFoundError while writing to a Pickle file

I'm taking all the filenames in a certain directory in a list, and would like to write this list in a pickle file. Here's the code I am using: import _pickle as pickle with open(filepath+'filenames....
Patthebug's user avatar
  • 4,757
23 votes
3 answers
11k views

How to stop attributes from being pickled in Python [duplicate]

I am using gnosis.xml.pickle to convert an object of my own class to xml. The object is initialized so that: self.logger = MyLogger() But when I do dump the object to a string I get an exception ...
Ries's user avatar
  • 2,866
23 votes
4 answers
55k views

How to pickle Keras model?

Official documents state that "It is not recommended to use pickle or cPickle to save a Keras model." However, my need for pickling Keras model stems from hyperparameter optimization using sklearn's ...
Sida Zhou's user avatar
  • 3,597
23 votes
3 answers
31k views

Convert Pandas DataFrame to bytes-like object

Hi I am trying to convert my df to binary and store it in a variable. my_df: df = pd.DataFrame({'A':[1,2,3],'B':[4,5,6]}) my code: import io towrite = io.BytesIO() df.to_excel(towrite) # write ...
Pyd's user avatar
  • 6,117
23 votes
3 answers
11k views

How to test if a file has been created by pickle?

Is there any way of checking if a file has been created by pickle? I could just catch exceptions thrown by pickle.load but there is no specific "not a pickle file" exception.
Nova's user avatar
  • 2,610
23 votes
3 answers
11k views

Pickle: dealing with updated class definitions

After a class definition is updated by recompiling a script, pickle refuses to serialize previously instantiated objects of that class, giving the error: "Can't pickle object: it's not the same object ...
iestyn's user avatar
  • 331
23 votes
1 answer
11k views

What is a SwigPyObject objects?

I get an error when I try to save a custom object with with open(path + '/' + 'my_object.pickle', 'wb') as handle: pickle.dump(my_class_instance, handle, protocol=pickle.HIGHEST_PROTOCOL)...
paolof89's user avatar
  • 1,349
23 votes
2 answers
74k views

TypeError: can't pickle generator objects

I am trying to write some result on to pickle file as below: raw_X = (self.token_ques(text) for text in training_data) with open('/root/Desktop/classifier_result.pkl', 'wb') as handle: pickle....
nlper's user avatar
  • 2,337
22 votes
1 answer
15k views

How is dill different from Python's pickle module?

I have a large object in my Python3 code which, when tried to be pickled with the pickle module throws the following error: TypeError: cannot serialize '_io.BufferedReader' object However, dill.dump(...
sherlock's user avatar
  • 2,577
22 votes
6 answers
29k views

Is pickle file of python cross-platform?

I have created a small python script of mine. I saved the pickle file on Linux and then used it on windows and then again used it back on Linux but now that file is not working on Linux but it is ...
vikithakar's user avatar
22 votes
1 answer
15k views

dill vs cPickle speed difference

I am trying to serialize thousands of objects and some of these objects are lambda objects. Since cPickle doesn't work for lambdas, I tried using dill. However, the drop in computational speed is ...
Tohiko's user avatar
  • 1,922
22 votes
9 answers
24k views

ImportError: No module named copy_reg pickle

I'm trying to unpickle an object stored as a blob in a MySQL database. I've manually generated and stored the pickled object in the database, but when I try to unpickle the object, I get the following ...
Stephen Edmonds's user avatar
22 votes
3 answers
12k views

Python, cPickle, pickling lambda functions

I have to pickle an array of objects like this: import cPickle as pickle from numpy import sin, cos, array tmp = lambda x: sin(x)+cos(x) test = array([[tmp,tmp],[tmp,tmp]],dtype=object) pickle.dump( ...
Saullo G. P. Castro's user avatar
22 votes
1 answer
8k views

Pickles: Why are they called that? [closed]

I'm surprised this is such a hard answer for me to find. It's such a strange name. Why are pickles called pickles? http://en.wikipedia.org/wiki/Pickle_(Python) I understand what "pickling" means ...
Brad Johnson's user avatar
  • 1,868
22 votes
2 answers
51k views

_pickle.UnpicklingError: could not find MARK

I got exceptions like UnicodeDecodeError raised when pickling (a list of) objects of EventFrame with a member participants that was an empty set. class EventFrame: """Frame for an event""" ...
Sadık's user avatar
  • 4,329
22 votes
1 answer
5k views

How to check which detail of a complex object cannot be pickled

Overview I want to serialize my complex objects. It looks simple but every step creates a different problem. In the end, other programmers must also be able to create a complex object inherited ...
Bernard's user avatar
  • 703
21 votes
5 answers
20k views

module 'pickle' has no attribute 'dump'

import pickle imelda = ('More Mayhem', 'IMelda May', '2011', ((1, 'Pulling the Rug'), (2, 'Psycho'), (3, 'Mayhem'), (4, 'Kentish Town ...
Champa's user avatar
  • 595
21 votes
7 answers
4k views

Is there a way to really pickle compiled regular expressions in python?

I have a python console application that contains 300+ regular expressions. The set of regular expressions is fixed for each release. When users run the app, the entire set of regular expressions ...
Adam's user avatar
  • 211
20 votes
3 answers
35k views

How to close the file after pickle.load() in python

I saved a python dictionary in this way: import cPickle as pickle pickle.dump(dictname, open("filename.pkl", "wb")) And I load it in another script in this way: dictname = pickle.load(open("...
user2961420's user avatar
20 votes
7 answers
17k views

Serializing a suds object in python

Ok I'm working on getting better with python, so I'm not sure this is the right way to go about what I'm doing to begin with, but here's my current problem... I need to get some information via a ...
jeffff's user avatar
  • 1,319
20 votes
3 answers
44k views

Pickle File too large to load

The problem that I am having is that I have a very large pickle file (2.6 Gb) that I am trying to open but each time I do so I get a memory error. I realize now that I should have used a database to ...
Vineeth Bhuvanagiri's user avatar
20 votes
1 answer
16k views

Why pickle eat memory?

I trying to deal with writing huge amount of pickled data to disk by small pieces. Here is the example code: from cPickle import * from gc import collect PATH = r'd:\test.dat' @profile def func(item)...
Gill Bates's user avatar
  • 14.8k
19 votes
2 answers
37k views

Load pickle file(comes from python3) in python2

I have a pickle file, with >>> with open("wikilinks.pickle", "rb") as f: ... titles, links = pickle.load(f) ... >>> len(titles) 13421 I can load it in python3. However, when ...
epx's user avatar
  • 591
19 votes
2 answers
22k views

AttributeError when reading a pickle file

I get the following error when I'm reading my .pkl files on spyder (python 3.6.5): IN: with open(file, "rb") as f: data = pickle.load(f) Traceback (most recent call last): File "<...
Mathieu's user avatar
  • 5,536
19 votes
1 answer
9k views

Why does the shelve module in python sometimes create files with different extensions?

I'm running a Python program which uses the shelve module on top of pickle. After running this program sometimes I get one output file as a.data but at other times I get three output files as a.data....
Ali_IT's user avatar
  • 7,761
19 votes
4 answers
53k views

How to 'pickle' an object to a certain directory?

Normally, executing the following code will pickle an object to a file in my current directory: fp = open('somefile.txt', 'wb') pickle.dump(object, fp) How do I re-direct the output from pickle.dump ...
Houdini's user avatar
  • 3,502
19 votes
2 answers
16k views

AttributeError when unpickling an object

I'm trying to pickle an instance of a class in one module, and unpickle it in another. Here's where I pickle: import cPickle def pickleObject(): object = Foo() savefile = open('path/to/file'...
Max's user avatar
  • 1,360
19 votes
2 answers
27k views

What's the fastest way to pickle a pandas DataFrame?

Which is better, using Pandas built-in method or pickle.dump? The standard pickle method looks like this: pickle.dump(my_dataframe, open('test_pickle.p', 'wb')) The Pandas built-in method looks ...
tegan's user avatar
  • 2,215
18 votes
2 answers
16k views

python pickle UnicodeDecodeError

I'm trying to load the mnist character dataset (following the tutorial outlined here: http://neuralnetworksanddeeplearning.com/chap1.html ) when I run the load_data_wrapper function I get the error. ...
Nisitha Jayatilleka's user avatar
18 votes
3 answers
42k views

How to cPickle dump and load separate dictionaries to the same file?

I have a process which runs and creates three dictionaries: 2 rather small, and 1 large. I know I can store one dictionary like: import cPickle as pickle with open(filename, 'wb') as fp: pickle....
JBWhitmore's user avatar
  • 11.9k
18 votes
4 answers
27k views

Downloads folder from JupyterLab

I'd like to download a folder of pickle files from Jupyterlab (running on a google cloud instance) and I'm unable to find a way to do so. Apparently, downloading individual files is permitted but I ...
Alex Martin's user avatar
18 votes
4 answers
18k views

When can a Python object be pickled

I'm doing a fair amount of parallel processing in Python using the multiprocessing module. I know certain objects CAN be pickle (thus passed as arguments in multi-p) and others can't. E.g. class ...
Paul Nelson's user avatar
  • 1,321
18 votes
2 answers
52k views

UnpicklingError invalid load key, '{'

I am working on an application which is in Django. I am trying to store some data structure e.g. dictionary in MySQL db. So i use Python Pickle module. It works fine when i store it in db using pickle....
Aamir Rind's user avatar
  • 39.3k
18 votes
4 answers
337k views

Python - AttributeError: 'numpy.ndarray' object has no attribute 'append'

This is related to my question, here. I now have the updated code as follows: import numpy as np import _pickle as cPickle from PIL import Image import sys,os pixels = [] labels = [] traindata = [] ...
Simplicity's user avatar
  • 48.3k
18 votes
2 answers
54k views

Pickling weakref in Python

I am still pretty new to Python and even newer to pickling. I have a class Vertex(ScatterLayout) with a __getnewargs__(): def __getnewargs__(self): return (self.pos, self.size, self.idea.text) ...
David Poxon's user avatar
  • 2,435