Questions tagged [pickle]
Pickle is an object serialization module for Python. Use this tag together with the Python tag for questions related to storing or loading objects with Pickle.
4,559
questions
0
votes
0
answers
5
views
How to use external application instance in a pymoo problem without needing to serialise it to enable saving history?
I'm trying to include an external application instance in the evaluation of a pymoo problem. I've got this working, but having the external application in the pymoo problem means that I can't use the ...
0
votes
0
answers
10
views
How do I use an existing pickle file I have made in Thonny?
I am trying to use a pickle file that I receive from a friend and use it but every time I do load it I always get the the error "ValueError: Buffer dtype mismatch, expected 'const ITYPE_t' but ...
0
votes
1
answer
33
views
Get only the txt file you want from the folder containing the txt file - Python
I have a folder with a .txt files. the name of the files are:
my_file1.txt
my_file2.txt
my_file3.txt
my_file4.txt
In this way, only the last number is different.
import pickle
my_list = []
with open(...
0
votes
0
answers
5
views
A logger embedded in a class loses settings when reading from a pickle
I store my objects with an attribute being a logger in pickle and read them for other programs. I find that the logger's level is lost in the following situation.
In the following example, Cls has an ...
0
votes
0
answers
23
views
deepcopy and pickle in python 3.6 [closed]
I am running into a performance issue with a tool, and for security reasons, I cannot share the code. I see a couple of things which I am confused about and think could probably be the pain point.I ...
1
vote
1
answer
28
views
Pickle changes type in jax
I have a flax struct dataclass containing a jax numpy array.
When I pickle dump this object and load it again, the array is not anymore a jax numpy array and is converted to a numpy array, here is the ...
0
votes
1
answer
29
views
Saved Machine Learning Model using Pickle won't predict text values properly
I currently have a Machine Learning model which would predict what part of speech does a current word belong to
penn_results = penn_crf.predict_single(features)
and then, I made a code wherein it ...
0
votes
0
answers
28
views
Python - Pickle non-picklable object
I have an object which I want to pickle it. but inside it there is another object which is not picklable.
There is a function inside my none picklable object which serialize this object for me.
So now ...
0
votes
0
answers
20
views
UnpicklingError on data received from socket
Context:
I have a system with a server.py file and a live_client.py. Server is constantly updating an object. The system works correctly if I send the attributes of the object (which are numbers) via ...
0
votes
1
answer
12
views
I am trying to append two dataframes but its giving an empty dataframes
Here is the code:
with open('Epoch_SN001.pickle','rb') as f:
p=pickle.load(f)
p_des=p.describe()
print(type(p_des))
print(p_des)
epoch_count=set(p.epoch)
p_deddf=pd.DataFrame()
...
0
votes
0
answers
25
views
SciPy problem: "No module named 'scipy.sparse._coo'"
I've run a python script that generates sparse coo matrices and saves them into a pickle object. The python version that I used for this script was 3.8 with SciPy 1.8.0. In another environment, I'm ...
-1
votes
0
answers
15
views
Unable to pass webdriver in multiprocessing since it cannot be pickled
I am trying to pass the webdriver into the mutliprocessing function, but there is error. After searching for solution in the web, I knew that the webdriver cannot be pickled. Is there any way to ...
0
votes
1
answer
33
views
Trying to pickle a custom class where an attribute of that class is a list of another custom class
Here is an example code of what I am trying to do. My end goal is creating a JSON object for class real estate agent, but I am struggling because I don't know what to do about self.housesSold.
class ...
1
vote
1
answer
34
views
unable to pickle keras.layers.StringLookup
minimal code to reproduce:
import tensorflow
import pickle
print(tensorflow.__version__)
pickle.dumps(tensorflow.keras.layers.StringLookup())
output:
2.8.0
--------------------------------------------...
0
votes
0
answers
11
views
What's a proper way to pickle requests_html.HTMLResponse object?
I made a code to scrape a few urls to work later on response objects.
# %%
from requests_html import HTMLSession
import pickle
with open('urls.txt', encoding='utf-8-sig') as f:
urls = f.read()....
0
votes
0
answers
36
views
How to find min of each column attribute in pyspark sql rows type data in python given as dictionary to a column of another dataframe
I've the data extracted from a pickle file and it has column which have data of type dictionary. The data is of the following format:
Orders
ID
Data
1006
{'0': Rows(amount=100, quantity=10), '1': ...
0
votes
1
answer
12
views
cannot load pickle files for xgboost images of version > 1.2-2 in sagemaker - UnpicklingError
I can train a XGBoost model using Sagemaker images like so:
import boto3
import sagemaker
from sagemaker.inputs import TrainingInput
import os
folder = r"C:\Somewhere"
os.chdir(folder)
...
0
votes
0
answers
8
views
issue with concatenating multiple feature files in a folder that have different sizes
I have 1000 pkl files containing my audio features. I need to combine all these pkl files into one main audio feature file.
As my first try, I try to concatenate 5 of them first. Their sizes are ...
0
votes
0
answers
16
views
How to preserve args for unpickling object without storing them
I have a class with the following
def __init__(self, foo, **kwargs):
self._foo = foo
self._bar = []
# transform foo to become bar
I want to pickle and then unpickle an object of this ...
0
votes
0
answers
27
views
Best practices for using SQLAlchemy with multiprocessing
I have a class which creates a SQLAlchemy engine object. (I've left out some of the config args and replaced with <> tags).
class DBClient:
def __init__(self, isolation_level='AUTOCOMMIT'):
...
-1
votes
0
answers
30
views
Trying to create a multiplayer game using sockets python
I'm trying to send a pygame sprite object over a server using sockets but I get the below error. The main.py file will get the player information form the server in the players list then send back any ...
1
vote
0
answers
45
views
Alternative to pickle for javascript
I'm working on a deep learning project and I got stucked: I have a jupiter notebook that processes a dataset of images and returns a list of tensors in python, which I pickle with pickle.dump(). ...
0
votes
0
answers
17
views
Why can not save/pickle OAuth2Session from requests_oauthlib?
I am making python wrapper for some web api service.
I have faced a problem, which I do not how to solve and quite can not understand.
I need to save authorized session object. But I get error:
import ...
0
votes
0
answers
25
views
EOFError when reading pickle object (with Pandas) that's non-empty
I have a .pkl file that stores a a large pandas DataFrame on my computer; every morning a script runs that loads the file, appends new data, and re-saves it.
Today, I stopped the script shortly after ...
0
votes
0
answers
17
views
_pickle.PicklingError, in training networks with num_workers positive
I was training a big dataset, using pytorch Dataset and DataLoader, and this problem appeared. I deleted much of my code to ask this question, and the problem exists, these code can run independently:
...
0
votes
0
answers
33
views
Unable to solve multiprocessing.Manager.Lock() error in Python code (VS editor)
I am using machine learning in my Python (version 3.8.5) code. In the preprocessing part, I need to hash encode few features. So earlier I have dumped a hash encoder pickle file using the features in ...
0
votes
1
answer
36
views
for key, value in other: ValueError: too many values to unpack (expected 2)(pickle)
I use this code to get cookies and save them to file
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import ...
1
vote
0
answers
28
views
how can I get rid of the error "Can't get attribute 'ut' on <module '__main__'>" which appears while I try to pickle load in jupyter notebook?
I have searched around but unfortunately found nothing that could help me to solve this problem.
I have made this unigram tagger and have saved it in this way :
output = open("nameofthefile.pkl&...
0
votes
1
answer
34
views
Why does APScheduler SQLAlchemyJobStore cannot pickle the job but the default MemoryJobStore can?
I have a job that involves sqlite database operation and I use the app scheduler do it for me in the background. It is working fine if it uses the default memory job store but if it uses the ...
0
votes
0
answers
28
views
PyTorch: Load/unpickle only the state dict of a model saved with torch.save
I saved a nn.Module model using (logically):
model = MyWeirdModel()
model.patched_features = .....
train(model)
torch.save(model, file)
Ideally, one would load this model using
model = torch.load(...
-1
votes
1
answer
80
views
Dumping data with pickle
I am very confused about how to use pickle, I'm trying to make the computation a lot faster by storing the values that have been computed, and creating a pickle file for the ones computed. When a ...
0
votes
0
answers
38
views
Pickling an lru_cache used not as a decorator
I've stumbled upon something strange. Basically, I use a library that takes the urllib.parse.urljoin function and wraps it in functools.lru_cache. But it applies the lru_cache "decorator" on ...
0
votes
0
answers
29
views
Python Geocoder call getting frozen after a certain number of executions
I am working on a Python application, which uses the geocoder library to find the latitude and longitude information for a given address and store it in a Redis key. All the information is serialized ...
0
votes
0
answers
29
views
How to save a fbprophet forecast model using pickle such that the model takes user input when loaded
I have built an fbProphet model which contains date and items as columns. There are 50 such item columns. I need to save this model in pickle format. Each time I load the model, it should take '...
0
votes
1
answer
26
views
How to transform new data when ML model is loaded from pickle?
i have KNN model pickled with StandartScaler.
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
when im trying to load model and pass new values via StandartScaler().transform() it ...
0
votes
0
answers
51
views
python stream file : no idea why this doesnt work ( but i have same stream id every time )
PS : (due to my country's teaching program, I need to write python code like this )
Hello, I have no idea why this doesn't work:
from pickle import load,dump
# verif function to check existence of a ...
1
vote
1
answer
42
views
Somehow an error is not being picked up by try... except
I am using a try... except loop to deal with opening a file that is updated throughout the day. Every now and then it would throw an error "pickle data is truncated", so I introduced the ...
0
votes
0
answers
7
views
Distributed Anchor Tabular pickle loading issue
I am trying to pickle distributed anchor tabular and use it later. I am able to pickle it (using dill) without any issue but while loading it takes forever to load.
Here's my code:
## Creating ...
1
vote
1
answer
27
views
pickle files loading time
I was just testing which file format suits my problem best and benchmarked a few formats. While doing this I stumbled upon the following, in my opinion, weird time difference when loading files.
I ...
0
votes
0
answers
25
views
What's the fastest way to go down a directory and read in multiple excel workbooks with multiple tabs?
I would like to go into a directory and retrieve every excel workbook that meets the criteria in my if statement (if the extension is ".xlsb" and the letters "USA" are in the name)....
0
votes
1
answer
34
views
pickle.load() adds a None object at the end
Hi there, I'm learning Python and trying to create a command-line Address Book as a dictionary, which is saved to a file using pickle (that's the briefing).
I've coded the add_contact and ...
-1
votes
0
answers
18
views
How to use pickle with class serialization [duplicate]
How do you use pickle to send a class object over tcp socket to use it over at the client
client.py
has a basic function to accept data from server and convert the data into a class
def connection(...
1
vote
1
answer
24
views
Pickle Not Dumping Everything?
I am currently working on a simple robot, and am looking to use Pickle to store the main object of the robot. Trying to interact with components initially declared does not seem to be working however. ...
0
votes
2
answers
25
views
Change default multiprocessing unpickler class
I have a multiprocessing program on Device A which uses a queue and a SyncManager to make this accessible over the network. The queue stores a custom class from a module on the device which gets ...
0
votes
0
answers
17
views
Using Pickle.dump() to Save a PKL file to an Object
I am trying to save a pkl file to an object M of the class obstacle_struct() using pickle.dump(). The line of code in question is:
pickle.dump(M,open('RRT_Models.pkl','wb'))
I am getting the ...
0
votes
0
answers
17
views
does pandas to_pickle and read_pickle put a lock on the file and release when done
I'm thinking that these functions don't lock the file, since there's nothing in the documentation to mention it. I have my own code for doing the locking and unlocking, which I can easily use, but I ...
0
votes
1
answer
14
views
How to pickle a subclass instance if super (from fixed library) rejects it?
I'm subclassing logging.Logger and although I'm implementing __getstate__ very safely, I get an error by the superclass saying you cannot pickle this. How to get around this?
Minimal reporducible code:...
-3
votes
1
answer
119
views
Pickle error: AttributeError: Can't get attribute on <module '__main__' (built-in)>
I am having a problem with pickle in python
import time,sys,pickle
print("welcome to vom")
#what if rooms are mycelium that connect tothe vom pokeballs
with open('test.p', 'rb') as f:
...
0
votes
0
answers
15
views
"Pickling" a weakref
import os
import discord
import pickle
if message.content.startswith("~setAdminChannel"):
file = 'C:/Users/Zekeb/PycharmProjects/pythonProject4/venv/botInformation.txt'
with open(...
0
votes
0
answers
9
views
Is there any way I can convert a C struct to a Python Pickle dump?
Good Evening,
I have a program written in C that I am trying to build a Python API for, however, I've found that the only way which I can accomplish this is through serialization.
I have already found ...