2

I recently changed my IDE from Spyder to PyCharm, as I felt Spyder used through Anaconda was somewhat bloated. The code below runs more or less fine in Spyder, but I'm having issues in PyCharm. The code runs smoothly up until the data frame gets put in the model(1.1).Observing the metadata(?) related to the file, it should be suitable for analysis.

Every other places I have looked have been dealing with wrong formatting for file import. Far as I can see, this isn't the case with me, as the data fram works fine prior to the model.

Things I have done with hopes of resolving the isse:

  • Decreased the sample size from 100K to 50K to 10K to 1K
  • Changed the project directory with the actual
  • I reinstalled the following dependencies via conda contra pip: numpy, PyMC3, Theano
  • I increased the VM to 100GB
  • Installed new GPU driver for processing
  • I added preprocessing thinking it might have something to do with that

The error that I keep getting when running the model is:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Greencom\miniconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\Greencom\miniconda3\lib\multiprocessing\spawn.py", line 125, in _main
    prepare(preparation_data)
  File "C:\Users\Greencom\miniconda3\lib\multiprocessing\spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\Greencom\miniconda3\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "C:\Users\Greencom\miniconda3\lib\runpy.py", line 264, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "C:\Users\Greencom\miniconda3\lib\runpy.py", line 234, in _get_code_from_file
    with io.open_code(decoded_path) as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\Greencom\\OneDrive\\Dokumenter\\Trading\\Quant\\Python Analysis\\<input>'

I cannot see where the file directory has any relevancy to the actual model, so I don't understand the counter argument.

How do I solve this?

from scipy import stats
import arviz as az
import numpy as np
import matplotlib.pyplot as plt
import pymc3 as pm
import seaborn as sns
import pandas as pd
from theano import shared
from sklearn import preprocessing

print ( 'Running on PyMC3 v{}'.format ( pm.__version__ ) )

# data import #
EU50p1d = pd.read_excel (
    "C:\\Users\\Greencom/OneDrive\\Dokumenter\\Trading\\Quant\\Python Analysis\\CURRENCYCOM_EU501Dp.xlsx" )

# checking for 0-values #
EU50p1d.isnull ().sum () / len ( EU50p1d )

# Determining upper & lower value for model
lower = int ( EU50p1d.min () ) if int ( EU50p1d.min () ) == float ( EU50p1d.min () ) else float ( EU50p1d.min () )
upper = int ( EU50p1d.max () ) if int ( EU50p1d.max () ) == float ( EU50p1d.max () ) else float ( EU50p1d.max () )

# Gaussian Inferences #
az.plot_kde ( EU50p1d.close, rug=True )
plt.yticks ( [0], alpha=0 )


### 1.1 Model ##
with pm.Model() as model_g:
    μ = pm.Uniform('μ' , lower=2375.7 , upper=3858.3)
    σ = pm.HalfNormal('σ' , sd=100)
    y = pm.Normal('y' , mu=μ , sd=σ , observed=EU50p1d.values)
    trace_g = pm.sample(100000 , tune=1000)
5
  • Although it shouldn't matter, I see 1 / and a lot of escaped \\ in your code - but the error message has all \\ and you've removed the actual filename from it for some reason and replaced it with <input> even though the filename is still in the code. Can you please share the actual code and the full error message? Does the file actually exist at that location? Have you tried simply opening the same file from a test script that just does open('the_file_name', 'r') to see if it's a simple IO or permissions issue?
    – Grismar
    Aug 16, 2020 at 23:42
  • I changed the one / and it didn't alter the outcome. In addition, I couldn't find the escaped \\ you mentioned. But yes, the whole error will be added to OP. What you see above is the whole code. I tried open() and got the following: <_io.TextIOWrapper name='CURRENCYCOM_EU501Dp.xlsx' mode='r' encoding='cp1252'> Aug 17, 2020 at 0:04
  • It looks like the problem is with your conda environment - the error message makes it clear that the problem occurs during execution of a call to _main() in spawn.py. It ends up (bottom line of the error message) trying to open your .xlsx file as it if were a python file (using .open_code()). It's unclear why it would do that, but I would recommend simply creating a new virtual environment for your project and only installing the required packages to see if your code is fine otherwise. Apologies about the confusion about 'escape \\', I meant double backslash (i.e. escaped backslash)
    – Grismar
    Aug 17, 2020 at 2:13
  • Alright. No problem about the confusion, I'm still learning the terminology for python. :) But right, so I uninstalled and reinstalled conda and etc, starting completely from scratch, only installing the dependencies needed for my project. But the error still pops up, identical as before. I find this extremely strange. Aug 17, 2020 at 11:39
  • But, I would like to point out that your suggestions are very welcome. Aug 17, 2020 at 15:53

1 Answer 1

0

I decided to give your problem a try, but it appears your code has several problems that don't have much to do with the error, while the error doesn't present itself if I follow this procedure:

  • create a new empty project with only a main.py, set up a new Python 3.8 virtual environment for it
  • copy your code into main.py, only removing all unused imports and reformatting
  • create a new test.xlsx with some arbitrary data, change the file reference in the code
  • install the requirements:

Now, you can run the code and it will proceed happily to read the file. Of course, there's some problems in the code that stop it from working:

  • the conversion of EU50p1d.min() to an integer won't work
  • the Uniform() first parameter isn't expected to be a string, but that's what's passed

Also, I don't know what data is in your .xlsx, so of course there may be further problems that don't present themselves. But I doubt those would cause the issues you're describing.

It seems most likely that you're problem starts with not setting up an appropriate virtual environment. Assuming you use a recent Python and just using some example paths which you can change to match needs.

Ensure you're in a location where python on the command line actually executes the Python version you want a virtual environment for, or refer to that python.exe directly:

python -m venv "c:\my_venv\folder"
"c:\my_venv\folder\Scripts\activate"
pip install arviz
pip install pymc3
pip install xlrd

If you're using PyCharm, you can have PyCharm do this for you, just make sure you select the appropriate base Python interpreter before creating the new environment. If you do it manually as described, you'll need to tell PyCharm to use the environment you just created for the project you're working on, in the Project Settings for the Interpreter.

7
  • I've done as you've suggested and I'm seriously dumbfounded. I have created new environments x-amounts of times, and rerun the code as you suggest x-amount of times, but the error reappears everytime. Only this time the error has a new directory, depending on the project location. C:\\Users\\Greencom\\PycharmProjects\\pythonProject1\\<input> Could it have something to do with running Python 3.8.5? Aug 18, 2020 at 0:39
  • That version of Python should be fine, rather I'd suspect the specific distribution you base your virtual environment on. I would recommend downloading and installing a clean copy of Python. You can install a new copy without affecting existing copies, so it's safe to test. Use that version of Python and do as I described and your script should run just fine, well apart from the errors in it.
    – Grismar
    Aug 18, 2020 at 10:46
  • I'm still having issues, even after installing a new copy. Then I decided to clean everything. Removed Python 3.8.5, PyCharm and the Miniconda. Then I installed it again. The error still occur. Maybe PyCharm is just shit and I have to go back to my Spyder (though I'd rather not). Aug 19, 2020 at 17:52
  • PyCharm definitely isn't "shit" and also not the cause of your issue - I use PyCharm on all my Python development machines and are not having this issue. It seems more likely your problem is with your project or environment still. Did you take your code and dropped it in an entirely new project, with a new environment, based on a new Python install? PyCharm doesn't really affect anything, it's just an IDE.
    – Grismar
    Aug 20, 2020 at 12:43
  • Yeh, that's what I find so wierd. I've reinstalled the IDE and reinstalled Python. In addition I made venvs through cmd, once using pip and another time using conda. No use. Is it possible I could DM you, so that we could skype or something, so that I could share my screen or so? I'd appreciate the help. :) Aug 26, 2020 at 20:34

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.