I am attempting to call another notebook from a pyspark notebook on AWS EMR. I can successfully call a notebook that runs functions such as f
and the function exists in memory to be called within cell 1. However, the function does not seem to stay in memory to be called for cell 2.
I am hoping there is a way to do this as I want to have all of my functions defined in one file. My testing functions and feedback in two other files which both separately call the functions file as well as my data processing file which finally processes the data and produces derivative data products.
Thank you
# called.ipynb
# Notebook to be called
print('Hello World')
f = lambda x: x**2
# caller.ipynb
# Notebook doing the calling
## -- notebook cell 1
%run called.ipynb
f(4)
## -- end of cell 1
# Prints:
# 'hello world'
# 16
## -- notebook cell 2
f(4)
## -- end of cell 2
# Prints:
# name 'f' is not defined
# Traceback (most recent call last):
# NameError: name 'f' is not defined
Please note I attempted:
sc.install_pypi_package("ipynb")
# Separate Cell
import ipynb
from ipynb import fs.full
from ipynb.fs.full.called import *
# No module named 'ipynb.fs.full.called'
# Traceback (most recent call last):
# ModuleNotFoundError: No module named 'ipynb.fs.full.called'
# and
from ipynb.fs.full.called.ipynb import *
# No module named 'ipynb.fs.full.called'
# Traceback (most recent call last):
# ModuleNotFoundError: No module named 'ipynb.fs.full.called'
.py
file and just import the file in the notebook. e.g., thefuncs.py
file having functions in the same directory as the caller notebook can be called in the notebook asfrom funcs import *