函数中的 Jupyter shell 命令

Question

我正在尝试创建一个函数，使用 shell 命令在 jupyter notebook 中加载 Sagemaker 模型。当我尝试将函数存储在 utilities.py 文件中并将其用于多个笔记本时，问题就出现了。

这是我在 jupyter 实验室笔记本中获取的 utilities.py 文件的内容。

def get_aws_sagemaker_model(model_loc):
    """
    TO BE USED IN A JUPYTER NOTEBOOK
    
    extracts a sagemaker model that has ran and been completed
    
    deletes the copied items and leaves you with the model
    
    note that you will need to have the package installed with correct
    versioning for whatever model you have trained
    ie. if you are loading an XGBoost model, have XGBoost installed
    
    Args:
        model_loc (str) : s3 location of the model including file name
        
    Return:
        model: unpacked and loaded model
    """ 
    
    import re
    import tarfile
    import os
    import pickle as pkl

    # extract the filename from beyond the last backslash
    packed_model_name = re.search("(.*\/)(.*)$" , model_loc)[2]
    
    # copy and paste model file locally
    command_string = "!aws s3 cp {model_loc} ."
    exec(command_string)
    
    # use tarfile to extract
    tar = tarfile.open(packed_model_name)
    
    # extract filename from tarfile
    unpacked_model_name = tar.getnames()[0]
    
    tar.extractall()
    tar.close()
    
    model = pkl.load(open(unpacked_model_name, 'rb'))
    
    # cleanup copied files and unpacked model
    os.remove(packed_model_name)
    os.remove(unpacked_model_name)
    
    return model

尝试执行命令字符串时出现错误：

Traceback (most recent call last):

  File "/home/ec2-user/anaconda3/envs/env/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3444, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "/tmp/ipykernel_10889/996524724.py", line 1, in <module>
    model = get_aws_sagemaker_model("my-model-loc")

  File "/home/ec2-user/SageMaker/env/src/utilities/model_helper_functions.py", line 167, in get_aws_sagemaker_model
    exec(command_string)

  File "<string>", line 1
    !aws s3 cp my-model-loc .
    ^
SyntaxError: invalid syntax

在 exec 检查语法之前，jupyter 似乎没有收到命令。除了将函数复制到我使用的每个 jupyter notebook 之外，还有其他解决方法吗？

谢谢！

Answer 1

一个!魔法可以包含在函数中，但不能通过exec执行。

def foo(astr):
    !ls $astr

foo('*.py')

会和

一样

!ls *.py

Answer 2

您可以使用 IPython 的 shell 的 transform_cell 方法将 IPython 语法转换为有效的普通-Python:

from IPython import get_ipython
ipython = get_ipython()

code = ipython.transform_cell('!ls')
print(code)

这将显示：

get_ipython().system('!ls')

您可以将其用作 exec 的输入：

exec(code)

或直接：

exec(ipython.transform_cell('!ls'))

函数中的 Jupyter shell 命令

Jupyter shell commands in a function

python

jupyter-notebook

jupyter

jupyter-lab