对字典元素进行多处理

Question

Objective:

ssh 到 excel 文件中列出的每个设备，执行命令，捕获输出并以表格格式显示输出（输出 table A 列：设备名称，B 列：命令输出）
使用多处理优化处理时间以扩展 100 秒的设备

到目前为止，我可以使用以下代码实现此目的，但我觉得应该有更好的方法来实现这一点。 reference

问题陈述：

我的输入是字典而不是列表，作为解决方法，我在下面的代码中将其转换为列表。
目标函数需要字典元素作为输入
每个进程的输出（switchname 和命令输出）应该返回到主进程以形成数据帧

示例 Table 来自 excel 文件

DeviceName	DeviceIPaddress
router1121	192.168.1.1
router1122	192.168.1.2
router1131	192.168.1.3
router1132	192.168.1.4

样本Table转换为字典

{0: {'DeviceName': 'router1121', 'DeviceIPaddress': '192.168.1.1'}, 1: {'DeviceName': 'router1122', 'DeviceIPaddress': '192.168.1.2'}, 2: {'DeviceName': 'router1131', 'DeviceIPaddress': '192.168.1.3'}, 3: {'DeviceName': 'router1132', 'DeviceIPaddress': '192.168.1.4'}}

需要优化和简化的工作代码

import multiprocessing
import paramiko
from collections import defaultdict
import pandas as pd
import stdiomask

ssh = paramiko.SSHClient()

def connect_device(admin_username, admin_password, devicename, ipaddress):
    print(f"Trying to connect to {devicename}")
    
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

    ssh.connect(hostname=ipaddress, port=22, allow_agent=False, username=admin_username, password=admin_password)

    print(f"Connection established to {devicename}")

    command = "show interface description | i Gi1/0/23"
    stdin, stdout, stderr = ssh.exec_command(command)
    out = stdout.read()
    ssh.close()
    return devicename, out

if __name__ == '__main__':
    
    df = exceltodataframe('Deviceinfo.xlsx', 'Devicedetails')

    # create dictionaries from the dataframes
    dfdict = df.T.to_dict()
    mylist = list(dfdict.items())

    admin_username = input("Enter the username: ")
    admin_password = stdiomask.getpass("Enter the password: ")

    def_dct_all = defaultdict(dict)
    
    with concurrent.futures.ProcessPoolExecutor() as executor:
        
        results = [executor.submit(connect_device, admin_username, admin_password, mylist[idx][1]['DeviceName'], mylist[idx][1]['DeviceIPaddress']) for idx, n in enumerate(mylist)]

        for index, f in enumerate(concurrent.futures.as_completed(results)):
            def_dct_all[index]['DeviceName'] = f.result()[0]
            def_dct_all[index]['Description'] = f.result()[1]

    df = pd.DataFrame(def_dct_all).T
    print(df)

Answer 1

我找到了另一种更好的方法来实现 objective 通过使用多处理池 class。

import multiprocessing
import paramiko
from collections import defaultdict
import pandas as pd
import stdiomask

ssh = paramiko.SSHClient()

def connect_device(admin_username, admin_password, devicename, ipaddress):

    devicename = mylist[1]['DeviceName']
    ipaddress = mylist[1]['DeviceIPaddress']

    print(f"Trying to connect to {devicename}")
    
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

    ssh.connect(hostname=ipaddress, port=22, allow_agent=False, username=admin_username, password=admin_password)

    print(f"Connection established to {devicename}")

    command = "show interface description | i Gi1/0/23"
    stdin, stdout, stderr = ssh.exec_command(command)
    out = stdout.read()
    ssh.close()
    return devicename, out

if __name__ == '__main__':
    
    df = exceltodataframe('Deviceinfo.xlsx', 'Devicedetails')

    # create dictionaries from the dataframes
    dfdict = df.T.to_dict()
    mylist = list(dfdict.items())

    admin_username = input("Enter the username: ")
    admin_password = stdiomask.getpass("Enter the password: ")

    p = multiprocessing.Pool()
    result = p.map(connect_device, mylist)

    def_dct_all = defaultdict(dict)
    
    for r in range(len(result)):
        def_dct_all[r]['DeviceName'] = result[r][0]
        def_dct_all[r]['Description'] = result[r][1].decode("utf-8").rstrip()

    resultdf = pd.DataFrame(def_dct_all).T
    print(resultdf)

对字典元素进行多处理

multiprocessing over dictionary elements

python

paramiko

python-multiprocessing