在具有大量输入的 for 循环上实现 Pool
Implementing Pool on a for loop with a lot of inputs
我一直在尝试改进我的代码(使用 numba 和多处理),但我不太明白,因为我的函数有很多参数。
我已经用其他函数简化了它(见下文)...
由于每个代理(一个 class 实例)对于这些操作都是相互独立的,因此我想将 for
替换为 Pool
。
所以我会得到一个大函数pooling()
,我会调用它并传递代理列表
from multiprocessing import Pool
p = Pool(4)
p.map(pooling, list(agents))
但是,我在哪里添加池函数需要的所有参数?
原样:
def check_demographics(month, my_agents, families, firms, year, mortality_men, mortality_women, fertility, state_id):
dummy = list(my_agents)
d = str(state_id.iloc[0])
# Place where I would like to replace the LOOP. All below would be a function
for agent in dummy:
if agent.get_region_id()[:2] == d:
# Brithday
if month % 12 == agent.month - 1:
agent.update_age()
# Mortality probability
if agent.get_gender() == 'Male':
prob = mortality_men[mortality_men['age'] == agent.get_age()][year].iloc[0]
# When gender is Female
else:
# Extract specific agent data to calculate mortality 'Female'
prob = mortality_women[mortality_women['age'] == agent.get_age()][year].iloc[0]
# Give birth decision
age = agent.get_age()
if 14 < age < 50:
pregnant(agent, fertility, year, families, my_agents)
# Mortality procedures
if fixed_seed.random() < prob:
mortal(my_agents, my_graveyard, families, agent, firms)
这是我程序中最耗时的函数。
@jit
并没有多大帮助。
非常感谢
是的,有很多参数!考虑使用 class.
好吧,由于 Pool.map
只支持一个可迭代参数,您需要将所有内容组合在一个地方。我建议你使用 "Facade" 模式:一个中间 class 用于存储所有必需的参数并有一个没有参数的方法(我称之为 check
)(这是一个方法)。
class Facade(object):
def __init__(self, agent, d, families, fertility, firms, month, mortality_men, mortality_women, my_agents,
my_graveyard, year):
self.agent = agent
self.d = d
self.families = families
self.fertility = fertility
self.firms = firms
self.month = month
self.mortality_men = mortality_men
self.mortality_women = mortality_women
self.my_agents = my_agents
self.my_graveyard = my_graveyard
self.year = year
def check(self):
(agent, d, families, fertility, firms,
month, mortality_men, mortality_women,
my_agents, my_graveyard, year) = (
self.agent, self.d, self.families, self.fertility, self.firms,
self.month, self.mortality_men, self.mortality_women,
self.my_agents, self.my_graveyard, self.year)
if agent.get_region_id()[:2] == d:
# Brithday
if month % 12 == agent.month - 1:
agent.update_age()
# Mortality probability
if agent.get_gender() == 'Male':
prob = mortality_men[mortality_men['age'] == agent.get_age()][year].iloc[0]
# When gender is Female
else:
# Extract specific agent data to calculate mortality 'Female'
prob = mortality_women[mortality_women['age'] == agent.get_age()][year].iloc[0]
# Give birth decision
age = agent.get_age()
if 14 < age < 50:
pregnant(agent, fertility, year, families, my_agents)
# Mortality procedures
if fixed_seed.random() < prob:
mortal(my_agents, my_graveyard, families, agent, firms)
备注:我的重构真的很丑,但为了清楚起见,我想保持变量名不变。
那么你的循环可以是这样的:
def check_demographics(month, my_agents, families, firms,
year, mortality_men, mortality_women,
fertility, state_id, my_graveyard):
d = str(state_id.iloc[0])
pool = Pool(4)
facades = [Facade(agent, d, families, fertility, firms,
month, mortality_men, mortality_women,
my_agents, my_graveyard, year)
for agent in my_agents]
pool.map(Facade.check, facades)
你说每个代理都是相互独立的,但是在分析循环之后,我看到你需要完整的代理列表(my_agents
参数)。 Facade
class 很明显。所以你的代理列表不能改变,每个代理的内部状态必须在循环期间被冻结。
我一直在尝试改进我的代码(使用 numba 和多处理),但我不太明白,因为我的函数有很多参数。
我已经用其他函数简化了它(见下文)...
由于每个代理(一个 class 实例)对于这些操作都是相互独立的,因此我想将 for
替换为 Pool
。
所以我会得到一个大函数pooling()
,我会调用它并传递代理列表
from multiprocessing import Pool
p = Pool(4)
p.map(pooling, list(agents))
但是,我在哪里添加池函数需要的所有参数?
原样:
def check_demographics(month, my_agents, families, firms, year, mortality_men, mortality_women, fertility, state_id):
dummy = list(my_agents)
d = str(state_id.iloc[0])
# Place where I would like to replace the LOOP. All below would be a function
for agent in dummy:
if agent.get_region_id()[:2] == d:
# Brithday
if month % 12 == agent.month - 1:
agent.update_age()
# Mortality probability
if agent.get_gender() == 'Male':
prob = mortality_men[mortality_men['age'] == agent.get_age()][year].iloc[0]
# When gender is Female
else:
# Extract specific agent data to calculate mortality 'Female'
prob = mortality_women[mortality_women['age'] == agent.get_age()][year].iloc[0]
# Give birth decision
age = agent.get_age()
if 14 < age < 50:
pregnant(agent, fertility, year, families, my_agents)
# Mortality procedures
if fixed_seed.random() < prob:
mortal(my_agents, my_graveyard, families, agent, firms)
这是我程序中最耗时的函数。
@jit
并没有多大帮助。
非常感谢
是的,有很多参数!考虑使用 class.
好吧,由于 Pool.map
只支持一个可迭代参数,您需要将所有内容组合在一个地方。我建议你使用 "Facade" 模式:一个中间 class 用于存储所有必需的参数并有一个没有参数的方法(我称之为 check
)(这是一个方法)。
class Facade(object):
def __init__(self, agent, d, families, fertility, firms, month, mortality_men, mortality_women, my_agents,
my_graveyard, year):
self.agent = agent
self.d = d
self.families = families
self.fertility = fertility
self.firms = firms
self.month = month
self.mortality_men = mortality_men
self.mortality_women = mortality_women
self.my_agents = my_agents
self.my_graveyard = my_graveyard
self.year = year
def check(self):
(agent, d, families, fertility, firms,
month, mortality_men, mortality_women,
my_agents, my_graveyard, year) = (
self.agent, self.d, self.families, self.fertility, self.firms,
self.month, self.mortality_men, self.mortality_women,
self.my_agents, self.my_graveyard, self.year)
if agent.get_region_id()[:2] == d:
# Brithday
if month % 12 == agent.month - 1:
agent.update_age()
# Mortality probability
if agent.get_gender() == 'Male':
prob = mortality_men[mortality_men['age'] == agent.get_age()][year].iloc[0]
# When gender is Female
else:
# Extract specific agent data to calculate mortality 'Female'
prob = mortality_women[mortality_women['age'] == agent.get_age()][year].iloc[0]
# Give birth decision
age = agent.get_age()
if 14 < age < 50:
pregnant(agent, fertility, year, families, my_agents)
# Mortality procedures
if fixed_seed.random() < prob:
mortal(my_agents, my_graveyard, families, agent, firms)
备注:我的重构真的很丑,但为了清楚起见,我想保持变量名不变。
那么你的循环可以是这样的:
def check_demographics(month, my_agents, families, firms,
year, mortality_men, mortality_women,
fertility, state_id, my_graveyard):
d = str(state_id.iloc[0])
pool = Pool(4)
facades = [Facade(agent, d, families, fertility, firms,
month, mortality_men, mortality_women,
my_agents, my_graveyard, year)
for agent in my_agents]
pool.map(Facade.check, facades)
你说每个代理都是相互独立的,但是在分析循环之后,我看到你需要完整的代理列表(my_agents
参数)。 Facade
class 很明显。所以你的代理列表不能改变,每个代理的内部状态必须在循环期间被冻结。