从两个 Pandas DataFrames 添加一列到 DataFrame 并且当前使用两个带条件的循环：有更快的方法吗？

Question

我目前正在循环访问数据框中的 GPS 坐标。我正在使用此循环查看另一个具有特定位置 GPS 坐标的数据框，并使用最近的位置更新原始数据框。这工作正常，但速度很慢。有没有更快的方法？

这是示例数据：

进口：

from shapely.geometry import Point
import pandas as pd
from geopy import distance

创建样本 df1

gps_points = [Point(37.773972,-122.431297) , Point(35.4675602,-97.5164276) , Point(42.35843, -71.05977)]
df_gps = pd.DataFrame()
df_gps['points'] = gps_points

创建样本 df2

locations = {'location':['San Diego', 'Austin', 'Washington DC'],
        'gps':[Point(32.715738 , -117.161084), Point(30.267153 , -97.7430608), Point(38.89511 , -77.03637)]}
df_locations = pd.DataFrame(locations)

两次循环和更新：

lst = [] #create empty list to populate new df column
for index , row in df_gps.iterrows(): # iterate over first dataframe rows
    point = row['points'] # pull out GPS point
    closest_distance = 999999 # create container for distance
    closest_location = None #create container for closest location
    for index1 , row1 in df_locations.iterrows(): # iterate over second dataframe
        name = row1['location'] # assign name of location
        point2 = row1['gps'] # assign coordinates of location
        distances = distance.distance((point.x , point.y) , (point2.x , point2.y)).miles # calculate distance
        if distances < closest_distance: # check to see if distance is closer
            closest_distance = distances # if distance is closer assign it
            closest_location = name # if distance is closer assign name
    lst.append(closest_location) # append closest city
df_gps['closest_city'] = lst # add new column with closest cities

我真的很想以最快的方式做到这一点。我已经阅读了 pandas 的矢量化，并考虑过创建一个函数，然后使用 How to iterate over rows in a DataFrame in Pandas 中提到的 apply 但是我的代码中需要两个循环和一个条件，以便模式分解。谢谢你的帮助。

Answer 1

您可以使用 Scipy 中的 KDTree:

from scipy.spatial import KDTree

# Extract lat/lon from your dataframes
points = df_gps['points'].apply(lambda p: (p.x, p.y)).apply(pd.Series)
cities = df_locations['gps'].apply(lambda p: (p.x, p.y)).apply(pd.Series)

distances, indices = KDTree(cities).query(points)

df_gps['closest_city'] = df_locations.iloc[indices]['location'].values
df_gps['distance'] = distances

您可以使用 np.where 过滤掉太远的距离。

为了性能，检查是否存在类似问题，df_gps 有 25k 行，df_locations 有 200k 行。

Answer 2

基于Corralien的见解，代码中的最终答案：

from sklearn.neighbors import BallTree, DistanceMetric

points = df_gps['points'].apply(lambda p: np.radians((p.x, p.y))).apply(pd.Series)
cities = df_locations['gps'].apply(lambda p: np.radians((p.x, p.y))).apply(pd.Series)
dist = DistanceMetric.get_metric('haversine')
tree = BallTree(cities, metric=dist)
dists, cities = tree.query(points)
df_gps['dist'] = dists.flatten() * 3956
df_gps['closest_city'] = df_locations.iloc[cities.flatten()]['location'].values

从两个 Pandas DataFrames 添加一列到 DataFrame 并且当前使用两个带条件的循环：有更快的方法吗？

adding a column to a dataframe from two Pandas DataFrames and currently using two loops with a conditional: Is there a faster way?

iteration

loops

vectorization

apply

pandas