concatenate tables from loop getting error - InvalidIndexError: Reindexing only valid with uniquely valued Index objects
concatenate tables from loop getting error - InvalidIndexError: Reindexing only valid with uniquely valued Index objects
我需要连接通过循环创建的表。列中有重复的名称,但它们讲述的是不同的故事,但由于某些原因,当 运行 这段代码出现错误时:
InvalidIndexError: Reindexing only valid with uniquely valued Index objects
代码如下:
url = 'https://www.impactfees.com/publications%20pdf/2019survey.pdf'
tables = camelot.read_pdf(url, flavor = 'stream', edge_tol = 500, pages = '4-end')
i = 0
while i in range(0,tables.n):
table_value = tables[i].df.loc[0,4]
header = 1
header = tables[i].df.iloc[header]
tables[i].df = tables[i].df.rename(columns = header)
nan_v = float("NaN")
tables[i].df.replace('',nan_v,inplace = True)
tables[i].df.dropna(subset = ['Jurisdiction'], inplace = True)
tables[i].df.replace(['Jurisdiction'], nan_v, inplace = True)
tables[i].df.dropna(subset = ['Jurisdiction'], inplace = True)
# Tot_col = tables[i].df.columns.get_loc('Total')
# tables[i].df = tables[i].df.iloc[:,0:Tot_col+1]
tables[i].df['report_name'] = table_value
tables[i].df.loc[~tables[i].df.index.duplicated(keep = 'first')]
i = i + 1
dfs = pd.concat([table.df for table in tables])
dfs
这是我得到的错误:
InvalidIndexError Traceback (most recent call last)
<ipython-input-133-2617eb5ae448> in <module>
23 i = i + 1
24
---> 25 dfs = pd.concat([table.df for table in tables])
26
27
~\anaconda3\lib\site-packages\pandas\core\reshape\concat.py in concat(objs, axis, join, ignore_index, keys, levels, names, verify_integrity, sort, copy)
296 )
297
--> 298 return op.get_result()
299
300
~\anaconda3\lib\site-packages\pandas\core\reshape\concat.py in get_result(self)
514 obj_labels = obj.axes[1 - ax]
515 if not new_labels.equals(obj_labels):
--> 516 indexers[ax] = obj_labels.get_indexer(new_labels)
517
518 mgrs_indexers.append((obj._mgr, indexers))
~\anaconda3\lib\site-packages\pandas\core\indexes\base.py in get_indexer(self, target, method, limit, tolerance)
3169
3170 if not self.is_unique:
-> 3171 raise InvalidIndexError(
3172 "Reindexing only valid with uniquely valued Index objects"
3173 )
InvalidIndexError: Reindexing only valid with uniquely valued Index objects
- camelot 有问题。我必须修补 utils.py 才能使用不同的
user-agent
- 页面不完全一致,因此将 list 传递给
rename(columns=)
无效。你需要传递一个 dict
- 保留了两个数据框 - 一个包含目标行,另一个包含排除行
- 仍然存在不一致的列,例如排水公园
import pandas as pd
import camelot
url = 'https://www.impactfees.com/publications%20pdf/2019survey.pdf'
tables = camelot.read_pdf(url, flavor = 'stream', edge_tol = 500, pages = '4-end')
df = pd.DataFrame()
dfexc = pd.DataFrame()
for i in range(tables.n):
dft = tables[i].df.rename(columns={i:v.replace("\n"," ") for i,v in tables[i].df.iloc[1].items() if v!=""})
if " " in dft.columns[0]:
dft = dft.rename(columns={dft.columns[0]:dft.columns[0].split(" ")[0], 1:dft.columns[0].split(" ")[1]})
m = (dft.State.str.len()!=2) | (dft.index < 2)
dfexc = pd.concat([dfexc, tables[i].df.loc[m].assign(page=i)])
df = pd.concat([df, dft.loc[~m].assign(page=i)])#.reset_index(drop=True)
我需要连接通过循环创建的表。列中有重复的名称,但它们讲述的是不同的故事,但由于某些原因,当 运行 这段代码出现错误时:
InvalidIndexError: Reindexing only valid with uniquely valued Index objects
代码如下:
url = 'https://www.impactfees.com/publications%20pdf/2019survey.pdf'
tables = camelot.read_pdf(url, flavor = 'stream', edge_tol = 500, pages = '4-end')
i = 0
while i in range(0,tables.n):
table_value = tables[i].df.loc[0,4]
header = 1
header = tables[i].df.iloc[header]
tables[i].df = tables[i].df.rename(columns = header)
nan_v = float("NaN")
tables[i].df.replace('',nan_v,inplace = True)
tables[i].df.dropna(subset = ['Jurisdiction'], inplace = True)
tables[i].df.replace(['Jurisdiction'], nan_v, inplace = True)
tables[i].df.dropna(subset = ['Jurisdiction'], inplace = True)
# Tot_col = tables[i].df.columns.get_loc('Total')
# tables[i].df = tables[i].df.iloc[:,0:Tot_col+1]
tables[i].df['report_name'] = table_value
tables[i].df.loc[~tables[i].df.index.duplicated(keep = 'first')]
i = i + 1
dfs = pd.concat([table.df for table in tables])
dfs
这是我得到的错误:
InvalidIndexError Traceback (most recent call last)
<ipython-input-133-2617eb5ae448> in <module>
23 i = i + 1
24
---> 25 dfs = pd.concat([table.df for table in tables])
26
27
~\anaconda3\lib\site-packages\pandas\core\reshape\concat.py in concat(objs, axis, join, ignore_index, keys, levels, names, verify_integrity, sort, copy)
296 )
297
--> 298 return op.get_result()
299
300
~\anaconda3\lib\site-packages\pandas\core\reshape\concat.py in get_result(self)
514 obj_labels = obj.axes[1 - ax]
515 if not new_labels.equals(obj_labels):
--> 516 indexers[ax] = obj_labels.get_indexer(new_labels)
517
518 mgrs_indexers.append((obj._mgr, indexers))
~\anaconda3\lib\site-packages\pandas\core\indexes\base.py in get_indexer(self, target, method, limit, tolerance)
3169
3170 if not self.is_unique:
-> 3171 raise InvalidIndexError(
3172 "Reindexing only valid with uniquely valued Index objects"
3173 )
InvalidIndexError: Reindexing only valid with uniquely valued Index objects
- camelot 有问题。我必须修补 utils.py 才能使用不同的
user-agent
- 页面不完全一致,因此将 list 传递给
rename(columns=)
无效。你需要传递一个 dict - 保留了两个数据框 - 一个包含目标行,另一个包含排除行
- 仍然存在不一致的列,例如排水公园
import pandas as pd
import camelot
url = 'https://www.impactfees.com/publications%20pdf/2019survey.pdf'
tables = camelot.read_pdf(url, flavor = 'stream', edge_tol = 500, pages = '4-end')
df = pd.DataFrame()
dfexc = pd.DataFrame()
for i in range(tables.n):
dft = tables[i].df.rename(columns={i:v.replace("\n"," ") for i,v in tables[i].df.iloc[1].items() if v!=""})
if " " in dft.columns[0]:
dft = dft.rename(columns={dft.columns[0]:dft.columns[0].split(" ")[0], 1:dft.columns[0].split(" ")[1]})
m = (dft.State.str.len()!=2) | (dft.index < 2)
dfexc = pd.concat([dfexc, tables[i].df.loc[m].assign(page=i)])
df = pd.concat([df, dft.loc[~m].assign(page=i)])#.reset_index(drop=True)