您如何正确快速地比较数据行/数据表?
How do you correctly and quickly compare Datarows / Datatables?
更新:解释我正在比较的数据类型table-
“比较具有相同列的两个数据 table,一个数据 table 正在从外部服务器中拉出并插入 最初,从那时起,只有最近 6 个月的记录从外部数据库中拉出(对于各种reasons) ,并将数据与本地数据(6 个月的日期范围)进行比较,以查看 DataRow 是否已更改,需要删除或添加行标识符 (PKey) 本质上是 SalesID + LineRow 匹配和其他列是要比较的值,以查看该行是否需要 re-added/deleted,因为传入列与当前列不同,并且还会删除传入数据不包含这些行的行
所以基本上我想要一个
Exclusive Left Join [插入数据]
和
独占权加入[删除该数据]
“
我一直在做一些数据库编码以及 JSON 拉动,我想知道什么是标准的/正确的做事方式,我从 2 小时的比较时间开始(在虚拟数据库上 table) 减少到 1 小时到 1 秒(在将我的 janky 方法应用到 DB table Compare 之后),然后最终将它用于实时拉取,结果似乎是正确且一致的,所以我开始进行测试在虚拟数据上,它从 1 小时到 26 分钟到最后 <1 秒(使用我自己的简陋方式),Table 测试和假设的大小在 100,000 和 200,000 行之间
所以让我们通过我尝试过的标准方法,然后继续我制作的简陋解决方案。
第一个明显的想法是使用两次 ForEach
迭代(即使在心理上这似乎会很慢,但我认为考虑到 Add 的速度有多快以及您可以多快在遍历 Jarrays 时比较 JSON 个标记)。代码类似于以下内容:
DataTable dtQueryItemsDiff = dtItems.Clone();
DataTable dtItemsDiff = dtItems.Clone();
int maxRowCountCache = dtItems.AsEnumerable().OrderBy(row => Convert.ToDateTime(row.Field<String>("Date"))).ThenBy(row => row.Field<String>("Name")).Count();
int rowcountCCache = 0;
var query = dtQuery.AsEnumerable().OrderBy(row => Convert.ToDateTime(row.Field<String>("Date"))).ThenBy(row => row.Field<String>("Name"));
foreach (DataRow drDTI in dtItems.AsEnumerable().OrderBy(row => Convert.ToDateTime(row.Field<String>("Date"))).ThenBy(row => row.Field<String>("Name")))
{
int innerrowcount = 0;
bool rowfound = false;
if (query.Count() != 0)
{
foreach (DataRow drDTQ in query)
{
if (drDTI["SalesID"].ToString() == drDTI["SalesID"].ToString() && drDTI["LineNumber"].ToString() == drDTI["LineNumber"].ToString())
{
rowfound = true;
break;
}
innerrowcount++;
}
}
else
{
dtItemsDiff.ImportRow(drDTI);
continue;
}
if (rowfound == true)
{
orderedDtquery.ElementAt(innerrowcount).Delete();
}
else
{
dtItemsDiff.ImportRow(drDTI);
}
rowcountCCache++;
BeginInvoke(new MethodInvoker(delegate
{
lblDataLoadC.Text = rowcountCCache.ToString() + " / " + maxRowCountCache.ToString();
}));
}
if (query.Count() != 0)
{
foreach (DataRow drDTQ in query)
{
dtQueryItemsDiff.ImportRow(drDTQ);
}
}
这花费了大约 1H(1 小时)到 1.5H 的相当长的时间,具体取决于数据的排序方式等。好处是我可以精细地更改代码,它在两个方面都为我提供了不匹配的数据tables,它还减少了搜索的查询大小,但这对我来说还不够快,所以我尝试了 Linq 搜索,但我并没有减少列表大小(先删除再搜索再慢一些)只是搜索),这花了大约 40-50 分钟,看起来像:
int maxRowCountCache = dtItems.AsEnumerable().OrderBy(row => Convert.ToDateTime(row.Field<String>("Date"))).ThenBy(row => row.Field<String>("Name")).Count();
int rowcountCCache = 0;
dtItems.AcceptChanges();
foreach (DataRow drDTI in dtItems.AsEnumerable().OrderBy(row => Convert.ToDateTime(row.Field<String>("Date"))).ThenBy(row => row.Field<String>("Name")))
{
var checkIfRecordInIDB = progSettings.query.AsEnumerable().Where(row => row.Field<string>("CardRecordID") == drDTI["CardRecordID"].ToString()
&& row.Field<string>("Date") == drDTI["Date"].ToString() && row.Field<string>("SaleID") == drDTI["SaleID"].ToString()
&& row.Field<string>("ItemID") == drDTI["ItemID"].ToString() && row.Field<Int64>("LineNumber") == Convert.ToInt64(drDTI["LineNumber"].ToString())).FirstOrDefault();
if (checkIfRecordInIDB != null)
{
drDTI.Delete();
}
rowcountCCache++;
BeginInvoke(new MethodInvoker(delegate
{
lblDataLoadC.Text = rowcountCCache.ToString() + " / " + maxRowCountCache.ToString();
}));
}
dtItems.AcceptChanges();
这样做的好处是它稍微更懒惰、更快和简洁,但是它只为您提供一个 table 中的数据,就像 Except 所做的一样,这正是我接下来使用 ~100,000 行虚拟数据尝试的, 这花了 26 分 35 秒。
dtItems.Rows.Clear();
query.Rows.Clear();
Thread start = new Thread(timerAndUIupdate);
start.Start();
dtItems.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "72421ee8-459b-46fb-bf5a-f51e80976e5a", "Pioneer 1kg (FT), RRP ", "100115", 1, 25.0, "N");
dtItems.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "8885a911-8d32-4dfe-93e5-2e453fd54db9", "Decaf Beans 250g FT", "1002302", 2, 2.0, "N");
dtItems.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "e3aa4b15-b774-4f6a-ac21-77fa05a4332f", "P&R Cups 06oz (1000)", "30056", 3, 1.0, "N");
dtItems.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "51e1a867-4079-4a3c-9ddc-e93d87d80b46", "P&R Cups 12oz (1000)", "30058", 4, 1.0, "N");
query.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "72421ee8-459b-46fb-bf5a-f51e80976e5a", "Pioneer 1kg (FT), RRP ", "100115", 1, 25.0, "N");
query.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "8885a911-8d32-4dfe-93e5-2e453fd54db9", "Decaf Beans 250g FT", "1002302", 2, 2.0, "N");
query.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "e3aa4b15-b774-4f6a-ac21-77fa05a4332f", "P&R Cups 06oz (1000)", "30056", 3, 1.0, "N");
query.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "51e1a867-4079-4a3c-9ddc-e93d87d80b46", "P&R Cups 12oz (1000)", "30058", 4, 1.0, "N");
for (int i = 1; i < 100000; i++)
{
dtItems.Rows.Add("Bennett St Dairy", "ed0c8d30-6469-4e13-af5a-36d7357a4a70", "2019-07-01", "2019-07-01", "8b909a4b-a07b-4a06-bebc-6a3387433aaf", "c8cc1115-da02-42cf-b427-accc1b6d07e3", "Trailblazer 1Kg, RRP ", "10011", i, (i * 4), "N");
query.Rows.Add("Bennett St Dairy", "ed0c8d30-6469-4e13-af5a-36d7357a4a70", "2019-07-01", "2019-07-01", "8b909a4b-a07b-4a06-bebc-6a3387433aaf", "c8cc1115-da02-42cf-b427-accc1b6d07e3", "Trailblazer 1Kg, RRP ", "10011", i, (i * 4), "N");
}
dtItems.Rows.Add("Air Coffee International Cafe Pty Ltd", "bb4fa724-9759-4c60-93fe-70fbdfd00417", "2019-07-01", "2019-07-01", "b972f020-3740-4ef2-941f-78b1a9edefa8", "0be54733-ac0e-43f9-8ea5-204c7cdb5f48", "Custom 1kg", "100116", 1, 4.0, "N");
dtItems.Rows.Add("Allure Cafe & Co.", "f76f383f-e9f4-45c9-bb93-81102629b9c3", "2019-07-01", "2019-07-01", "2ad0667f-2254-4df5-8b24-eb36736cabb0", "6edc584b-a8eb-4f0b-a449-dbcb76a40a24", "Porter St 1Kg, RRP ", "100111", 1, 10.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "6edc584b-a8eb-4f0b-a449-dbcb76a40a24", "Porter St 1Kg, RRP ", "100111", 1, 30.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "51e1a867-4079-4a3c-9ddc-e93d87d80b46", "P&R Cups 12oz (1000)", "30058", 2, 12.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "401ce902-e158-4f21-85a5-3312c32457fc", "Lids 06/08/12oz (White) (1000)", "30062", 3, 7.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "9b80c825-6e9f-4f6b-9c77-f3378cc220e4", "4-Cup Cardboard Holders (300)", "41003", 4, 1.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "ea4c906e-fab1-4b15-8845-619f20e53c6a", "Organic Panela 1kg", "20014", 5, 2.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "bb3e1c10-9e67-46d3-99b4-17df45dead90", "Chocolate Powder 1Kg, RRP ", "20034", 6, 1.0, "N");
query.Rows.Add("Aussie Bites Cafe", "30389aca-9089-4b37-9a1e-5fbc3c2af485", "2019-07-01", "2019-07-01", "85df1af6-3d1e-4e04-8fe9-d90462a59d4c", "ea89ade4-c7ff-4d79-abcd-dcdbb8122562", "X Blend 1Kg, RRP ", "100112", 1, 4.0, "N");
query.Rows.Add("Aussie Bites Cafe", "30389aca-9089-4b37-9a1e-5fbc3c2af485", "2019-07-01", "2019-07-01", "85df1af6-3d1e-4e04-8fe9-d90462a59d4c", "21fe57ad-08f9-4c8b-81d0-d7b88b291571", "webfreight", "webfreight", 2, 1.0, "N");
query.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "6edc584b-a8eb-4f0b-a449-dbcb76a40a24", "Porter St 1Kg, RRP ", "100111", 1, 30.0, "N");
query.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "51e1a867-4079-4a3c-9ddc-e93d87d80b46", "P&R Cups 12oz (1000)", "30058", 2, 1.0, "N");
query.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "401ce902-e158-4f21-85a5-3312c32457fc", "Lids 06/08/12oz (White) (1000)", "30062", 3, 2.0, "N");
query.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "9b80c825-6e9f-4f6b-9c77-f3378cc220e4", "4-Cup Cardboard Holders (300)", "41003", 4, 1.0, "N");
Stopwatch pullTime = new();
pullTime.Start();
BeginInvoke(new MethodInvoker(delegate
{
lblTimerAddRowEnd.Text = "Start Time,Except: " + pullTime.Elapsed.ToString("mm\:ss\.ff");
}));
var orderedDtItems = dtItems.AsEnumerable().OrderBy(row => Convert.ToDateTime(row.Field<String>("Date"))).ThenBy(row => row.Field<String>("Name"));
var orderedDtquery = query.AsEnumerable().OrderBy(row => Convert.ToDateTime(row.Field<String>("Date"))).ThenBy(row => row.Field<String>("Name"));
DataTable excepteditems = orderedDtItems.Except(orderedDtquery, DataRowComparer.Default).CopyToDataTable();
BeginInvoke(new MethodInvoker(delegate
{
labelControl1.Text = "End Time,Except: " + pullTime.Elapsed.ToString("mm\:ss\.ff");
}));
BeginInvoke(new MethodInvoker(delegate
{
dgvResults.DataSource = excepteditems;
btnStart.Enabled = true;
simpleButton1.Enabled = true;
}));
使用此 UI 的更新程序代码(已线程化并用于所有测试比较):
private void timerAndUIupdate()
{
Stopwatch pullTime = new();
pullTime.Start();
do
{
Thread.Sleep(500);
BeginInvoke(new MethodInvoker(delegate
{
lblTimer.Text = "Timer: " + pullTime.Elapsed.ToString("mm\:ss\.ff");
Application.DoEvents();
}));
} while (btnStart.Enabled == false);
pullTime.Stop();
BeginInvoke(new MethodInvoker(delegate
{
lblTimer.Text = "Timer: " + pullTime.Elapsed.ToString("mm\:ss\.ff");
Application.DoEvents();
}));
}
Winforms 上的结果如下所示:
然后我做了我的简陋的方法,结果非常快而且看起来很准确,而且因为它只花了几分之一秒我可以多次执行这个来得到,新行,旧行不在拉那个不应删除和应删除的旧行 -> 代码如下所示
dtItems.Rows.Clear();
query.Rows.Clear();
Thread start = new Thread(timerAndUIupdate);
start.Start();
dtItems.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "72421ee8-459b-46fb-bf5a-f51e80976e5a", "Pioneer 1kg (FT), RRP ", "100115", 1, 25.0, "N");
dtItems.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "8885a911-8d32-4dfe-93e5-2e453fd54db9", "Decaf Beans 250g FT", "1002302", 2, 2.0, "N");
dtItems.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "e3aa4b15-b774-4f6a-ac21-77fa05a4332f", "P&R Cups 06oz (1000)", "30056", 3, 1.0, "N");
dtItems.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "51e1a867-4079-4a3c-9ddc-e93d87d80b46", "P&R Cups 12oz (1000)", "30058", 4, 1.0, "N");
query.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "72421ee8-459b-46fb-bf5a-f51e80976e5a", "Pioneer 1kg (FT), RRP ", "100115", 1, 25.0, "N");
query.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "8885a911-8d32-4dfe-93e5-2e453fd54db9", "Decaf Beans 250g FT", "1002302", 2, 2.0, "N");
query.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "e3aa4b15-b774-4f6a-ac21-77fa05a4332f", "P&R Cups 06oz (1000)", "30056", 3, 1.0, "N");
query.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "51e1a867-4079-4a3c-9ddc-e93d87d80b46", "P&R Cups 12oz (1000)", "30058", 4, 1.0, "N");
for (int i = 1; i < 100000; i++)
{
dtItems.Rows.Add("Bennett St Dairy", "ed0c8d30-6469-4e13-af5a-36d7357a4a70", "2019-07-01", "2019-07-01", "8b909a4b-a07b-4a06-bebc-6a3387433aaf", "c8cc1115-da02-42cf-b427-accc1b6d07e3", "Trailblazer 1Kg, RRP ", "10011", i, (i * 4), "N");
query.Rows.Add("Bennett St Dairy", "ed0c8d30-6469-4e13-af5a-36d7357a4a70", "2019-07-01", "2019-07-01", "8b909a4b-a07b-4a06-bebc-6a3387433aaf", "c8cc1115-da02-42cf-b427-accc1b6d07e3", "Trailblazer 1Kg, RRP ", "10011", i, (i * 4), "N");
}
dtItems.Rows.Add("Air Coffee International Cafe Pty Ltd", "bb4fa724-9759-4c60-93fe-70fbdfd00417", "2019-07-01", "2019-07-01", "b972f020-3740-4ef2-941f-78b1a9edefa8", "0be54733-ac0e-43f9-8ea5-204c7cdb5f48", "Custom 1kg", "100116", 1, 4.0, "N");
dtItems.Rows.Add("Allure Cafe & Co.", "f76f383f-e9f4-45c9-bb93-81102629b9c3", "2019-07-01", "2019-07-01", "2ad0667f-2254-4df5-8b24-eb36736cabb0", "6edc584b-a8eb-4f0b-a449-dbcb76a40a24", "Porter St 1Kg, RRP ", "100111", 1, 10.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "6edc584b-a8eb-4f0b-a449-dbcb76a40a24", "Porter St 1Kg, RRP ", "100111", 1, 30.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "51e1a867-4079-4a3c-9ddc-e93d87d80b46", "P&R Cups 12oz (1000)", "30058", 2, 12.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "401ce902-e158-4f21-85a5-3312c32457fc", "Lids 06/08/12oz (White) (1000)", "30062", 3, 7.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "9b80c825-6e9f-4f6b-9c77-f3378cc220e4", "4-Cup Cardboard Holders (300)", "41003", 4, 1.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "ea4c906e-fab1-4b15-8845-619f20e53c6a", "Organic Panela 1kg", "20014", 5, 2.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "bb3e1c10-9e67-46d3-99b4-17df45dead90", "Chocolate Powder 1Kg, RRP ", "20034", 6, 1.0, "N");
query.Rows.Add("Aussie Bites Cafe", "30389aca-9089-4b37-9a1e-5fbc3c2af485", "2019-07-01", "2019-07-01", "85df1af6-3d1e-4e04-8fe9-d90462a59d4c", "ea89ade4-c7ff-4d79-abcd-dcdbb8122562", "X Blend 1Kg, RRP ", "100112", 1, 4.0, "N");
query.Rows.Add("Aussie Bites Cafe", "30389aca-9089-4b37-9a1e-5fbc3c2af485", "2019-07-01", "2019-07-01", "85df1af6-3d1e-4e04-8fe9-d90462a59d4c", "21fe57ad-08f9-4c8b-81d0-d7b88b291571", "webfreight", "webfreight", 2, 1.0, "N");
query.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "6edc584b-a8eb-4f0b-a449-dbcb76a40a24", "Porter St 1Kg, RRP ", "100111", 1, 30.0, "N");
query.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "51e1a867-4079-4a3c-9ddc-e93d87d80b46", "P&R Cups 12oz (1000)", "30058", 2, 1.0, "N");
query.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "401ce902-e158-4f21-85a5-3312c32457fc", "Lids 06/08/12oz (White) (1000)", "30062", 3, 2.0, "N");
query.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "9b80c825-6e9f-4f6b-9c77-f3378cc220e4", "4-Cup Cardboard Holders (300)", "41003", 4, 1.0, "N");
Stopwatch pullTime = new();
pullTime.Start();
BeginInvoke(new MethodInvoker(delegate
{
lblTimerAddRowEnd.Text = "Start Time,Except: " + pullTime.Elapsed.ToString("mm\:ss\.ff");
}));
var orderedDtItems = dtItems.AsEnumerable().OrderBy(row => Convert.ToDateTime(row.Field<String>("Date"))).ThenBy(row => row.Field<String>("Name"));
var orderedDtquery = query.AsEnumerable().OrderBy(row => Convert.ToDateTime(row.Field<String>("Date"))).ThenBy(row => row.Field<String>("Name"));
dtOnlyNewRows.Rows.Clear();
HashSet<String> orderedDtItemsHS = new();
HashSet<String> orderedDtqueryHS = new();
HashSet<String> orderedDtItemsHSRemains = new();
HashSet<String> orderedDtqueryHSRemains = new();
foreach (DataRow dr in orderedDtquery)
{
orderedDtqueryHSRemains.Add(dr["CardRecordID"].ToString() + "⌁" + dr["Date"].ToString() + "⌁" + dr["SaleID"].ToString() + "⌁" + dr["ItemID"].ToString()
+ "⌁" + dr["LineNumber"].ToString() + "⌁" + dr["Quantity"].ToString());
orderedDtqueryHS.Add(dr["CardRecordID"].ToString() + "⌁" + dr["Date"].ToString() + "⌁" + dr["SaleID"].ToString() + "⌁" + dr["ItemID"].ToString()
+ "⌁" + dr["LineNumber"].ToString() + "⌁" + dr["Quantity"].ToString());
}
foreach (DataRow dr in orderedDtItems)
{
orderedDtItemsHSRemains.Add(dr["CardRecordID"].ToString() + "⌁" + dr["Date"].ToString() + "⌁" + dr["SaleID"].ToString() + "⌁" + dr["ItemID"].ToString()
+ "⌁" + dr["LineNumber"].ToString() + "⌁" + dr["Quantity"].ToString());
orderedDtItemsHS.Add(dr["CardRecordID"].ToString() + "⌁" + dr["Date"].ToString() + "⌁" + dr["SaleID"].ToString() + "⌁" + dr["ItemID"].ToString()
+ "⌁" + dr["LineNumber"].ToString() + "⌁" + dr["Quantity"].ToString());
bool added = orderedDtqueryHSRemains.Add(dr["CardRecordID"].ToString() + "⌁" + dr["Date"].ToString() + "⌁" + dr["SaleID"].ToString() + "⌁" + dr["ItemID"].ToString()
+ "⌁" + dr["LineNumber"].ToString() + "⌁" + dr["Quantity"].ToString());
if (added == false)
{
orderedDtqueryHSRemains.Remove(dr["CardRecordID"].ToString() + "⌁" + dr["Date"].ToString() + "⌁" + dr["SaleID"].ToString() + "⌁" + dr["ItemID"].ToString()
+ "⌁" + dr["LineNumber"].ToString() + "⌁" + dr["Quantity"].ToString());
}
else if (added == true)
{
dtOnlyNewRows.ImportRow(dr);
orderedDtqueryHSRemains.Remove(dr["CardRecordID"].ToString() + "⌁" + dr["Date"].ToString() + "⌁" + dr["SaleID"].ToString() + "⌁" + dr["ItemID"].ToString()
+ "⌁" + dr["LineNumber"].ToString() + "⌁" + dr["Quantity"].ToString());
}
}
foreach (DataRow dr in orderedDtquery)
{
bool added = orderedDtItemsHSRemains.Add(dr["CardRecordID"].ToString() + "⌁" + dr["Date"].ToString() + "⌁" + dr["SaleID"].ToString() + "⌁" + dr["ItemID"].ToString()
+ "⌁" + dr["LineNumber"].ToString() + "⌁" + dr["Quantity"].ToString());
if (added == false)
{
orderedDtItemsHSRemains.Remove(dr["CardRecordID"].ToString() + "⌁" + dr["Date"].ToString() + "⌁" + dr["SaleID"].ToString() + "⌁" + dr["ItemID"].ToString()
+ "⌁" + dr["LineNumber"].ToString() + "⌁" + dr["Quantity"].ToString());
}
else if (added == true)
{
DateTime rowTime = Convert.ToDateTime(dr["date"].ToString());
if (rowTime <= MonthCutOff)
{
dtOnlyLeftoverRows.ImportRow(dr);
}
else
{
dtOnlyDeleteRows.ImportRow(dr);
}
orderedDtItemsHSRemains.Remove(dr["CardRecordID"].ToString() + "⌁" + dr["Date"].ToString() + "⌁" + dr["SaleID"].ToString() + "⌁" + dr["ItemID"].ToString()
+ "⌁" + dr["LineNumber"].ToString() + "⌁" + dr["Quantity"].ToString());
}
}
Debug.WriteLine(dtOnlyNewRows.Rows.Count.ToString());
BeginInvoke(new MethodInvoker(delegate
{
labelControl1.Text = "End Time,Except: " + pullTime.Elapsed.ToString("mm\:ss\.ff");
}));
pullTime.Stop();
BeginInvoke(new MethodInvoker(delegate
{
dgvRowsRemaing.DataSource = dtOnlyLeftoverRows;
dgvResults.DataSource = dtOnlyNewRows;
dgvDeleteRows.DataSource = dtOnlyDeleteRows;
btnStart.Enabled = true;
}));
最终结果如下:
所有这些解释之后我的问题是:
- 我在其他方法中做错了什么,它们可以更快吗?
- 如果我的 janky 方法不对,我应该如何比较数据table?
- 只要能正常工作,即使卡顿也很快,就可以吗?
- 我的 janky 方法可能存在哪些问题?
编辑时间:2021-08-03 11:25 下午 AEST(澳大利亚东部标准时间)
Juris 编写的代码更简洁、更快捷,
应用于我的虚拟数据时的样子
Windows Forms
3 倍更快,更少混乱的代码,更短
这正是我要找的谢谢
我将通过使用一对字典为数据表编制索引来完成此操作。 DataTable 可以定义主键并执行在内部使用字典的快速查找,但通常使用数据表是非常难看的东西,所以没有必要向它添加更多 PK 丑陋
所以我们在右边有一些数据表,它是从数据库下载的,你已经决定“Foo”和“Bar”列是 PK。 Foo 是一个字符串,Bar 是一个整数:
Dim rIndex = new Dictionary(Of (ValueTuple(Of String, Integer), DataRow)
For Each r as DataRow In rightDt.Rows
Dim key = ( r.Field(Of String)("Foo"), r.Field(Of Integer)("Bar") )
rIndex(key) = r
Next r
我们有一些文件已读入左侧数据表。该文件的列恰好称为 Wit(字符串)和 Woo(int)
Dim lIndex = new Dictionary(Of (ValueTuple(Of String, Integer), DataRow)
For Each r as DataRow In leftDt.Rows
Dim key = (r.Field(Of String)("Wit"), r.Field(Of Integer)("Woo") )
lIndex(key) = r
Next r
现在,如果我们将密钥也存储到哈希集中,可能会让生活变得轻松;这代表左右结合
Dim allKeys as New HashSet(Of ValueTuple(Of String, Integer))
Dim rIndex = new Dictionary(Of (ValueTuple(Of String, Integer), DataRow)
For Each r as DataRow In rightDt.Rows
Dim key = ( r.Field(Of String)("Foo"), r.Field(Of Integer)("Bar") )
rIndex(key) = r
allKeys.Add(key)
Next r
Dim lIndex = new Dictionary(Of (ValueTuple(Of String, Integer), DataRow)
For Each r as DataRow In leftDt.Rows
Dim key = (r.Field(Of String)("Wit"), r.Field(Of Integer)("Woo") )
lIndex(key) = r
allKeys.Add(key)
Next r
剩下的就是枚举 allKeys 并询问字典是否包含它并决定做什么
For Each k in allKeys
Dim inL = lIndex.ContainsKey(k)
Dim inR = rIndex.ContainsKey(k)
If inL AndAlso inR Then
Dim updateRo = lIndex(k) 'update the db using this datarow
...
ElseIf inL Then
Dim insertRo = lIndex(k) 'insert this row to the db
...
Else
Dim deleteRo = rIndex(k) 'delete this row from the db
...
End If
Next k
--
哈,我才发现我的大脑还处于VB模式。这是上面的 C# 版本:
var allKeys = new HashSet<(string, int)>();
var rIndex = new Dictionary<(string, int), DataRow>();
foreach(DataRow r in rightDt.Rows){
var key = (r.Field<string>("Foo"), r.Field<int>("Bar"));
rIndex[key] = r;
allKeys.Add(key);
}
var lIndex = new Dictionary<(string, int), DataRow>();
foreach(DataRow r in leftDt.Rows){
var key = (r.Field<string>("Wit"), r.Field<int>("Woo"));
lIndex[key] = r;
allKeys.Add(key);
}
foreach(var k in allKeys){
var inL = lIndex.ContainsKey(k);
var inR = rIndex.ContainsKey(k);
if(inL && inR){
var updateRo = lIndex[k]; //update the db using this datarow
...
} else if(inL){
var insertRo = lIndex[k]; //insert this row to the db
...
} else {
var deleteRo = rIndex[k]; //delete this row from the db
...
}
}
查看工作示例
更新:解释我正在比较的数据类型table- “比较具有相同列的两个数据 table,一个数据 table 正在从外部服务器中拉出并插入 最初,从那时起,只有最近 6 个月的记录从外部数据库中拉出(对于各种reasons) ,并将数据与本地数据(6 个月的日期范围)进行比较,以查看 DataRow 是否已更改,需要删除或添加行标识符 (PKey) 本质上是 SalesID + LineRow 匹配和其他列是要比较的值,以查看该行是否需要 re-added/deleted,因为传入列与当前列不同,并且还会删除传入数据不包含这些行的行
所以基本上我想要一个 Exclusive Left Join [插入数据] 和 独占权加入[删除该数据] “
我一直在做一些数据库编码以及 JSON 拉动,我想知道什么是标准的/正确的做事方式,我从 2 小时的比较时间开始(在虚拟数据库上 table) 减少到 1 小时到 1 秒(在将我的 janky 方法应用到 DB table Compare 之后),然后最终将它用于实时拉取,结果似乎是正确且一致的,所以我开始进行测试在虚拟数据上,它从 1 小时到 26 分钟到最后 <1 秒(使用我自己的简陋方式),Table 测试和假设的大小在 100,000 和 200,000 行之间 所以让我们通过我尝试过的标准方法,然后继续我制作的简陋解决方案。
第一个明显的想法是使用两次 ForEach
迭代(即使在心理上这似乎会很慢,但我认为考虑到 Add 的速度有多快以及您可以多快在遍历 Jarrays 时比较 JSON 个标记)。代码类似于以下内容:
DataTable dtQueryItemsDiff = dtItems.Clone();
DataTable dtItemsDiff = dtItems.Clone();
int maxRowCountCache = dtItems.AsEnumerable().OrderBy(row => Convert.ToDateTime(row.Field<String>("Date"))).ThenBy(row => row.Field<String>("Name")).Count();
int rowcountCCache = 0;
var query = dtQuery.AsEnumerable().OrderBy(row => Convert.ToDateTime(row.Field<String>("Date"))).ThenBy(row => row.Field<String>("Name"));
foreach (DataRow drDTI in dtItems.AsEnumerable().OrderBy(row => Convert.ToDateTime(row.Field<String>("Date"))).ThenBy(row => row.Field<String>("Name")))
{
int innerrowcount = 0;
bool rowfound = false;
if (query.Count() != 0)
{
foreach (DataRow drDTQ in query)
{
if (drDTI["SalesID"].ToString() == drDTI["SalesID"].ToString() && drDTI["LineNumber"].ToString() == drDTI["LineNumber"].ToString())
{
rowfound = true;
break;
}
innerrowcount++;
}
}
else
{
dtItemsDiff.ImportRow(drDTI);
continue;
}
if (rowfound == true)
{
orderedDtquery.ElementAt(innerrowcount).Delete();
}
else
{
dtItemsDiff.ImportRow(drDTI);
}
rowcountCCache++;
BeginInvoke(new MethodInvoker(delegate
{
lblDataLoadC.Text = rowcountCCache.ToString() + " / " + maxRowCountCache.ToString();
}));
}
if (query.Count() != 0)
{
foreach (DataRow drDTQ in query)
{
dtQueryItemsDiff.ImportRow(drDTQ);
}
}
这花费了大约 1H(1 小时)到 1.5H 的相当长的时间,具体取决于数据的排序方式等。好处是我可以精细地更改代码,它在两个方面都为我提供了不匹配的数据tables,它还减少了搜索的查询大小,但这对我来说还不够快,所以我尝试了 Linq 搜索,但我并没有减少列表大小(先删除再搜索再慢一些)只是搜索),这花了大约 40-50 分钟,看起来像:
int maxRowCountCache = dtItems.AsEnumerable().OrderBy(row => Convert.ToDateTime(row.Field<String>("Date"))).ThenBy(row => row.Field<String>("Name")).Count();
int rowcountCCache = 0;
dtItems.AcceptChanges();
foreach (DataRow drDTI in dtItems.AsEnumerable().OrderBy(row => Convert.ToDateTime(row.Field<String>("Date"))).ThenBy(row => row.Field<String>("Name")))
{
var checkIfRecordInIDB = progSettings.query.AsEnumerable().Where(row => row.Field<string>("CardRecordID") == drDTI["CardRecordID"].ToString()
&& row.Field<string>("Date") == drDTI["Date"].ToString() && row.Field<string>("SaleID") == drDTI["SaleID"].ToString()
&& row.Field<string>("ItemID") == drDTI["ItemID"].ToString() && row.Field<Int64>("LineNumber") == Convert.ToInt64(drDTI["LineNumber"].ToString())).FirstOrDefault();
if (checkIfRecordInIDB != null)
{
drDTI.Delete();
}
rowcountCCache++;
BeginInvoke(new MethodInvoker(delegate
{
lblDataLoadC.Text = rowcountCCache.ToString() + " / " + maxRowCountCache.ToString();
}));
}
dtItems.AcceptChanges();
这样做的好处是它稍微更懒惰、更快和简洁,但是它只为您提供一个 table 中的数据,就像 Except 所做的一样,这正是我接下来使用 ~100,000 行虚拟数据尝试的, 这花了 26 分 35 秒。
dtItems.Rows.Clear();
query.Rows.Clear();
Thread start = new Thread(timerAndUIupdate);
start.Start();
dtItems.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "72421ee8-459b-46fb-bf5a-f51e80976e5a", "Pioneer 1kg (FT), RRP ", "100115", 1, 25.0, "N");
dtItems.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "8885a911-8d32-4dfe-93e5-2e453fd54db9", "Decaf Beans 250g FT", "1002302", 2, 2.0, "N");
dtItems.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "e3aa4b15-b774-4f6a-ac21-77fa05a4332f", "P&R Cups 06oz (1000)", "30056", 3, 1.0, "N");
dtItems.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "51e1a867-4079-4a3c-9ddc-e93d87d80b46", "P&R Cups 12oz (1000)", "30058", 4, 1.0, "N");
query.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "72421ee8-459b-46fb-bf5a-f51e80976e5a", "Pioneer 1kg (FT), RRP ", "100115", 1, 25.0, "N");
query.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "8885a911-8d32-4dfe-93e5-2e453fd54db9", "Decaf Beans 250g FT", "1002302", 2, 2.0, "N");
query.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "e3aa4b15-b774-4f6a-ac21-77fa05a4332f", "P&R Cups 06oz (1000)", "30056", 3, 1.0, "N");
query.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "51e1a867-4079-4a3c-9ddc-e93d87d80b46", "P&R Cups 12oz (1000)", "30058", 4, 1.0, "N");
for (int i = 1; i < 100000; i++)
{
dtItems.Rows.Add("Bennett St Dairy", "ed0c8d30-6469-4e13-af5a-36d7357a4a70", "2019-07-01", "2019-07-01", "8b909a4b-a07b-4a06-bebc-6a3387433aaf", "c8cc1115-da02-42cf-b427-accc1b6d07e3", "Trailblazer 1Kg, RRP ", "10011", i, (i * 4), "N");
query.Rows.Add("Bennett St Dairy", "ed0c8d30-6469-4e13-af5a-36d7357a4a70", "2019-07-01", "2019-07-01", "8b909a4b-a07b-4a06-bebc-6a3387433aaf", "c8cc1115-da02-42cf-b427-accc1b6d07e3", "Trailblazer 1Kg, RRP ", "10011", i, (i * 4), "N");
}
dtItems.Rows.Add("Air Coffee International Cafe Pty Ltd", "bb4fa724-9759-4c60-93fe-70fbdfd00417", "2019-07-01", "2019-07-01", "b972f020-3740-4ef2-941f-78b1a9edefa8", "0be54733-ac0e-43f9-8ea5-204c7cdb5f48", "Custom 1kg", "100116", 1, 4.0, "N");
dtItems.Rows.Add("Allure Cafe & Co.", "f76f383f-e9f4-45c9-bb93-81102629b9c3", "2019-07-01", "2019-07-01", "2ad0667f-2254-4df5-8b24-eb36736cabb0", "6edc584b-a8eb-4f0b-a449-dbcb76a40a24", "Porter St 1Kg, RRP ", "100111", 1, 10.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "6edc584b-a8eb-4f0b-a449-dbcb76a40a24", "Porter St 1Kg, RRP ", "100111", 1, 30.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "51e1a867-4079-4a3c-9ddc-e93d87d80b46", "P&R Cups 12oz (1000)", "30058", 2, 12.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "401ce902-e158-4f21-85a5-3312c32457fc", "Lids 06/08/12oz (White) (1000)", "30062", 3, 7.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "9b80c825-6e9f-4f6b-9c77-f3378cc220e4", "4-Cup Cardboard Holders (300)", "41003", 4, 1.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "ea4c906e-fab1-4b15-8845-619f20e53c6a", "Organic Panela 1kg", "20014", 5, 2.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "bb3e1c10-9e67-46d3-99b4-17df45dead90", "Chocolate Powder 1Kg, RRP ", "20034", 6, 1.0, "N");
query.Rows.Add("Aussie Bites Cafe", "30389aca-9089-4b37-9a1e-5fbc3c2af485", "2019-07-01", "2019-07-01", "85df1af6-3d1e-4e04-8fe9-d90462a59d4c", "ea89ade4-c7ff-4d79-abcd-dcdbb8122562", "X Blend 1Kg, RRP ", "100112", 1, 4.0, "N");
query.Rows.Add("Aussie Bites Cafe", "30389aca-9089-4b37-9a1e-5fbc3c2af485", "2019-07-01", "2019-07-01", "85df1af6-3d1e-4e04-8fe9-d90462a59d4c", "21fe57ad-08f9-4c8b-81d0-d7b88b291571", "webfreight", "webfreight", 2, 1.0, "N");
query.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "6edc584b-a8eb-4f0b-a449-dbcb76a40a24", "Porter St 1Kg, RRP ", "100111", 1, 30.0, "N");
query.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "51e1a867-4079-4a3c-9ddc-e93d87d80b46", "P&R Cups 12oz (1000)", "30058", 2, 1.0, "N");
query.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "401ce902-e158-4f21-85a5-3312c32457fc", "Lids 06/08/12oz (White) (1000)", "30062", 3, 2.0, "N");
query.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "9b80c825-6e9f-4f6b-9c77-f3378cc220e4", "4-Cup Cardboard Holders (300)", "41003", 4, 1.0, "N");
Stopwatch pullTime = new();
pullTime.Start();
BeginInvoke(new MethodInvoker(delegate
{
lblTimerAddRowEnd.Text = "Start Time,Except: " + pullTime.Elapsed.ToString("mm\:ss\.ff");
}));
var orderedDtItems = dtItems.AsEnumerable().OrderBy(row => Convert.ToDateTime(row.Field<String>("Date"))).ThenBy(row => row.Field<String>("Name"));
var orderedDtquery = query.AsEnumerable().OrderBy(row => Convert.ToDateTime(row.Field<String>("Date"))).ThenBy(row => row.Field<String>("Name"));
DataTable excepteditems = orderedDtItems.Except(orderedDtquery, DataRowComparer.Default).CopyToDataTable();
BeginInvoke(new MethodInvoker(delegate
{
labelControl1.Text = "End Time,Except: " + pullTime.Elapsed.ToString("mm\:ss\.ff");
}));
BeginInvoke(new MethodInvoker(delegate
{
dgvResults.DataSource = excepteditems;
btnStart.Enabled = true;
simpleButton1.Enabled = true;
}));
使用此 UI 的更新程序代码(已线程化并用于所有测试比较):
private void timerAndUIupdate()
{
Stopwatch pullTime = new();
pullTime.Start();
do
{
Thread.Sleep(500);
BeginInvoke(new MethodInvoker(delegate
{
lblTimer.Text = "Timer: " + pullTime.Elapsed.ToString("mm\:ss\.ff");
Application.DoEvents();
}));
} while (btnStart.Enabled == false);
pullTime.Stop();
BeginInvoke(new MethodInvoker(delegate
{
lblTimer.Text = "Timer: " + pullTime.Elapsed.ToString("mm\:ss\.ff");
Application.DoEvents();
}));
}
Winforms 上的结果如下所示:
然后我做了我的简陋的方法,结果非常快而且看起来很准确,而且因为它只花了几分之一秒我可以多次执行这个来得到,新行,旧行不在拉那个不应删除和应删除的旧行 -> 代码如下所示
dtItems.Rows.Clear();
query.Rows.Clear();
Thread start = new Thread(timerAndUIupdate);
start.Start();
dtItems.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "72421ee8-459b-46fb-bf5a-f51e80976e5a", "Pioneer 1kg (FT), RRP ", "100115", 1, 25.0, "N");
dtItems.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "8885a911-8d32-4dfe-93e5-2e453fd54db9", "Decaf Beans 250g FT", "1002302", 2, 2.0, "N");
dtItems.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "e3aa4b15-b774-4f6a-ac21-77fa05a4332f", "P&R Cups 06oz (1000)", "30056", 3, 1.0, "N");
dtItems.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "51e1a867-4079-4a3c-9ddc-e93d87d80b46", "P&R Cups 12oz (1000)", "30058", 4, 1.0, "N");
query.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "72421ee8-459b-46fb-bf5a-f51e80976e5a", "Pioneer 1kg (FT), RRP ", "100115", 1, 25.0, "N");
query.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "8885a911-8d32-4dfe-93e5-2e453fd54db9", "Decaf Beans 250g FT", "1002302", 2, 2.0, "N");
query.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "e3aa4b15-b774-4f6a-ac21-77fa05a4332f", "P&R Cups 06oz (1000)", "30056", 3, 1.0, "N");
query.Rows.Add("4 Beans Cafe", "2af0f4bf-52ea-44fb-b1b3-36181fe7bfdf", "2019-07-01", "2019-07-01", "7fc4f98a-35af-4da3-afe3-f7cfcd922ea7", "51e1a867-4079-4a3c-9ddc-e93d87d80b46", "P&R Cups 12oz (1000)", "30058", 4, 1.0, "N");
for (int i = 1; i < 100000; i++)
{
dtItems.Rows.Add("Bennett St Dairy", "ed0c8d30-6469-4e13-af5a-36d7357a4a70", "2019-07-01", "2019-07-01", "8b909a4b-a07b-4a06-bebc-6a3387433aaf", "c8cc1115-da02-42cf-b427-accc1b6d07e3", "Trailblazer 1Kg, RRP ", "10011", i, (i * 4), "N");
query.Rows.Add("Bennett St Dairy", "ed0c8d30-6469-4e13-af5a-36d7357a4a70", "2019-07-01", "2019-07-01", "8b909a4b-a07b-4a06-bebc-6a3387433aaf", "c8cc1115-da02-42cf-b427-accc1b6d07e3", "Trailblazer 1Kg, RRP ", "10011", i, (i * 4), "N");
}
dtItems.Rows.Add("Air Coffee International Cafe Pty Ltd", "bb4fa724-9759-4c60-93fe-70fbdfd00417", "2019-07-01", "2019-07-01", "b972f020-3740-4ef2-941f-78b1a9edefa8", "0be54733-ac0e-43f9-8ea5-204c7cdb5f48", "Custom 1kg", "100116", 1, 4.0, "N");
dtItems.Rows.Add("Allure Cafe & Co.", "f76f383f-e9f4-45c9-bb93-81102629b9c3", "2019-07-01", "2019-07-01", "2ad0667f-2254-4df5-8b24-eb36736cabb0", "6edc584b-a8eb-4f0b-a449-dbcb76a40a24", "Porter St 1Kg, RRP ", "100111", 1, 10.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "6edc584b-a8eb-4f0b-a449-dbcb76a40a24", "Porter St 1Kg, RRP ", "100111", 1, 30.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "51e1a867-4079-4a3c-9ddc-e93d87d80b46", "P&R Cups 12oz (1000)", "30058", 2, 12.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "401ce902-e158-4f21-85a5-3312c32457fc", "Lids 06/08/12oz (White) (1000)", "30062", 3, 7.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "9b80c825-6e9f-4f6b-9c77-f3378cc220e4", "4-Cup Cardboard Holders (300)", "41003", 4, 1.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "ea4c906e-fab1-4b15-8845-619f20e53c6a", "Organic Panela 1kg", "20014", 5, 2.0, "N");
dtItems.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "bb3e1c10-9e67-46d3-99b4-17df45dead90", "Chocolate Powder 1Kg, RRP ", "20034", 6, 1.0, "N");
query.Rows.Add("Aussie Bites Cafe", "30389aca-9089-4b37-9a1e-5fbc3c2af485", "2019-07-01", "2019-07-01", "85df1af6-3d1e-4e04-8fe9-d90462a59d4c", "ea89ade4-c7ff-4d79-abcd-dcdbb8122562", "X Blend 1Kg, RRP ", "100112", 1, 4.0, "N");
query.Rows.Add("Aussie Bites Cafe", "30389aca-9089-4b37-9a1e-5fbc3c2af485", "2019-07-01", "2019-07-01", "85df1af6-3d1e-4e04-8fe9-d90462a59d4c", "21fe57ad-08f9-4c8b-81d0-d7b88b291571", "webfreight", "webfreight", 2, 1.0, "N");
query.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "6edc584b-a8eb-4f0b-a449-dbcb76a40a24", "Porter St 1Kg, RRP ", "100111", 1, 30.0, "N");
query.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "51e1a867-4079-4a3c-9ddc-e93d87d80b46", "P&R Cups 12oz (1000)", "30058", 2, 1.0, "N");
query.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "401ce902-e158-4f21-85a5-3312c32457fc", "Lids 06/08/12oz (White) (1000)", "30062", 3, 2.0, "N");
query.Rows.Add("Mad Hatter Wine Co", "49340e5f-c7ef-41d9-9f1b-200711e6e629", "2021-07-28", "2021-07-28", "e16cbbac-c319-45f3-ac53-89d979fbcdc1", "9b80c825-6e9f-4f6b-9c77-f3378cc220e4", "4-Cup Cardboard Holders (300)", "41003", 4, 1.0, "N");
Stopwatch pullTime = new();
pullTime.Start();
BeginInvoke(new MethodInvoker(delegate
{
lblTimerAddRowEnd.Text = "Start Time,Except: " + pullTime.Elapsed.ToString("mm\:ss\.ff");
}));
var orderedDtItems = dtItems.AsEnumerable().OrderBy(row => Convert.ToDateTime(row.Field<String>("Date"))).ThenBy(row => row.Field<String>("Name"));
var orderedDtquery = query.AsEnumerable().OrderBy(row => Convert.ToDateTime(row.Field<String>("Date"))).ThenBy(row => row.Field<String>("Name"));
dtOnlyNewRows.Rows.Clear();
HashSet<String> orderedDtItemsHS = new();
HashSet<String> orderedDtqueryHS = new();
HashSet<String> orderedDtItemsHSRemains = new();
HashSet<String> orderedDtqueryHSRemains = new();
foreach (DataRow dr in orderedDtquery)
{
orderedDtqueryHSRemains.Add(dr["CardRecordID"].ToString() + "⌁" + dr["Date"].ToString() + "⌁" + dr["SaleID"].ToString() + "⌁" + dr["ItemID"].ToString()
+ "⌁" + dr["LineNumber"].ToString() + "⌁" + dr["Quantity"].ToString());
orderedDtqueryHS.Add(dr["CardRecordID"].ToString() + "⌁" + dr["Date"].ToString() + "⌁" + dr["SaleID"].ToString() + "⌁" + dr["ItemID"].ToString()
+ "⌁" + dr["LineNumber"].ToString() + "⌁" + dr["Quantity"].ToString());
}
foreach (DataRow dr in orderedDtItems)
{
orderedDtItemsHSRemains.Add(dr["CardRecordID"].ToString() + "⌁" + dr["Date"].ToString() + "⌁" + dr["SaleID"].ToString() + "⌁" + dr["ItemID"].ToString()
+ "⌁" + dr["LineNumber"].ToString() + "⌁" + dr["Quantity"].ToString());
orderedDtItemsHS.Add(dr["CardRecordID"].ToString() + "⌁" + dr["Date"].ToString() + "⌁" + dr["SaleID"].ToString() + "⌁" + dr["ItemID"].ToString()
+ "⌁" + dr["LineNumber"].ToString() + "⌁" + dr["Quantity"].ToString());
bool added = orderedDtqueryHSRemains.Add(dr["CardRecordID"].ToString() + "⌁" + dr["Date"].ToString() + "⌁" + dr["SaleID"].ToString() + "⌁" + dr["ItemID"].ToString()
+ "⌁" + dr["LineNumber"].ToString() + "⌁" + dr["Quantity"].ToString());
if (added == false)
{
orderedDtqueryHSRemains.Remove(dr["CardRecordID"].ToString() + "⌁" + dr["Date"].ToString() + "⌁" + dr["SaleID"].ToString() + "⌁" + dr["ItemID"].ToString()
+ "⌁" + dr["LineNumber"].ToString() + "⌁" + dr["Quantity"].ToString());
}
else if (added == true)
{
dtOnlyNewRows.ImportRow(dr);
orderedDtqueryHSRemains.Remove(dr["CardRecordID"].ToString() + "⌁" + dr["Date"].ToString() + "⌁" + dr["SaleID"].ToString() + "⌁" + dr["ItemID"].ToString()
+ "⌁" + dr["LineNumber"].ToString() + "⌁" + dr["Quantity"].ToString());
}
}
foreach (DataRow dr in orderedDtquery)
{
bool added = orderedDtItemsHSRemains.Add(dr["CardRecordID"].ToString() + "⌁" + dr["Date"].ToString() + "⌁" + dr["SaleID"].ToString() + "⌁" + dr["ItemID"].ToString()
+ "⌁" + dr["LineNumber"].ToString() + "⌁" + dr["Quantity"].ToString());
if (added == false)
{
orderedDtItemsHSRemains.Remove(dr["CardRecordID"].ToString() + "⌁" + dr["Date"].ToString() + "⌁" + dr["SaleID"].ToString() + "⌁" + dr["ItemID"].ToString()
+ "⌁" + dr["LineNumber"].ToString() + "⌁" + dr["Quantity"].ToString());
}
else if (added == true)
{
DateTime rowTime = Convert.ToDateTime(dr["date"].ToString());
if (rowTime <= MonthCutOff)
{
dtOnlyLeftoverRows.ImportRow(dr);
}
else
{
dtOnlyDeleteRows.ImportRow(dr);
}
orderedDtItemsHSRemains.Remove(dr["CardRecordID"].ToString() + "⌁" + dr["Date"].ToString() + "⌁" + dr["SaleID"].ToString() + "⌁" + dr["ItemID"].ToString()
+ "⌁" + dr["LineNumber"].ToString() + "⌁" + dr["Quantity"].ToString());
}
}
Debug.WriteLine(dtOnlyNewRows.Rows.Count.ToString());
BeginInvoke(new MethodInvoker(delegate
{
labelControl1.Text = "End Time,Except: " + pullTime.Elapsed.ToString("mm\:ss\.ff");
}));
pullTime.Stop();
BeginInvoke(new MethodInvoker(delegate
{
dgvRowsRemaing.DataSource = dtOnlyLeftoverRows;
dgvResults.DataSource = dtOnlyNewRows;
dgvDeleteRows.DataSource = dtOnlyDeleteRows;
btnStart.Enabled = true;
}));
最终结果如下:
所有这些解释之后我的问题是:
- 我在其他方法中做错了什么,它们可以更快吗?
- 如果我的 janky 方法不对,我应该如何比较数据table?
- 只要能正常工作,即使卡顿也很快,就可以吗?
- 我的 janky 方法可能存在哪些问题?
编辑时间:2021-08-03 11:25 下午 AEST(澳大利亚东部标准时间)
Juris 编写的代码更简洁、更快捷, 应用于我的虚拟数据时的样子 Windows Forms
3 倍更快,更少混乱的代码,更短 这正是我要找的谢谢
我将通过使用一对字典为数据表编制索引来完成此操作。 DataTable 可以定义主键并执行在内部使用字典的快速查找,但通常使用数据表是非常难看的东西,所以没有必要向它添加更多 PK 丑陋
所以我们在右边有一些数据表,它是从数据库下载的,你已经决定“Foo”和“Bar”列是 PK。 Foo 是一个字符串,Bar 是一个整数:
Dim rIndex = new Dictionary(Of (ValueTuple(Of String, Integer), DataRow)
For Each r as DataRow In rightDt.Rows
Dim key = ( r.Field(Of String)("Foo"), r.Field(Of Integer)("Bar") )
rIndex(key) = r
Next r
我们有一些文件已读入左侧数据表。该文件的列恰好称为 Wit(字符串)和 Woo(int)
Dim lIndex = new Dictionary(Of (ValueTuple(Of String, Integer), DataRow)
For Each r as DataRow In leftDt.Rows
Dim key = (r.Field(Of String)("Wit"), r.Field(Of Integer)("Woo") )
lIndex(key) = r
Next r
现在,如果我们将密钥也存储到哈希集中,可能会让生活变得轻松;这代表左右结合
Dim allKeys as New HashSet(Of ValueTuple(Of String, Integer))
Dim rIndex = new Dictionary(Of (ValueTuple(Of String, Integer), DataRow)
For Each r as DataRow In rightDt.Rows
Dim key = ( r.Field(Of String)("Foo"), r.Field(Of Integer)("Bar") )
rIndex(key) = r
allKeys.Add(key)
Next r
Dim lIndex = new Dictionary(Of (ValueTuple(Of String, Integer), DataRow)
For Each r as DataRow In leftDt.Rows
Dim key = (r.Field(Of String)("Wit"), r.Field(Of Integer)("Woo") )
lIndex(key) = r
allKeys.Add(key)
Next r
剩下的就是枚举 allKeys 并询问字典是否包含它并决定做什么
For Each k in allKeys
Dim inL = lIndex.ContainsKey(k)
Dim inR = rIndex.ContainsKey(k)
If inL AndAlso inR Then
Dim updateRo = lIndex(k) 'update the db using this datarow
...
ElseIf inL Then
Dim insertRo = lIndex(k) 'insert this row to the db
...
Else
Dim deleteRo = rIndex(k) 'delete this row from the db
...
End If
Next k
--
哈,我才发现我的大脑还处于VB模式。这是上面的 C# 版本:
var allKeys = new HashSet<(string, int)>();
var rIndex = new Dictionary<(string, int), DataRow>();
foreach(DataRow r in rightDt.Rows){
var key = (r.Field<string>("Foo"), r.Field<int>("Bar"));
rIndex[key] = r;
allKeys.Add(key);
}
var lIndex = new Dictionary<(string, int), DataRow>();
foreach(DataRow r in leftDt.Rows){
var key = (r.Field<string>("Wit"), r.Field<int>("Woo"));
lIndex[key] = r;
allKeys.Add(key);
}
foreach(var k in allKeys){
var inL = lIndex.ContainsKey(k);
var inR = rIndex.ContainsKey(k);
if(inL && inR){
var updateRo = lIndex[k]; //update the db using this datarow
...
} else if(inL){
var insertRo = lIndex[k]; //insert this row to the db
...
} else {
var deleteRo = rIndex[k]; //delete this row from the db
...
}
}
查看工作示例