fread() 运行 时间比报告的速度要长
fread() run time is taking longer than reported speed
我正在尝试将一个大文件读入 EC2 实例上的 R。然而，我已经经历了 运行 次，这远远超过 fread 在读取某些数据后报告的时间量。
例如，在下面，当我只读入我的 csv 文件的第一行数据时，我有 fread 的 verbose=TRUE 输出。如您所见，报告的 运行 时间比实际 运行 时间短得多。你知道为什么会这样吗？有什么办法可以加快这个过程，使其更符合读入数据后 fread 报告的运行时间？
> start_time <- Sys.time()
> fread(file_name_1, nrows=1, verbose=TRUE)
Input contains no \n. Taking this to be a filename to open
File opened, filesize is 68.770914 GB.
Memory mapping ... ok
Detected eol as \n only (no \r afterwards), the UNIX and Mac standard.
Positioned on line 1 after skip or autostart
This line is the autostart and not blank so searching up for the last non-blank ... line 1
Detecting sep ... ','
Detected 55 columns. Longest stretch was from line 1 to line 30
Starting data input on line 1 (either column names or first row of data). First 10 characters: bank_num,b
All the fields on line 1 are character fields. Treating as the column names.
nrow set to nrows passed in (1)
Type codes (point 0): 1114434134111034444411333333333333333333333333333311111
Type codes: 1114434134111034444411333333333333333333333333333311111 (after applying colClasses and integer64)
Type codes: 1114434134111034444411333333333333333333333333333311111 (after applying drop or select (if supplied)
Allocating 55 column slots (55 - 0 dropped)
Read 1 rows and 55 (of 55) columns from 68.771 GB file in 00:00:27
Read 1 rows. Exactly what was estimated and allocated up front
26.480s (100%) Memory map (rerun may be quicker)
0.000s ( 0%) sep and header detection
0.000s ( 0%) Count rows (wc -l)
0.000s ( 0%) Column type detection (100 rows at 10 points)
0.000s ( 0%) Allocation of 1x55 result (xMB) in RAM
0.000s ( 0%) Reading data
0.000s ( 0%) Allocation for type bumps (if any), including gc time if triggered
0.000s ( 0%) Coercing data already read in type bumps (if any)
0.000s ( 0%) Changing na.strings to NA
26.480s Total
> end_time <- Sys.time()
> end_time - start_time
Time difference of 9.695263 mins
请始终注明版本号；例如sessionInfo() 的输出。但我可以告诉您，您可能使用的是 CRAN 版本。
在 Stack Overflow 上询问之前，请始终检查 NEWS。
项目 3（在许多其他 fread 改进中）：
Memory maps lazily; e.g. reading just the first 10 rows with nrow=10 is 12s down to 0.01s from cold for a 9GB file. Large files close to your RAM limit may work more reliably too. The progress meter will commence sooner and more consistently.
使用 this install command . You wrote EC2, so presumably Linux but any Windows users can use the Windows.zip from dev 即可轻松试用 dev 的最新版本，无需任何工具。
既然你有一个 68GB 的 csv，你肯定会从 data.table v1.10.5+ 中受益匪浅。请在这里更新你的进展情况。
我正在尝试将一个大文件读入 EC2 实例上的 R。然而，我已经经历了 运行 次，这远远超过 fread 在读取某些数据后报告的时间量。
例如，在下面，当我只读入我的 csv 文件的第一行数据时，我有 fread 的 verbose=TRUE 输出。如您所见，报告的 运行 时间比实际 运行 时间短得多。你知道为什么会这样吗？有什么办法可以加快这个过程，使其更符合读入数据后 fread 报告的运行时间？
> start_time <- Sys.time()
> fread(file_name_1, nrows=1, verbose=TRUE)
Input contains no \n. Taking this to be a filename to open
File opened, filesize is 68.770914 GB.
Memory mapping ... ok
Detected eol as \n only (no \r afterwards), the UNIX and Mac standard.
Positioned on line 1 after skip or autostart
This line is the autostart and not blank so searching up for the last non-blank ... line 1
Detecting sep ... ','
Detected 55 columns. Longest stretch was from line 1 to line 30
Starting data input on line 1 (either column names or first row of data). First 10 characters: bank_num,b
All the fields on line 1 are character fields. Treating as the column names.
nrow set to nrows passed in (1)
Type codes (point 0): 1114434134111034444411333333333333333333333333333311111
Type codes: 1114434134111034444411333333333333333333333333333311111 (after applying colClasses and integer64)
Type codes: 1114434134111034444411333333333333333333333333333311111 (after applying drop or select (if supplied)
Allocating 55 column slots (55 - 0 dropped)
Read 1 rows and 55 (of 55) columns from 68.771 GB file in 00:00:27
Read 1 rows. Exactly what was estimated and allocated up front
26.480s (100%) Memory map (rerun may be quicker)
0.000s ( 0%) sep and header detection
0.000s ( 0%) Count rows (wc -l)
0.000s ( 0%) Column type detection (100 rows at 10 points)
0.000s ( 0%) Allocation of 1x55 result (xMB) in RAM
0.000s ( 0%) Reading data
0.000s ( 0%) Allocation for type bumps (if any), including gc time if triggered
0.000s ( 0%) Coercing data already read in type bumps (if any)
0.000s ( 0%) Changing na.strings to NA
26.480s Total
> end_time <- Sys.time()
> end_time - start_time
Time difference of 9.695263 mins
请始终注明版本号；例如sessionInfo() 的输出。但我可以告诉您，您可能使用的是 CRAN 版本。
在 Stack Overflow 上询问之前，请始终检查 NEWS。
项目 3（在许多其他 fread 改进中）：
Memory maps lazily; e.g. reading just the first 10 rows with
nrow=10is 12s down to 0.01s from cold for a 9GB file. Large files close to your RAM limit may work more reliably too. The progress meter will commence sooner and more consistently.
使用 this install command . You wrote EC2, so presumably Linux but any Windows users can use the Windows.zip from dev 即可轻松试用 dev 的最新版本，无需任何工具。
既然你有一个 68GB 的 csv，你肯定会从 data.table v1.10.5+ 中受益匪浅。请在这里更新你的进展情况。