R: 如何读取带有 data.table::fread 的 CSV 文件，该文件的逗号为小数点，点为千位分隔符="."

Question

我得到了几个 CSV 文件，其中包含当地德语风格的数字，即以逗号作为小数点分隔符，点作为千位分隔符，例如10.380,45。 CSV 文件中的值以“;”分隔。这些文件还包含来自类字符、日期、日期和时间以及逻辑的列。

read.table 函数的问题是，您可以使用 dec="," 指定小数点分隔符，但不能指定千点分隔符。（如有错误，请指正）

我知道预处理是一种解决方法，但我想以某种方式编写我的代码，其他人可以在没有我的情况下使用它。

我找到了一种使用 read.csv2 以我想要的方式读取 CSV 文件的方法，方法是设置我自己的类，如以下示例所示。基于

# Create test example
df_test_write <- cbind.data.frame(c("a","b","c","d","e","f","g","h","i","j",rep("k",times=200)),
                            c("5.200,39","250,36","1.000.258,25","3,58","5,55","10.550,00","10.333,00","80,33","20.500.000,00","10,00",rep("3.133,33",times=200)),
                            c("25.03.2015","28.04.2015","03.05.2016","08.08.2016","08.08.2016","08.08.2016","08.08.2016","08.08.2016","08.08.2016","08.08.2016",rep("08.08.2016",times=200)),
                            stringsAsFactors=FALSE)
colnames(df_test_write) <- c("col_text","col_num","col_date")

# write test csv
write.csv2(df_test_write,file="Test.csv",quote=FALSE,row.names=FALSE)

#### read with read.csv2 ####

# First, define your own class

#define your own numeric class
setClass('myNum')
#define conversion
setAs("character","myNum", function(from) as.numeric(gsub(",","\.",gsub("\.","",from))))

# own date class
library(lubridate)
setClass('myDate')
setAs("character","myDate",function(from) dmy(from))

# Read the csv file, in colClasses the columns class can be defined
df_test_readcsv <- read.csv2(paste0(getwd(),"/Test.csv"),
                       stringsAsFactors = FALSE,
                       colClasses = c(
                         col_text = "character",
                         col_num = "myNum",
                         col_date = "myDate"
                       )
                )

我现在的问题是，不同的数据集最多有 200 列和 350000 行。使用上面的解决方案，我需要 40 到 60 秒来加载一个 CSV 文件，我想加快速度。

通过我的研究，我从 data.table 包中找到 fread()，这真的很快。加载 CSV 文件大约需要 3 到 5 秒。

遗憾的是，也无法指定千位分隔符。所以我尝试将我的解决方案与 colClasses 一起使用，但似乎存在一个问题，即您不能将个人类与 fread https://github.com/Rdatatable/data.table/issues/491

一起使用

另请参阅我的以下测试代码：

##### read with fread ####
library(data.table)

# Test without colclasses
df_test_readfread1 <- fread(paste0(getwd(),"/Test.csv"),
                            stringsAsFactors = FALSE,
                            dec = ",",
                            sep=";",
                            verbose=TRUE)
str(df_test_readfread1)

# PROBLEM: In my real dataset it turns the number into an numeric column, 
# unforunately it sees the "." as decimal separator, so it turns e.g. 10.550, 
# into 10.5
# Here it keeps everything as character

# Test with colclasses
df_test_readfread2 <- fread(paste0(getwd(),"/Test.csv"),
                            stringsAsFactors = FALSE,
                            colClasses = c(
                              col_text = "character",
                              col_num = "myNum",
                              col_date = "myDate"
                            ),
                            sep=";",
                            verbose=TRUE)
str(df_test_readfread2)

# Keeps everything as character

所以我的问题是：有没有办法用 fread 读取数值为 10.380,45 的 CSV 文件？

（或者：读取具有此类数值的 CSV 文件的最快方法是什么？）

Answer 1

也许先删除所有逗号。

filepath<-paste0(getwd(),"/Test.csv")
filestring<-readChar(filepath, file.info(filepath)$size)
filestring<-gsub('.','',filestring,fixed=TRUE)
fread(filestring)

Answer 2

我自己没用过package，不过是Hadley Wickham的，应该是个好东西

https://cran.r-project.org/web/packages/readr/readr.pdf

它应该处理语言环境：

locale(date_names = "en", date_format = "%AD", time_format = "%AT", decimal_mark = ".", grouping_mark = ",", tz = "UTC", encoding = "UTF-8", asciify = FALSE)

decimal_mark 和 grouping_mark 就是您要查找的内容

编辑 PhiSeu 表格：解决方案

感谢您的建议，这里有两个使用 readr 包中的 read_csv2() 的解决方案。对于我的 350000 行 CSV 文件，大约需要 8 秒，这比 read.csv2 解决方案快得多。（来自 hadley 和 RStudio 的另一个有用的软件包，谢谢）

library(readr)

# solution 1 with specified columns
df_test_readr <- read_csv2(paste0(getwd(),"/Test.csv"),
                           locale = locale("de"),
                           col_names = TRUE,
                           cols(
                             col_text = col_character(),
                             col_num = col_number(), # number is automatically regcognized through locale=("de")
                             col_date2 = col_date(format ="%d.%m.%Y") # Date specification
                           )
                           )

# solution 2 with overall definition of date format
df_test_readr <- read_csv2(paste0(getwd(),"/Test.csv"),
                           locale = locale("de",date_format = "%d.%m.%Y"), # specifies the date format for the whole file
                           col_names = TRUE
)

R: 如何读取带有 data.table::fread 的 CSV 文件，该文件的逗号为小数点，点为千位分隔符="."

R: How can I read a CSV file with data.table::fread, that has a comma as decimal and point as thousand separator="."

r

data.table

fread

read.csv

readr