用R从NCBI基因数据库获取数据

Question

Rentrez 套餐

根据 this manual，我在 Linux (Ubuntu 20.04.2) 的实验室计算机上发现了 RStudio（版本 1.1.442）中的 rentrez 包。但是，稍后当我想在 Windows 8 Pro (RStudio 2021.09.0)

中运行我的笔记本电脑上使用相同的代码时

library (rentrez)
entrez_dbs()
entrez_db_searchable("gene")
#res <- entrez_search (db = "gene", term = "(Vibrio[Organism] OR vibrio[All Fields]) AND (16s[All Fields]) AND (rna[All Fields]) AND (owensii[All Fields] OR navarrensis[All Fields])", retmax = 500, use_history = TRUE)

即使在关闭会话或重新安装 rentrez 程序包后，我也无法摆脱这个错误

Error in curl::curl_fetch_memory(url, handle = handle) : schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed).

这是我面临的主要问题。

RSelenium 包

后来我决定处理包含有关 genes and their sequences in FASTA format 修改我以前使用的代码的详细信息的页面。它使用 rvest 和 rselenium 包，结果很完美。

# Specifying a webpage

url <- "https://www.ncbi.nlm.nih.gov/gene/66940694"    # the last 9 numbers is gene id

library(rvest)
library(RSelenium)

# Opening a browser

driver <- rsDriver(browser = c("firefox"))
remDr <- driver[["client"]]
remDr$errorDetails
remDr$navigate(url)

# Clicked outside in an empty space next to the FASTA button and copied a full xPath (redirecting to a FASTA data containing webpage)

remDr$findElement(using = "xpath", value = '/html/body/div[1]/div[1]/form/div[1]/div[5]/div/div[6]/div[2]/div[3]/div/div/div[3]/div/p/a[2]')$clickElement()

webElem <- remDr$findElement("css", "body")

#scrolling to the end of a webpage: left it from the old code for the case of a long gene

for (i in 1:5){
  Sys.sleep(2)
  webElem$sendKeysToElement(list(key = "end"))

# Let's get gene FASTA, for example

page <- read_html(remDr$getPageSource()[[1]])
fasta <- page %>%
html_nodes('pre') %>%
  html_text()
print(fasta)

Output: ">NZ_QKKR01000022.1:c3037-151 Vibrio paracholerae strain 2016V-1111 2016V-1111_ori_contig_18, whole genome shotgun sequence\nGGT...

该代码可以很好地获取有关该基因的其他详细信息，例如其登录号、位置、生物体等。

多个基因 ID 的循环过程

后来我尝试更改代码以同时获取几个基因 ID 的相同信息我来到这里是为了我的另一个项目。

# Specifying a list of gene IDs

res_id <- c('57838769','61919208','66940694')
dt <- res_id    # <lapply> looping function refused to work if an argument had a different name rather than <dt>
  
driver <- rsDriver(browser = c("firefox"))  
remDr <- driver[["client"]]

## Writing a function of GET_FASTA dependent on GENE_ID (x)

get_fasta <- function(x){
  link = paste0('https://www.ncbi.nlm.nih.gov/gene/',x)
  remDr$navigate(link)
  remDr$findElement(using = "xpath", value = '/html/body/div[1]/div[1]/form/div[1]/div[5]/div/div[6]/div[2]/div[3]/div/div/div[3]/div/p/a[2]')$clickElement()

...下面有续篇但是这里出现错误，说找不到之前成功使用的同一个xPath

Error: Summary: NoSuchElement Detail: An element could not be located on the page using the given search parameters. class: org.openqa.selenium.NoSuchElementException Further Details: run errorDetails method

我试图删除 /a[2] 以获得 xPath 末尾的 /html/.../p，因为它在初始代码中工作，但稍后再次出现错误。

  webElem <- remDr$findElement("css", "body")

  for (i in 1:5){
    Sys.sleep(2)
    webElem$sendKeysToElement(list(key = "end"))
  } 

 # Addressing selectors of FASTA on the website
  
  fasta <- remDr$getPageSource()[[1]] %>% 
    read_html() %>%
    html_nodes('pre') %>%
    html_text()
  fasta
  return(fasta)
}

## Writing a function of GET_ACC_NUM dependent on GENE_ID (x)

get_acc_num <- function(x){
  link = paste0( 'https://www.ncbi.nlm.nih.gov/gene/', x)
  remDr$navigate(link)
  remDr$findElement(using = "xpath", value = '/html/body/div[1]/div[1]/form/div[1]/div[5]/div/div[6]/div[2]/div[3]/div/div/div[3]/div/p')$clickElement()
  webElem <- remDr$findElement("css", "body")

  for (i in 1:5){
    Sys.sleep(2)
    webElem$sendKeysToElement(list(key = "end"))
  } 
  
  # Addressing selectors of ACC_NUM on the website
  
  acc_num <- remDr$getPageSource()[[1]] %>% 
    read_html() %>%
    html_nodes('.itemid') %>%
    html_text() %>%
    str_sub(start= -17)
  acc_num
  return(acc_num)
}

## Collecting all FUNCTION into tibble

get_data_table <- function(x){

      # Extract the Basic information from the HTML
      fasta <- get_fasta(x)
      acc_num <- get_acc_num(x)

      # Combine into a tibble
      combined_data <- tibble( Acc_Number = acc_num,
                               FASTA = fasta)
}

## Running FUNCTION for all x

df <- lapply(dt, get_data_table)

head(df)

我也试过写代码

仅适用于 rvest、
用for (i in res_id) {}、
使用if () {} else {}

/html/.../p/a[2]

.../p

但结果更加令人困惑。

我在处理此类任务时正在学习 R 编码，因此欢迎任何建议和批评。

Answer 1

节点 pre 无效。我们必须在 class 或 'id` 等

中寻找值

webElem$sendKeysToElement(list(key = "end") 您不需要此命令，因为没有必要滚动页面。

下面是获取基因序列的代码。

首先我们必须通过rvest

获得基因序列的链接

library(rvest)
library(dplyr)
res_id <- c('57838769','61919208','66940694')

link = vector()
for(i in res_id){
  url = paste0('https://www.ncbi.nlm.nih.gov/gene/', i)
  df = url %>%
    read_html() %>% 
    html_node('.note-link') 
  
  link1 = xml_attrs(xml_child(df, 3))[["href"]]
  link1 = paste0('https://www.ncbi.nlm.nih.gov', link1)
  link = rbind(link, link1)
}

link1 "https://www.ncbi.nlm.nih.gov/nuccore/NZ_ADAF01000001.1?report=fasta&from=257558&to=260444"       
link1 "https://www.ncbi.nlm.nih.gov/nuccore/NZ_VARQ01000103.1?report=fasta&from=64&to=2616&strand=true" 
link1 "https://www.ncbi.nlm.nih.gov/nuccore/NZ_QKKR01000022.1?report=fasta&from=151&to=3037&strand=true"

获得链接后，我们将获得我们通过 RSelenium 完成的基因序列。我试着用 rvest 来做，但无法得到序列。

启动浏览器

library(RSelenium)
driver = rsDriver(browser = c("firefox"))
remDr <- driver[["client"]]

获取序列的函数

get_seq = function(link){
  remDr$navigate(link)
  Sys.sleep(5)
  df = remDr$getPageSource()[[1]] %>% 
    read_html() %>% 
    html_nodes(xpath = '//*[@id="viewercontent1"]') %>% 
    html_text()
  return(df)
}

df = lapply(link, get_seq)

现在我们有了包含所有信息的列表 df。

用R从NCBI基因数据库获取数据

Obtaining data from NCBI gene database with R

html

r

rselenium

ncbi

rentrez

Rentrez 套餐

RSelenium 包

多个基因 ID 的循环过程