从另一个文件读取 Urls 并抓取数据- Bash

Question

我想从 URL.txt 中获取 URL，然后将它们附加到基 URL https://www.mcdelivery.com.pk/pk/browse/menu.html 的末尾，它存在于另一个文件中 menu.sh

Url.text 文件包含

?daypartId=1&amp;catId=1
?daypartId=1&amp;catId=2
?daypartId=1&amp;catId=11
?daypartId=1&amp;catId=10
?daypartId=1&amp;catId=6
?daypartId=1&amp;catId=4
?daypartId=1&amp;catId=14
?daypartId=1&amp;catId=5
?daypartId=1&amp;catId=3
?daypartId=1&amp;catId=8

我想像 https://www.mcdelivery.com.pk/pk/browse/menu.html?daypartId=1&catId=11 一样附加 url 基础 url + url 来自 URL.txt 文件

我想出了这段代码，但问题是我只能从第一页获取价格，并且它会不断重复同一页的值，直到循环结束。

ARRAY=()
while read -r LINE
do
ARRAY+=("$LINE")
done < URL.txt
for LINE in "${ARRAY[@]}"
do   
echo $LINE
curl https://www.mcdelivery.com.pk/pk/browse/menu.html$LINE | grep -o '<span class="starting-price">.*</span>' | sed 's/<[^>]\+>//g' >> price.txt 
done

我得到的输出

Rs 398
Rs 487
Rs 841
Rs 752
Rs 398
Rs 398
Rs 487
Rs 841
Rs 752
....

我想从每个页面获取价格并将它们存储到 price.txt

Answer 1

#!/bin/bash
curl -sL https://www.mcdelivery.com.pk/pk/browse/menu.html | grep -o '<li class="secondary-menu-item ">.*</li>' | sed 's/href=/\nhref=/g' | \
grep 'href=\"' | \
sed 's/.*href="//g;s/".*//g' > URL.txt
sed -i 's/amp;//' URL.txt

ARRAY=()
while read -r LINE
do
    ARRAY+=("$LINE")
done < URL.txt

for LINE in "${ARRAY[@]}"
do    
    echo $LINE
    curl https://www.mcdelivery.com.pk/pk/browse/menu.html"$LINE" | grep -o '<h5 class="product-title">.*</h5>' | sed 's/<[^>]\+>//g' >> name.txt
    curl https://www.mcdelivery.com.pk/pk/browse/menu.html"$LINE" | grep -o '<span class="starting-price">.*</span>' | sed 's/<[^>]\+>//g' >> price.txt 
done

这就是我的问题的答案。谢谢大家的帮助！！！

Answer 2

Please don't use regular expressions to parse html. Use a true html-parser / web-scraper like xidel 代替。
事实上，根本不需要 Bash-script。 xidel可以为所欲为。

解析html of "★What's New★"-menu-item and string-join price + product-name:

$ xidel -s "https://www.mcdelivery.com.pk/pk/browse/menu.html?daypartId=1&catId=1" -e '
  //div[ends-with(@class,"panel-product")]/join(
    (.//span[@class="starting-price"],.//h5),
    " - "
  )
'
Rs 288 - Cappuccino with Milk Chocolate Cookie
Rs 288 - Cappuccino with Double Chocolate Cookie
Rs 288 - Latte with Milk Chocolate Cookie
[...]
Rs 239 - Salted Caramel Shake

列出所有 menu-item 和 string-join url + 标题：

$ xidel -s https://www.mcdelivery.com.pk/pk/browse/menu.html -e '
  //ul[@class="secondary-menu"]//a/join((resolve-uri(@href),span)," - ")
'
https://www.mcdelivery.com.pk/pk/browse/menu.html?daypartId=1&catId=12 - Deals
https://www.mcdelivery.com.pk/pk/browse/menu.html?daypartId=1&catId=1 - ★What's New★
https://www.mcdelivery.com.pk/pk/browse/menu.html?daypartId=1&catId=2 - Ala carte & Value Meals
[...]
https://www.mcdelivery.com.pk/pk/browse/menu.html?daypartId=1&catId=8 - Snack Time

对于每一个menu-item,string-joinurl+标题，打开url/解析html和string-join价格+product-name:

$ xidel -s https://www.mcdelivery.com.pk/pk/browse/menu.html -e '
  //ul[@class="secondary-menu"]//a/(
    join((resolve-uri(@href),span)," - "),
    doc(@href)//div[ends-with(@class,"panel-product")]/join(
      (.//span[@class="starting-price"],.//h5),
      " - "
    )
  )
'
https://www.mcdelivery.com.pk/pk/browse/menu.html?daypartId=1&catId=12 - Deals
Rs 487 - Grand Chicken Spicy with Drink
Rs 398 - Big Mac + Regular Drink
https://www.mcdelivery.com.pk/pk/browse/menu.html?daypartId=1&catId=1 - ★What's New★
Rs 288 - Cappuccino with Milk Chocolate Cookie
Rs 288 - Cappuccino with Double Chocolate Cookie
Rs 288 - Latte with Milk Chocolate Cookie
Rs 288 - Latte with Double Chocolate Cookie
Rs 159 - McFizz Guava
Rs 195 - Date Pie
Rs 416 - Spicy McCrispy Deluxe - Regular Meal
Rs 416 - McChicken - Regular Meal
Rs 239 - Curly Fries
Rs 239 - Salted Caramel Shake
https://www.mcdelivery.com.pk/pk/browse/menu.html?daypartId=1&catId=2 - Ala carte & Value Meals
Rs 257 - Chicken Burger with Cheese
Rs 265 - Value McArabia Chicken
Rs 265 - Mini McRoyale
[...]
https://www.mcdelivery.com.pk/pk/browse/menu.html?daypartId=1&catId=8 - Snack Time
Rs 301 - Spicy Chicken Burger
Rs 301 - 4pcs McNuggets
Rs 301 - Fries & Drink
Rs 150 - Apple Pie with Tea

从另一个文件读取 Urls 并抓取数据- Bash

Read Urls from another file and Scrape the data- Bash

arrays

bash

readfile

web-scraping

web-scraping-language