如何在 PHP 上同时通过 CURL 发送多个 POST 请求？（超过 2.000 个请求）

Question

好吧，我需要发送多个 http post 请求（超过 2500）并通过 Curl（php 或命令）获得结果，但我没有知道什么是最好和更好的发送方式..

在我发送的查询中，我还必须发送具有大约 150,000 和字符（约）的 "javax.faces.ViewState" 参数...

我有一个示例代码可以理解并查看它是如何工作的：

<?php
// i need send especific data via post
// and this data have a combination between internal id and year(from 2005 to current[2015])
$ids            =   array("988","992","6","989","993","14","26","948","949","950","951","952","27","34","386","953","954","955","956","47","51","53","927","928","929","930","931","933","932","88","94","103","660","1045","1046","1047","1049","1050","1051","1052","1053","1054","1055","1056","1057","1058","1059","1060","1061","1062","1048","114","119","1063","1064","1065","1067","1068","1069","1070","1097","1151","1150","1071","127","132","140","957","959","960","961","962","963","964","965","966","967","968","958","150","151","152","1072","1073","1074","1093","157","158","159","160","188","189","190","195","385","1075","1076","1077","1078","1079","1080","1081","1082","1083","1094","193","1152","1325","1326","206","207","209","214","216","934","935","936","937","938","939","940","941","942","943","944","946","947","223","225","226","227","1084","1085","1086","1087","1088","1095","1251","240","241","244","245","659","662","1089","1090","1091","1092","1096","1328","1013","248","249","250","990","994","996","257","258","991","995","1220","1221","1222","1223","1224","1225","1226","1227","1228","1232","1233","1235","1244","1245","1246","1247","1248","1250","1321","1229","1230","1231","1234","1236","1237","1238","1239","1240","1249","1320","1322","1323","1355");
$startYear      =   2005;
$currentYear    =   intval(date("Y"));
// this is "javax.faces.ViewState" a param with between 50.000 to 150.000 characters
$javaxFacesViewState = "H4sIAAAAAAAAAOy9CWAcSXkv3josWT5l+dxb7OXdxZYszSFpvSyMRrI09ujYmZGMvYBozbQ0bc1Mj3t6dJh/uF5CDkKAAAkECBAWSAjkgBCW5WaBQMJNSEhCwhneIy8hLwTCJnmw//qqr+qjprur24t3nw3b6qnjV1Vffd9XX3119Dv+hdvWkLm7JHl1gL/Ab8YvNFYG+Hq9IhZ5RZRqAwVZEPKK3CwqTVmYlUrCC1/6r69+eOWW7bs4brN+ieM6Oe4WS+6iVK1LNaGmDKRQ0KIobOQkSeF6..... ... ...";

// So, i dont have more server, i have only one, so i think use a proxy list over 50 ips
$proxyList      =   array(
    "xxx.xx.x.x:8080",
    "xxx.xx.x.x:2353",
    "xxx.xx.x.x:80",
    "xxx.xx.x.x:434",
    //...
    //...
);
echo "<ul>";
$index = 1;
for ($i = 0; $i < count($ids); $i++) {
    echo "<li>";
        echo "<strong>ID: <em>".$ids[$i]."</em></strong>";
        echo "<ol>";
        for ($y = $startYear; $y <= $currentYear; $y++) {
            echo "<li value='$index'>Year: $y; curl command:<br /><pre><code>curl --proxy ".$proxyList[array_rand($proxyList)]." http://example.com/information.jsp --data=id=".$ids[$i]."&year=$y$y&javax.faces.ViewState=$javaxFacesViewState...</code></pre></li>";
            $index++;
        }
        echo "</ol>";
    echo "</li>";
}
echo "</ul>";
echo "<h1>Total request: ".number_format($index,0)."</h1>";
?>

输出类似于：

编号：988
1. 年份：2005； curl 命令：curl --proxy xxx.xx.x.x:455 http://example.com/information.jsp --data=id=12&year=2005&...
2. 年份：2005； curl 命令：curl --proxy xxx.xx.x.x:80 http://example.com/information.jsp --data=id=23&year=2005&...
3. 年份：2005； curl 命令：curl --proxy xxx.xx.x.x:8080 http://example.com/information.jsp --data=id=4556&year=2005&...
4. 年份：2005； curl 命令：curl --proxy xxx.xx.x.x:235 http://example.com/information.jsp --data=id=34&year=2005&...
5. ...
6. ...

请求总数：2,135

所以，我需要在最短的时间内发送多个 post 请求...最佳和更好的方法是什么？

我的服务器是 (MT) - DV LVL 1:

2GB 内存
2TB 带宽
CentOS 6

少/proc/cpuinfo

[root ~]# less /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 62
model name      : Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
stepping        : 4
microcode       : 1064
cpu MHz         : 2094.833
cache size      : 15360 KB
physical id     : 0
siblings        : 12
core id         : 0
cpu cores       : 6
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf cpuid_faulting pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt
bogomips        : 4189.66
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 62
model name      : Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
stepping        : 4

PHP: 5.4.13
卷曲：7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 基本 ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
阿帕奇：2.2.15 (Unix)
iptables: 1.4.7

谢谢

Answer 1

与curl multi requests。原理：

$ids            =   array("988","992","6","989","993","14","26","948","949","950","951","952","27","34","386","953","954","955","956","47","51","53","927","928","929","930","931","933","932","88","94","103","660","1045","1046","1047","1049","1050","1051","1052","1053","1054","1055","1056","1057","1058","1059","1060","1061","1062","1048","114","119","1063","1064","1065","1067","1068","1069","1070","1097","1151","1150","1071","127","132","140","957","959","960","961","962","963","964","965","966","967","968","958","150","151","152","1072","1073","1074","1093","157","158","159","160","188","189","190","195","385","1075","1076","1077","1078","1079","1080","1081","1082","1083","1094","193","1152","1325","1326","206","207","209","214","216","934","935","936","937","938","939","940","941","942","943","944","946","947","223","225","226","227","1084","1085","1086","1087","1088","1095","1251","240","241","244","245","659","662","1089","1090","1091","1092","1096","1328","1013","248","249","250","990","994","996","257","258","991","995","1220","1221","1222","1223","1224","1225","1226","1227","1228","1232","1233","1235","1244","1245","1246","1247","1248","1250","1321","1229","1230","1231","1234","1236","1237","1238","1239","1240","1249","1320","1322","1323","1355");

// this is "javax.faces.ViewState" a param with between 50.000 to 150.000 characters
$javaxFacesViewState = "H4sIAAAAAAAAAOy9CWAcSXkv3josWT5l+dxb7OXdxZYszSFpvSyMRrI09ujYmZGMvYBozbQ0bc1Mj3t6dJh/uF5CDkKAAAkECBAWSAjkgBCW5WaBQMJNSEhCwhneIy8hLwTCJnmw//qqr+qjprur24t3nw3b6qnjV1Vffd9XX3119Dv+hdvWkLm7JHl1gL/Ab8YvNFYG+Hq9IhZ5RZRqAwVZEPKK3CwqTVmYlUrCC1/6r69+eOWW7bs4brN+ieM6Oe4WS+6iVK1LNaGmDKRQ0KIobOQkSeF6..... ... ...";

// So, i dont have more server, i have only one, so i think use a proxy list over 50 ips
$proxyList      =   array(
  "xxx.xx.x.x:8080",
  "xxx.xx.x.x:2353",
  "xxx.xx.x.x:80",
  "xxx.xx.x.x:434",
  //...
  //...
);

// Processing... 

$total_requests = 0;

$step_urls_count = 10; // 20... 
$curl_batch_urls = array_chunk($ids,$step_urls_count);

// sending by 10 requests with curl_multi_exec  ( e.g. 1..10)
foreach($curl_batch_urls as $batch){

  $master = curl_multi_init();
  $curl_arr = array();

  // common curl options 
  $curl_options = array(
    'CURLOPT_RETURNTRANSFER'=> true,
   /* .....
      other curl options 
      .....
   */);

  // generate curl instances 
  foreach($batch as $url_id){
     $ch = curl_init();

     // set unique url for each ID
     $options['CURLOPT_URL'] = "http://example.com/information.jsp --data=id=".$url_id."&javax.faces.ViewState=".$javaxFacesViewState;

     // random proxy
     $rand_key = array_rand($proxyList);
     $options['CURLOPT_PROXY'] = $proxyList[$rand_key];

     curl_setopt_array($ch,$options);
     curl_multi_add_handle($master, $ch);
  }

   $running = null;
   do {

    // performing curl-handle batch
    while(($execrun = curl_multi_exec($master, $running)) == CURLM_CALL_MULTI_PERFORM);
    if($execrun != CURLM_OK)
        break;

    //  checking each response 
    while($response = curl_multi_info_read($master)) {
        $info = curl_getinfo($response['handle']);
        if ($info['http_code'] == 200)  {

            // $output - it`s response for each request
            $output = curl_multi_getcontent($response['handle']);
            var_dump($output);
        } else {
            // Error!
        }
    }
  } while ($running);

  curl_multi_close($master);     

} // there go to next loop step ( 10..20 ids )

注意: 每个循环步骤需要玩 $step_urls_count 。此外，增加代理列表 (100-500) 并在设置选项之前检查每个代理的可用性。希望，它会对你有用。

如何在 PHP 上同时通过 CURL 发送多个 POST 请求？（超过 2.000 个请求）

How to send multi POST request via CURL at same time on PHP? (over 2.000 requests)

php

apache

curl

httprequest

centos6

如何在 PHP 上同时通过 CURL 发送多个 POST 请求？ （超过 2.000 个请求）

How to send multi POST request via CURL at same time on PHP? (over 2.000 requests)

php

apache

curl

httprequest

centos6

如何在 PHP 上同时通过 CURL 发送多个 POST 请求？（超过 2.000 个请求）