HtmlUnit 没有创建 HtmlPage 对象
HtmlUnit not creating HtmlPage object
我是 HtmlUnit 的新手，我正在尝试抓取一个使用 Javascript 编辑代码的网站。我听说 HtmlUnit 是最好的方法，因为它 returns 使用无头浏览器的 final 代码。
然而，正如您将看到的那样，我什至无法在创建 HtmlPage 对象时不抛出一个巨大且无法理解的异常（至少考虑到我对 HtmlUnit 几乎没有经验）。
这是我的代码：
import com.gargoylesoftware.htmlunit.*;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
public class Main {
public static void main(String[] args) {
Main scraper = new Main();
scraper.testingGargoyle();
}
private void testingGargoyle() {
String myUrl = "https://www.wearvr.com/#game_id=game_4";
WebClient webClient = new WebClient();
try {
HtmlPage myPage = ((HtmlPage) webClient.getPage(myUrl));
} catch (FailingHttpStatusCodeException | IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
这是抛出的异常：
Apr 30, 2015 5:43:50 PM com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl notify
WARNING: Obsolete content type encountered: 'application/x-javascript'.
Apr 30, 2015 5:43:50 PM com.gargoylesoftware.htmlunit.javascript.StrictErrorReporter runtimeError
SEVERE: runtimeError: message=[The data necessary to complete this operation is not yet available.] sourceName=[https://load.sumome.com/] line=[1] lineSource=[null] lineOffset=[0]
Exception in thread "main" ======= EXCEPTION START ========
EcmaError: lineNumber=[19] column=[0] lineSource=[<no source>] name=[TypeError] sourceName=[https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js] message=[TypeError: Cannot find function bind in object function (e, n, r) {...}. (https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js#19)]
com.gargoylesoftware.htmlunit.ScriptException: TypeError: Cannot find function bind in object function (e, n, r) {...}. (https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js#19)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:847)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:620)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:513)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:733)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadExternalJavaScriptFile(HtmlPage.java:1096)
at com.gargoylesoftware.htmlunit.html.HtmlScript.executeScriptIfNeeded(HtmlScript.java:395)
at com.gargoylesoftware.htmlunit.html.HtmlScript.execute(HtmlScript.java:270)
at com.gargoylesoftware.htmlunit.html.HtmlScript.onAllChildrenAddedToPage(HtmlScript.java:290)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:793)
at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:751)
at org.cyberneko.html.HTMLTagBalancer.callEndElement(HTMLTagBalancer.java:1170)
at org.cyberneko.html.HTMLTagBalancer.endElement(HTMLTagBalancer.java:1072)
at org.cyberneko.html.filters.DefaultFilter.endElement(DefaultFilter.java:206)
at org.cyberneko.html.filters.NamespaceBinder.endElement(NamespaceBinder.java:330)
at org.cyberneko.html.HTMLScanner$ContentScanner.scanEndElement(HTMLScanner.java:3126)
at org.cyberneko.html.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2093)
at org.cyberneko.html.HTMLScanner.scanDocument(HTMLScanner.java:920)
at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:499)
at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:452)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.parse(HTMLParser.java:1017)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parse(HTMLParser.java:248)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parseHtml(HTMLParser.java:194)
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:268)
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:156)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:471)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:345)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:410)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:395)
at Main.testingGargoyle(Main.java:19)
at Main.main(Main.java:10)
Caused by: net.sourceforge.htmlunit.corejs.javascript.EcmaError: TypeError: Cannot find function bind in object function (e, n, r) {...}. (https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js#19)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3629)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3613)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3634)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError2(ScriptRuntime.java:3650)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.notFunctionError(ScriptRuntime.java:3714)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThisHelper(ScriptRuntime.java:2233)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThis(ScriptRuntime.java:2215)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1333)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:798)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:411)
at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:309)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3057)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.doRun(JavaScriptEngine.java:724)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:832)
... 31 more
Enclosed exception:
net.sourceforge.htmlunit.corejs.javascript.EcmaError: TypeError: Cannot find function bind in object function (e, n, r) {...}. (https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js#19)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3629)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3613)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3634)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError2(ScriptRuntime.java:3650)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.notFunctionError(ScriptRuntime.java:3714)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThisHelper(ScriptRuntime.java:2233)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThis(ScriptRuntime.java:2215)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1333)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:19)
at script.r(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:16)
at script.r(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:384)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:16)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:16)
at script.t(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:16)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:16)
at script.t(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:7)
at script.t(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:463)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:463)
at script.t(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
at script.t(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:798)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:411)
at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:309)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3057)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.doRun(JavaScriptEngine.java:724)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:832)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:620)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:513)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:733)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadExternalJavaScriptFile(HtmlPage.java:1096)
at com.gargoylesoftware.htmlunit.html.HtmlScript.executeScriptIfNeeded(HtmlScript.java:395)
at com.gargoylesoftware.htmlunit.html.HtmlScript.execute(HtmlScript.java:270)
at com.gargoylesoftware.htmlunit.html.HtmlScript.onAllChildrenAddedToPage(HtmlScript.java:290)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:793)
at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:751)
at org.cyberneko.html.HTMLTagBalancer.callEndElement(HTMLTagBalancer.java:1170)
at org.cyberneko.html.HTMLTagBalancer.endElement(HTMLTagBalancer.java:1072)
at org.cyberneko.html.filters.DefaultFilter.endElement(DefaultFilter.java:206)
at org.cyberneko.html.filters.NamespaceBinder.endElement(NamespaceBinder.java:330)
at org.cyberneko.html.HTMLScanner$ContentScanner.scanEndElement(HTMLScanner.java:3126)
at org.cyberneko.html.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2093)
at org.cyberneko.html.HTMLScanner.scanDocument(HTMLScanner.java:920)
at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:499)
at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:452)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.parse(HTMLParser.java:1017)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parse(HTMLParser.java:248)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parseHtml(HTMLParser.java:194)
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:268)
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:156)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:471)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:345)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:410)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:395)
at Main.testingGargoyle(Main.java:19)
at Main.main(Main.java:10)
======= EXCEPTION END ========
我告诉过你它很大。我怎样才能解决这个问题并获得该页面的最终来源以便进行抓取？
提前致谢！
尝试使用不同的浏览器，例如：
String myUrl = "https://www.wearvr.com/#game_id=game_4";
try (WebClient webClient = new WebClient(BrowserVersion.CHROME)) {
HtmlPage myPage = ((HtmlPage) webClient.getPage(myUrl));
System.out.println(myPage.asXml());
} catch (FailingHttpStatusCodeException | IOException e) {
e.printStackTrace();
}
不过也有可能是IE8模拟的bug
抛出异常有多种原因，错误 html，脚本页面错误，未找到 css 等资源，脚本文件或图像文件（例如 <img src="bla.gif"> <- bla.gif not found HTML404)
所以我们使用这些选项来保持 html 导航而不会首先停止 error/problem 我们使用 :
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
您还可以实施空 类 以停止 htmlUnity 在控制台上详细介绍 css/javaScript 错误：
webClient.setCssErrorHandler(new SilentCssErrorHandler());
webClient.setJavaScriptErrorListener(new JavaScriptErrorListener(){});
小样本测试用例：
@Test
public void TestCall() throws FailingHttpStatusCodeException, MalformedURLException, IOException {
WebClient webClient = new WebClient(BrowserVersion.CHROME);
webClient.getOptions().setUseInsecureSSL(true); //ignore ssl certificate
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
String url = "https://www.wearvr.com/#game_id=game_4";
HtmlPage myPage = webClient.getPage(url);
webClient.waitForBackgroundJavaScriptStartingBefore(200);
webClient.waitForBackgroundJavaScript(20000);
//do stuff on page ex: myPage.getElementById("main")
//myPage.asXml() <- tags and elements
System.out.println(myPage.asText());
}
我是 HtmlUnit 的新手，我正在尝试抓取一个使用 Javascript 编辑代码的网站。我听说 HtmlUnit 是最好的方法，因为它 returns 使用无头浏览器的 final 代码。
然而，正如您将看到的那样，我什至无法在创建 HtmlPage 对象时不抛出一个巨大且无法理解的异常（至少考虑到我对 HtmlUnit 几乎没有经验）。
这是我的代码：
import com.gargoylesoftware.htmlunit.*;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
public class Main {
public static void main(String[] args) {
Main scraper = new Main();
scraper.testingGargoyle();
}
private void testingGargoyle() {
String myUrl = "https://www.wearvr.com/#game_id=game_4";
WebClient webClient = new WebClient();
try {
HtmlPage myPage = ((HtmlPage) webClient.getPage(myUrl));
} catch (FailingHttpStatusCodeException | IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
这是抛出的异常：
Apr 30, 2015 5:43:50 PM com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl notify
WARNING: Obsolete content type encountered: 'application/x-javascript'.
Apr 30, 2015 5:43:50 PM com.gargoylesoftware.htmlunit.javascript.StrictErrorReporter runtimeError
SEVERE: runtimeError: message=[The data necessary to complete this operation is not yet available.] sourceName=[https://load.sumome.com/] line=[1] lineSource=[null] lineOffset=[0]
Exception in thread "main" ======= EXCEPTION START ========
EcmaError: lineNumber=[19] column=[0] lineSource=[<no source>] name=[TypeError] sourceName=[https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js] message=[TypeError: Cannot find function bind in object function (e, n, r) {...}. (https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js#19)]
com.gargoylesoftware.htmlunit.ScriptException: TypeError: Cannot find function bind in object function (e, n, r) {...}. (https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js#19)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:847)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:620)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:513)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:733)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadExternalJavaScriptFile(HtmlPage.java:1096)
at com.gargoylesoftware.htmlunit.html.HtmlScript.executeScriptIfNeeded(HtmlScript.java:395)
at com.gargoylesoftware.htmlunit.html.HtmlScript.execute(HtmlScript.java:270)
at com.gargoylesoftware.htmlunit.html.HtmlScript.onAllChildrenAddedToPage(HtmlScript.java:290)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:793)
at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:751)
at org.cyberneko.html.HTMLTagBalancer.callEndElement(HTMLTagBalancer.java:1170)
at org.cyberneko.html.HTMLTagBalancer.endElement(HTMLTagBalancer.java:1072)
at org.cyberneko.html.filters.DefaultFilter.endElement(DefaultFilter.java:206)
at org.cyberneko.html.filters.NamespaceBinder.endElement(NamespaceBinder.java:330)
at org.cyberneko.html.HTMLScanner$ContentScanner.scanEndElement(HTMLScanner.java:3126)
at org.cyberneko.html.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2093)
at org.cyberneko.html.HTMLScanner.scanDocument(HTMLScanner.java:920)
at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:499)
at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:452)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.parse(HTMLParser.java:1017)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parse(HTMLParser.java:248)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parseHtml(HTMLParser.java:194)
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:268)
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:156)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:471)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:345)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:410)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:395)
at Main.testingGargoyle(Main.java:19)
at Main.main(Main.java:10)
Caused by: net.sourceforge.htmlunit.corejs.javascript.EcmaError: TypeError: Cannot find function bind in object function (e, n, r) {...}. (https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js#19)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3629)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3613)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3634)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError2(ScriptRuntime.java:3650)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.notFunctionError(ScriptRuntime.java:3714)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThisHelper(ScriptRuntime.java:2233)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThis(ScriptRuntime.java:2215)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1333)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:798)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:411)
at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:309)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3057)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.doRun(JavaScriptEngine.java:724)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:832)
... 31 more
Enclosed exception:
net.sourceforge.htmlunit.corejs.javascript.EcmaError: TypeError: Cannot find function bind in object function (e, n, r) {...}. (https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js#19)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3629)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3613)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3634)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError2(ScriptRuntime.java:3650)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.notFunctionError(ScriptRuntime.java:3714)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThisHelper(ScriptRuntime.java:2233)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThis(ScriptRuntime.java:2215)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1333)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:19)
at script.r(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:16)
at script.r(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:384)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:16)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:16)
at script.t(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:16)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:16)
at script.t(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:7)
at script.t(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:463)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:463)
at script.t(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
at script.t(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:798)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:411)
at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:309)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3057)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.doRun(JavaScriptEngine.java:724)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:832)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:620)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:513)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:733)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadExternalJavaScriptFile(HtmlPage.java:1096)
at com.gargoylesoftware.htmlunit.html.HtmlScript.executeScriptIfNeeded(HtmlScript.java:395)
at com.gargoylesoftware.htmlunit.html.HtmlScript.execute(HtmlScript.java:270)
at com.gargoylesoftware.htmlunit.html.HtmlScript.onAllChildrenAddedToPage(HtmlScript.java:290)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:793)
at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:751)
at org.cyberneko.html.HTMLTagBalancer.callEndElement(HTMLTagBalancer.java:1170)
at org.cyberneko.html.HTMLTagBalancer.endElement(HTMLTagBalancer.java:1072)
at org.cyberneko.html.filters.DefaultFilter.endElement(DefaultFilter.java:206)
at org.cyberneko.html.filters.NamespaceBinder.endElement(NamespaceBinder.java:330)
at org.cyberneko.html.HTMLScanner$ContentScanner.scanEndElement(HTMLScanner.java:3126)
at org.cyberneko.html.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2093)
at org.cyberneko.html.HTMLScanner.scanDocument(HTMLScanner.java:920)
at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:499)
at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:452)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.parse(HTMLParser.java:1017)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parse(HTMLParser.java:248)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parseHtml(HTMLParser.java:194)
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:268)
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:156)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:471)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:345)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:410)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:395)
at Main.testingGargoyle(Main.java:19)
at Main.main(Main.java:10)
======= EXCEPTION END ========
我告诉过你它很大。我怎样才能解决这个问题并获得该页面的最终来源以便进行抓取？
提前致谢！
尝试使用不同的浏览器，例如：
String myUrl = "https://www.wearvr.com/#game_id=game_4";
try (WebClient webClient = new WebClient(BrowserVersion.CHROME)) {
HtmlPage myPage = ((HtmlPage) webClient.getPage(myUrl));
System.out.println(myPage.asXml());
} catch (FailingHttpStatusCodeException | IOException e) {
e.printStackTrace();
}
不过也有可能是IE8模拟的bug
抛出异常有多种原因，错误 html，脚本页面错误，未找到 css 等资源，脚本文件或图像文件（例如 <img src="bla.gif"> <- bla.gif not found HTML404)
所以我们使用这些选项来保持 html 导航而不会首先停止 error/problem 我们使用 :
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
您还可以实施空 类 以停止 htmlUnity 在控制台上详细介绍 css/javaScript 错误：
webClient.setCssErrorHandler(new SilentCssErrorHandler());
webClient.setJavaScriptErrorListener(new JavaScriptErrorListener(){});
小样本测试用例：
@Test
public void TestCall() throws FailingHttpStatusCodeException, MalformedURLException, IOException {
WebClient webClient = new WebClient(BrowserVersion.CHROME);
webClient.getOptions().setUseInsecureSSL(true); //ignore ssl certificate
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
String url = "https://www.wearvr.com/#game_id=game_4";
HtmlPage myPage = webClient.getPage(url);
webClient.waitForBackgroundJavaScriptStartingBefore(200);
webClient.waitForBackgroundJavaScript(20000);
//do stuff on page ex: myPage.getElementById("main")
//myPage.asXml() <- tags and elements
System.out.println(myPage.asText());
}