httpclient 提供不同的结果

Question

我在使用 Apache HttpClient 4.5.6 连接 PowerSchool 系统时遇到问题。我省略了这个问题不需要的方法，但我 100% 确定它们会按预期工作。

当使用 Postman 或 requests.py 时，使用完全相同的表单数据（我已经在 python、java 和 Firefox 上进行了验证），我得到了预期的 HTML 页面，在  和  评论之间包含所需的成绩和 class 信息（超过 1000 行）。但是，当在 Java 中使用相同的表单数据 和 headers 时，生成的 HTML 页面仅包含此：

<!-- start student content -->
<div id="quickLookup">

<tr>
<th class="right" colspan="19">Attendance Totals</th>
<th>0</th>
<th>0</th>
</table>
<table border="0" cellpadding="3" cellspacing="1" width="100%">


<tr>
<td align="center">Current Cumulative GPA (Q1): X.XXXX</td>
</tr>

<tr>
<td align="center"><a href="home.html?schoolid=XXXX&showdropped=true&91146885685933636948">Show dropped classes also</a></td>
</tr>
</table>

<tr>
<th class="right" colspan="10">Attendance Totals</th>
<th>0</th>
<th>0</th>
<th>0</th>
<th>0</th>
</table>

</div>
<!-- end student content -->

这两个不同结果之间的脱节对我来说毫无意义，因为据我所知，Postman 和 requests.py 不执行 java 脚本。我希望 HttpClient 的结果是一样的。这是我的代码：

private static final BasicCookieStore cookieStore = new BasicCookieStore();
private static final HttpClient client = HttpClientBuilder.create().setDefaultCookieStore(cookieStore).build();

public static void main(String[] args) {

        String baseUrl = "https://powerschoolinstallurl/";
        String username = "username";
        String password = "password";

        try {
            // get hidden data fields, calc hmac data
            HashMap<String, String> result = getAuthCodes(baseUrl);
            String dbpwField = getDBPWField(result.get("contextData"), password);
            String pwField = getPWField(result.get("contextData"), password);

            List<NameValuePair> form = new ArrayList<>();
            form.add(new BasicNameValuePair("pstoken", result.get("pstoken")));
            form.add(new BasicNameValuePair("contextData", result.get("pstoken")));
            form.add(new BasicNameValuePair("dbpw", dbpwField));
            form.add(new BasicNameValuePair("serviceName", "PS Parent Portal"));
            form.add(new BasicNameValuePair("pcasServerURL", "/"));
            form.add(new BasicNameValuePair("credentialType", "User Id and Password Credential"));
            form.add(new BasicNameValuePair("account", username));
            form.add(new BasicNameValuePair("pw", pwField));
            form.add(new BasicNameValuePair("ldappassword", password));

            UrlEncodedFormEntity requestEntity = new UrlEncodedFormEntity(form);

            HttpPost postMethod = new HttpPost(baseUrl + "guardian/home.html");
            postMethod.setEntity(requestEntity);

            HttpResponse rawResponse = client.execute(postMethod);
                    System.out.println(rawResponse.getStatusLine().getStatusCode());
            try {
                String responseString = new BasicResponseHandler().handleResponse(rawResponse);
                System.out.println(responseString);
            } catch (HttpResponseException ignore) {}
            System.out.println(cookieStore.getCookies().toString());

            HttpGet getMethod = new HttpGet(baseUrl + "guardian/home.html");
            // replicating headers, result is the same nontheless
            getMethod.setHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
            getMethod.setHeader("Accept-Encoding", "gzip, deflate, br");
            getMethod.setHeader("Accept-Language", "en-US,en;q=0.5");
            getMethod.setHeader("Referer", "https://powerschoolinstallurl/public/home.html");
            getMethod.setHeader("DNT", "1");
            getMethod.setHeader("Connection", "keep-alive");
            getMethod.setHeader("Cache-Control", "no-cache");
            getMethod.setHeader("Host", "ps.install.domain");
            getMethod.setHeader("Upgrade-Insecure-Requests", "1");
            getMethod.setHeader("Pragma", "no-cache");

            HttpResponse resp2 = client.execute(getMethod);
            String responseString2 = new BasicResponseHandler().handleResponse(resp2);
            System.out.println(responseString2);


        } catch (IOException e) {
            e.printStackTrace();
        }
}

注意：不存在任何身份验证问题，除此细节外，页面正常返回。

Answer 1

据我了解，

HttpPost postMethod = new HttpPost(baseUrl + "guardian/home.html");

home.html 不包含任何服务器站点脚本，您可以使用 java 脚本或使用 Ajax 调用另一个 api 来填充您的数据。

所以来自 java,

HttpResponse resp2 = client.execute(getMethod);

它正在接受 html 并响应 html、

Answer 2

我最终使用了 Jsoup，遵循完全相同的逻辑并得到了我想要的结果。仍然不知道为什么 HttpClient 不起作用。

httpclient 提供不同的结果

httpclient delivering different results

java

httprequest

apache-httpclient-4.x

powerschool