通过负载均衡器进行身份验证时的 Keycloak Invalid_code
Keycloak Invalid_code when authenticating through load balancer
我正在尝试为我的公司设置 Keycloak，但 运行 遇到了一些我无法解决的问题。我目前以独立集群模式在两台服务器上安装了 keycloak。我把这些服务器放在后面，外部负载均衡器由另一个组拥有。我创建了两个领域，一个用于 Jenkins，一个用于 Grafana。当我将它们中的每一个配置为直接指向服务器时，它都可以正常工作。当我尝试通过负载平衡 url 进行身份验证时，问题就出现了。下面是来自 Keycloak、grafana 和 Jenksins 的一些日志。两台服务器都配置了 standalone-ha.xml 并且配置完全一样。
Keycloak 日志：
2020-02-19 06:30:19,599 WARN [org.keycloak.events] (default task-1) type=CODE_TO_TOKEN_ERROR, realmId=CICD, clientId=grafana, userId=null, ipAddress=24.43.182.84, error=invalid_code, grant_type=authorization_code, code_id=08c751b6-d6a1-4b55-9f69-4b4c28c0c9c4, client_auth_method=client-secret
Grafana 日志：
t=2020-02-19T14:32:09+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/login/generic_oauth status=302 remote_addr=172.23.0.5 time_ms=0 size=345 referer=https://example.com/grafana/login
t=2020-02-19T14:32:52+0000 lvl=info msg="state check" logger=oauth queryState=0c5576040cb0984602e4e5a8ccc891e425065c740f9c6e4b3331494ad5c69b9b cookieState=0c5576040cb0984602e4e5a8ccc891e425065c740f9c6e4b3331494ad5c69b9b
t=2020-02-19T14:32:52+0000 lvl=eror msg=login.OAuthLogin(NewTransportWithCode) logger=context userId=0 orgId=0 uname= error="oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\",\"error_description\":\"Code not valid\"}"
t=2020-02-19T14:32:52+0000 lvl=eror msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/login/generic_oauth status=500 remote_addr=172.23.0.5 time_ms=21 size=1751 referer="https://load_balanced_example.com/auth/realms/CICD/protocol/openid-connect/auth?access_type=online&client_id=grafana&redirect_uri=https%3A%2F%2Fexample.com%2Fgrafana%2Flogin%2Fgeneric_oauth&response_type=code&scope=openid+email+profile&state=kyersRiz2wsAryohnLlZPPdtQjA6MJO8wanOnDaXgaY%3D"
詹金斯的回应：
com.google.api.client.auth.oauth2.TokenResponseException: 400 Bad Request
{
"error" : "invalid_grant",
"error_description" : "Code not valid"
}
Keycloak 服务：
[Unit]
Description=Keycloak
After=network.target
[Service]
Type=idle
User=keycloak
Group=keycloak
ExecStart=/opt/keycloak/current/bin/standalone.sh --server-config=standalone-ha.xml -b 0.0.0.0
TimeoutStartSec=600
TimeoutStopSec=600
[Install]
WantedBy=multi-user.target
我在 keycloak 中设置客户端的屏幕截图：
我在 Grafana 中的配置：
GF_AUTH_GENERIC_OAUTH_ENABLED=True
GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=True
GF_AUTH_GENERIC_OAUTH_NAME=KeyCloakOAuth
GF_AUTH_GENERIC_OAUTH_CLIENT_ID=grafana
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=MYSECRETKEY
GF_AUTH_GENERIC_OAUTH_SCOPES=openid email profile
GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://load_balanced_example/auth/realms/CICD/protocol/openid-connect/auth
GF_AUTH_GENERIC_OAUTH_TOKEN_URL=https://load_balanced_example.com/auth/realms/CICD/protocol/openid-connect/token
GF_AUTH_GENERIC_OAUTH_API_URL=https://load_balanced_example.com/auth/realms/CICD/protocol/openid-connect/userinfo
GF_AUTH_GENERIC_OAUTH_TLS_SKIP_VERIFY_INSECURE=True
GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH=contains(roles[*], 'Admin') && 'Admin' || contains(roles[*], 'Editor') && 'Editor' || 'Viewer'
我的一些独立-ha.xml配置：
<subsystem xmlns="urn:jboss:domain:datasources:5.0">
<datasources>
<datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true" statistics-enabled="${wildfly.datasources.statistics-enabled:${wildfly.statistics-enabled:false}}">
<connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</connection-url>
<driver>h2</driver>
<security>
<user-name>sa</user-name>
<password>sa</password>
</security>
</datasource>
<datasource jndi-name="java:jboss/datasources/KeycloakDS" pool-name="KeycloakDS" enabled="true" use-java-context="true" statistics-enabled="${wildfly.datasources.statistics-enabled:${wildfly.statistics-enabled:false}}">
<connection-url>jdbc:mariadb://example_db.com:3306/keycloak</connection-url>
<driver>mariadb</driver>
<security>
<user-name>keycloak</user-name>
<password>mydbpassword</password>
</security>
</datasource>
<drivers>
<driver name="h2" module="com.h2database.h2">
<xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
</driver>
<driver name="mariadb" module="org.mariadb">
<xa-datasource-class>org.mariadb.jdbc.MariaDbDataSource</xa-datasource-class>
</driver>
</drivers>
</datasources>
<server name="default-server">
<ajp-listener name="ajp" socket-binding="ajp"/>
<http-listener name="default" socket-binding="http" proxy-address-forwarding="true" redirect-socket="proxy-https"/>
<https-listener name="https" socket-binding="https" security-realm="ApplicationRealm" enable-http2="true"/>
<host name="default-host" alias="localhost">
<location name="/" handler="welcome-content"/>
<http-invoker security-realm="ApplicationRealm"/>
</host>
</server>
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
<socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
<socket-binding name="http" port="${jboss.http.port:8080}"/>
<socket-binding name="https" port="${jboss.https.port:8443}"/>
<socket-binding name="jgroups-mping" interface="private" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
<socket-binding name="jgroups-tcp" interface="private" port="7600"/>
<socket-binding name="jgroups-tcp-fd" interface="private" port="57600"/>
<socket-binding name="jgroups-udp" interface="private" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
<socket-binding name="jgroups-udp-fd" interface="private" port="54200"/>
<socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
<socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>
<socket-binding name="modcluster" multicast-address="${jboss.modcluster.multicast.address:224.0.1.105}" multicast-port="23364"/>
<socket-binding name="txn-recovery-environment" port="4712"/>
<socket-binding name="txn-status-manager" port="4713"/>
<socket-binding name="proxy-https" port="443"/>
<outbound-socket-binding name="mail-smtp">
<remote-destination host="localhost" port="25"/>
</outbound-socket-binding>
另一件可能值得指出的事情是我已经在 apache 反向代理后面设置了 keycloak 服务器并将负载均衡器指向它，并且我已经将 apache 排除在外并让负载均衡器指向直接连接到端口 8443 上的 keycloak 服务。
我还从执行负载平衡的团队那里验证了 X-Forwarded-For 和 X-Forwarded-Proto 已启用并且它们正在转发客户端 IP。如果在 LB 上还有其他设置要查找，请告诉我。
如果有人能帮我指出正确的方向，那就太好了！如果有未包含的日志或可以帮助解决问题的配置，请告诉我。
我成功了。问题在于两个节点的集群。我必须将 jgroups 配置为使用 TCP 而不是 UDP 并添加了 TCPPING。
<subsystem xmlns="urn:jboss:domain:jgroups:7.0">
<channels default="ee">
<channel name="ee" stack="tcp" cluster="ejb"/>
</channels>
<stacks>
<stack name="udp">
<transport type="UDP" socket-binding="jgroups-udp"/>
<protocol type="PING"/>
<protocol type="MERGE3"/>
<socket-protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="UFC"/>
<protocol type="MFC"/>
<protocol type="FRAG3"/>
</stack>
<stack name="tcp">
<transport type="TCP" socket-binding="jgroups-tcp"/>
<protocol type="TCPPING">
<property name="initial_hosts">server1_IP[7600],server2_IP[7600]</property>
<property name="ergonomics">false</property>
</protocol>
<protocol type="MERGE3"/>
<protocol type="MERGE3"/>
<socket-protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="MFC"/>
<protocol type="FRAG3"/>
</stack>
</stacks>
</subsystem>
然后我不得不更改服务以绑定我的私有 IP。
[Unit]
Description=Keycloak
After=network.target
[Service]
Type=idle
User=keycloak
Group=keycloak
ExecStart=/opt/keycloak/current/bin/standalone.sh --server-config=standalone-ha.xml -b 0.0.0.0 -bprivate PRIVATEIP
TimeoutStartSec=600
TimeoutStopSec=600
[Install]
WantedBy=multi-user.target
我正在尝试为我的公司设置 Keycloak，但 运行 遇到了一些我无法解决的问题。我目前以独立集群模式在两台服务器上安装了 keycloak。我把这些服务器放在后面，外部负载均衡器由另一个组拥有。我创建了两个领域，一个用于 Jenkins，一个用于 Grafana。当我将它们中的每一个配置为直接指向服务器时，它都可以正常工作。当我尝试通过负载平衡 url 进行身份验证时，问题就出现了。下面是来自 Keycloak、grafana 和 Jenksins 的一些日志。两台服务器都配置了 standalone-ha.xml 并且配置完全一样。
Keycloak 日志：
2020-02-19 06:30:19,599 WARN [org.keycloak.events] (default task-1) type=CODE_TO_TOKEN_ERROR, realmId=CICD, clientId=grafana, userId=null, ipAddress=24.43.182.84, error=invalid_code, grant_type=authorization_code, code_id=08c751b6-d6a1-4b55-9f69-4b4c28c0c9c4, client_auth_method=client-secret
Grafana 日志：
t=2020-02-19T14:32:09+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/login/generic_oauth status=302 remote_addr=172.23.0.5 time_ms=0 size=345 referer=https://example.com/grafana/login
t=2020-02-19T14:32:52+0000 lvl=info msg="state check" logger=oauth queryState=0c5576040cb0984602e4e5a8ccc891e425065c740f9c6e4b3331494ad5c69b9b cookieState=0c5576040cb0984602e4e5a8ccc891e425065c740f9c6e4b3331494ad5c69b9b
t=2020-02-19T14:32:52+0000 lvl=eror msg=login.OAuthLogin(NewTransportWithCode) logger=context userId=0 orgId=0 uname= error="oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\",\"error_description\":\"Code not valid\"}"
t=2020-02-19T14:32:52+0000 lvl=eror msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/login/generic_oauth status=500 remote_addr=172.23.0.5 time_ms=21 size=1751 referer="https://load_balanced_example.com/auth/realms/CICD/protocol/openid-connect/auth?access_type=online&client_id=grafana&redirect_uri=https%3A%2F%2Fexample.com%2Fgrafana%2Flogin%2Fgeneric_oauth&response_type=code&scope=openid+email+profile&state=kyersRiz2wsAryohnLlZPPdtQjA6MJO8wanOnDaXgaY%3D"
詹金斯的回应：
com.google.api.client.auth.oauth2.TokenResponseException: 400 Bad Request
{
"error" : "invalid_grant",
"error_description" : "Code not valid"
}
Keycloak 服务：
[Unit]
Description=Keycloak
After=network.target
[Service]
Type=idle
User=keycloak
Group=keycloak
ExecStart=/opt/keycloak/current/bin/standalone.sh --server-config=standalone-ha.xml -b 0.0.0.0
TimeoutStartSec=600
TimeoutStopSec=600
[Install]
WantedBy=multi-user.target
我在 keycloak 中设置客户端的屏幕截图：
我在 Grafana 中的配置：
GF_AUTH_GENERIC_OAUTH_ENABLED=True
GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=True
GF_AUTH_GENERIC_OAUTH_NAME=KeyCloakOAuth
GF_AUTH_GENERIC_OAUTH_CLIENT_ID=grafana
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=MYSECRETKEY
GF_AUTH_GENERIC_OAUTH_SCOPES=openid email profile
GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://load_balanced_example/auth/realms/CICD/protocol/openid-connect/auth
GF_AUTH_GENERIC_OAUTH_TOKEN_URL=https://load_balanced_example.com/auth/realms/CICD/protocol/openid-connect/token
GF_AUTH_GENERIC_OAUTH_API_URL=https://load_balanced_example.com/auth/realms/CICD/protocol/openid-connect/userinfo
GF_AUTH_GENERIC_OAUTH_TLS_SKIP_VERIFY_INSECURE=True
GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH=contains(roles[*], 'Admin') && 'Admin' || contains(roles[*], 'Editor') && 'Editor' || 'Viewer'
我的一些独立-ha.xml配置：
<subsystem xmlns="urn:jboss:domain:datasources:5.0">
<datasources>
<datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true" statistics-enabled="${wildfly.datasources.statistics-enabled:${wildfly.statistics-enabled:false}}">
<connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</connection-url>
<driver>h2</driver>
<security>
<user-name>sa</user-name>
<password>sa</password>
</security>
</datasource>
<datasource jndi-name="java:jboss/datasources/KeycloakDS" pool-name="KeycloakDS" enabled="true" use-java-context="true" statistics-enabled="${wildfly.datasources.statistics-enabled:${wildfly.statistics-enabled:false}}">
<connection-url>jdbc:mariadb://example_db.com:3306/keycloak</connection-url>
<driver>mariadb</driver>
<security>
<user-name>keycloak</user-name>
<password>mydbpassword</password>
</security>
</datasource>
<drivers>
<driver name="h2" module="com.h2database.h2">
<xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
</driver>
<driver name="mariadb" module="org.mariadb">
<xa-datasource-class>org.mariadb.jdbc.MariaDbDataSource</xa-datasource-class>
</driver>
</drivers>
</datasources>
<server name="default-server">
<ajp-listener name="ajp" socket-binding="ajp"/>
<http-listener name="default" socket-binding="http" proxy-address-forwarding="true" redirect-socket="proxy-https"/>
<https-listener name="https" socket-binding="https" security-realm="ApplicationRealm" enable-http2="true"/>
<host name="default-host" alias="localhost">
<location name="/" handler="welcome-content"/>
<http-invoker security-realm="ApplicationRealm"/>
</host>
</server>
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
<socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
<socket-binding name="http" port="${jboss.http.port:8080}"/>
<socket-binding name="https" port="${jboss.https.port:8443}"/>
<socket-binding name="jgroups-mping" interface="private" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
<socket-binding name="jgroups-tcp" interface="private" port="7600"/>
<socket-binding name="jgroups-tcp-fd" interface="private" port="57600"/>
<socket-binding name="jgroups-udp" interface="private" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
<socket-binding name="jgroups-udp-fd" interface="private" port="54200"/>
<socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
<socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>
<socket-binding name="modcluster" multicast-address="${jboss.modcluster.multicast.address:224.0.1.105}" multicast-port="23364"/>
<socket-binding name="txn-recovery-environment" port="4712"/>
<socket-binding name="txn-status-manager" port="4713"/>
<socket-binding name="proxy-https" port="443"/>
<outbound-socket-binding name="mail-smtp">
<remote-destination host="localhost" port="25"/>
</outbound-socket-binding>
另一件可能值得指出的事情是我已经在 apache 反向代理后面设置了 keycloak 服务器并将负载均衡器指向它，并且我已经将 apache 排除在外并让负载均衡器指向直接连接到端口 8443 上的 keycloak 服务。
我还从执行负载平衡的团队那里验证了 X-Forwarded-For 和 X-Forwarded-Proto 已启用并且它们正在转发客户端 IP。如果在 LB 上还有其他设置要查找，请告诉我。
如果有人能帮我指出正确的方向，那就太好了！如果有未包含的日志或可以帮助解决问题的配置，请告诉我。
我成功了。问题在于两个节点的集群。我必须将 jgroups 配置为使用 TCP 而不是 UDP 并添加了 TCPPING。
<subsystem xmlns="urn:jboss:domain:jgroups:7.0">
<channels default="ee">
<channel name="ee" stack="tcp" cluster="ejb"/>
</channels>
<stacks>
<stack name="udp">
<transport type="UDP" socket-binding="jgroups-udp"/>
<protocol type="PING"/>
<protocol type="MERGE3"/>
<socket-protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="UFC"/>
<protocol type="MFC"/>
<protocol type="FRAG3"/>
</stack>
<stack name="tcp">
<transport type="TCP" socket-binding="jgroups-tcp"/>
<protocol type="TCPPING">
<property name="initial_hosts">server1_IP[7600],server2_IP[7600]</property>
<property name="ergonomics">false</property>
</protocol>
<protocol type="MERGE3"/>
<protocol type="MERGE3"/>
<socket-protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="MFC"/>
<protocol type="FRAG3"/>
</stack>
</stacks>
</subsystem>
然后我不得不更改服务以绑定我的私有 IP。
[Unit]
Description=Keycloak
After=network.target
[Service]
Type=idle
User=keycloak
Group=keycloak
ExecStart=/opt/keycloak/current/bin/standalone.sh --server-config=standalone-ha.xml -b 0.0.0.0 -bprivate PRIVATEIP
TimeoutStartSec=600
TimeoutStopSec=600
[Install]
WantedBy=multi-user.target