使用 HAProxy 负载均衡器的 Oozie HA 配置
Oozie HA configuration with HAProxy load balancer
我正在尝试使用 Cloudera Manager 5 启用 Oozie HA。该过程非常简单。我所要做的就是为配置的负载均衡器提供一个 hostname:port，Cloudera Manager 会创建一个额外的 Oozie Server。
我已经在主机 XXX.XXX.XXX.246 上有一个 Oozie Server，我的附加 Oozie Server 应该在主机 XXX.XXX.XXX.247 上。 HAProxy 应在主机 XXX.XXX.XXX.248 上配置。对于负载均衡器的 hostname:port，我选择了 XXX.XXX.XXX.248:11000。
haproxy.cfg 文件：
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
frontend main *:80
acl url_static path_beg -i /static /images /javascript /stylesheets
acl url_static path_end -i .jpg .gif .png .css .js
use_backend static if url_static
default_backend app
backend static
balance roundrobin
server static 127.0.0.1:4331 check
backend app
balance roundrobin
mode http
stats enable
stats uri /haproxy?stats
option httpclose
option forwardfor
server oozie1 XXX.XXX.XXX.246:80 check
server oozie2 XXX.XXX.XXX.247:80 check
当我尝试启动 Oozie 服务器时，出现以下异常：
java.lang.NoSuchFieldError: EXTERNAL_PROPERTY
at org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector._findTypeResolver(JacksonAnnotationIntrospector.java:777)
at org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findPropertyTypeResolver(JacksonAnnotationIntrospector.java:214)
at org.codehaus.jackson.map.ser.BeanSerializerFactory.findPropertyTypeSerializer(BeanSerializerFactory.java:370)
at org.codehaus.jackson.map.ser.BeanSerializerFactory._constructWriter(BeanSerializerFactory.java:772)
at org.codehaus.jackson.map.ser.BeanSerializerFactory.findBeanProperties(BeanSerializerFactory.java:586)
at org.codehaus.jackson.map.ser.BeanSerializerFactory.constructBeanSerializer(BeanSerializerFactory.java:430)
at org.codehaus.jackson.map.ser.BeanSerializerFactory.findBeanSerializer(BeanSerializerFactory.java:343)
at org.codehaus.jackson.map.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:287)
at org.codehaus.jackson.map.ser.StdSerializerProvider._createUntypedSerializer(StdSerializerProvider.java:782)
at org.codehaus.jackson.map.ser.StdSerializerProvider._createAndCacheUntypedSerializer(StdSerializerProvider.java:735)
at org.codehaus.jackson.map.ser.StdSerializerProvider.findValueSerializer(StdSerializerProvider.java:344)
at org.codehaus.jackson.map.ser.StdSerializerProvider.findTypedValueSerializer(StdSerializerProvider.java:420)
at org.codehaus.jackson.map.ser.StdSerializerProvider._serializeValue(StdSerializerProvider.java:601)
at org.codehaus.jackson.map.ser.StdSerializerProvider.serializeValue(StdSerializerProvider.java:256)
at org.codehaus.jackson.map.ObjectMapper._configAndWriteValue(ObjectMapper.java:2566)
at org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:2056)
at org.apache.oozie.util.FixedJsonInstanceSerializer.serialize(FixedJsonInstanceSerializer.java:65)
at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.internalRegisterService(ServiceDiscoveryImpl.java:201)
at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.registerService(ServiceDiscoveryImpl.java:186)
at org.apache.oozie.util.ZKUtils.advertiseService(ZKUtils.java:217)
at org.apache.oozie.util.ZKUtils.<init>(ZKUtils.java:141)
at org.apache.oozie.util.ZKUtils.register(ZKUtils.java:154)
at org.apache.oozie.service.ZKLocksService.init(ZKLocksService.java:70)
at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
at org.apache.oozie.service.Services.setService(Services.java:372)
at org.apache.oozie.service.Services.loadServices(Services.java:305)
at org.apache.oozie.service.Services.init(Services.java:213)
at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4709)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:802)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:944)
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:505)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1068)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1060)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
at org.apache.catalina.core.StandardService.start(StandardService.java:525)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:759)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
我不确定是哪部分配置有问题，是什么原因导致的。我将不胜感激任何帮助。谢谢！
我只想更新一下。问题是 CDH 5.x 并不常见的错误。
由于与 Oozie 和 Hive 使用的 Jackson 版本不兼容，并且根据将 jars 加载到 Oozie 的类路径中的顺序，Oozie 可能无法启动并出现前面提到的异常。
如果您使用包裹，解决方案如下：
- 删除或移动 /opt/cloudera/parcels/CDH/lib/oozie/libserver/hive-exec.jar 和 /opt/cloudera/parcels/CDH/lib/oozie/libtools/hive-exec.jar.
- 从 Cloudera 存储库下载 hive-exec--core.jar 并将其放入 /opt/cloudera/parcels/CDH/lib/oozie/libserver/ 和 /opt/cloudera/parcels/CDH/lib/oozie/libtools/.
- 从 maven 仓库下载 kryo-2.22.jar 并将其放入 /opt/cloudera/parcels/CDH/lib/oozie/libserver/ 和 /opt/cloudera/parcels/CDH/lib/oozie/libtools/.
haproxy 配置和 Oozie HA 启用现在应该很简单了。
我正在尝试使用 Cloudera Manager 5 启用 Oozie HA。该过程非常简单。我所要做的就是为配置的负载均衡器提供一个 hostname:port，Cloudera Manager 会创建一个额外的 Oozie Server。 我已经在主机 XXX.XXX.XXX.246 上有一个 Oozie Server，我的附加 Oozie Server 应该在主机 XXX.XXX.XXX.247 上。 HAProxy 应在主机 XXX.XXX.XXX.248 上配置。对于负载均衡器的 hostname:port，我选择了 XXX.XXX.XXX.248:11000。 haproxy.cfg 文件：
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
frontend main *:80
acl url_static path_beg -i /static /images /javascript /stylesheets
acl url_static path_end -i .jpg .gif .png .css .js
use_backend static if url_static
default_backend app
backend static
balance roundrobin
server static 127.0.0.1:4331 check
backend app
balance roundrobin
mode http
stats enable
stats uri /haproxy?stats
option httpclose
option forwardfor
server oozie1 XXX.XXX.XXX.246:80 check
server oozie2 XXX.XXX.XXX.247:80 check
当我尝试启动 Oozie 服务器时，出现以下异常：
java.lang.NoSuchFieldError: EXTERNAL_PROPERTY
at org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector._findTypeResolver(JacksonAnnotationIntrospector.java:777)
at org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findPropertyTypeResolver(JacksonAnnotationIntrospector.java:214)
at org.codehaus.jackson.map.ser.BeanSerializerFactory.findPropertyTypeSerializer(BeanSerializerFactory.java:370)
at org.codehaus.jackson.map.ser.BeanSerializerFactory._constructWriter(BeanSerializerFactory.java:772)
at org.codehaus.jackson.map.ser.BeanSerializerFactory.findBeanProperties(BeanSerializerFactory.java:586)
at org.codehaus.jackson.map.ser.BeanSerializerFactory.constructBeanSerializer(BeanSerializerFactory.java:430)
at org.codehaus.jackson.map.ser.BeanSerializerFactory.findBeanSerializer(BeanSerializerFactory.java:343)
at org.codehaus.jackson.map.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:287)
at org.codehaus.jackson.map.ser.StdSerializerProvider._createUntypedSerializer(StdSerializerProvider.java:782)
at org.codehaus.jackson.map.ser.StdSerializerProvider._createAndCacheUntypedSerializer(StdSerializerProvider.java:735)
at org.codehaus.jackson.map.ser.StdSerializerProvider.findValueSerializer(StdSerializerProvider.java:344)
at org.codehaus.jackson.map.ser.StdSerializerProvider.findTypedValueSerializer(StdSerializerProvider.java:420)
at org.codehaus.jackson.map.ser.StdSerializerProvider._serializeValue(StdSerializerProvider.java:601)
at org.codehaus.jackson.map.ser.StdSerializerProvider.serializeValue(StdSerializerProvider.java:256)
at org.codehaus.jackson.map.ObjectMapper._configAndWriteValue(ObjectMapper.java:2566)
at org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:2056)
at org.apache.oozie.util.FixedJsonInstanceSerializer.serialize(FixedJsonInstanceSerializer.java:65)
at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.internalRegisterService(ServiceDiscoveryImpl.java:201)
at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.registerService(ServiceDiscoveryImpl.java:186)
at org.apache.oozie.util.ZKUtils.advertiseService(ZKUtils.java:217)
at org.apache.oozie.util.ZKUtils.<init>(ZKUtils.java:141)
at org.apache.oozie.util.ZKUtils.register(ZKUtils.java:154)
at org.apache.oozie.service.ZKLocksService.init(ZKLocksService.java:70)
at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
at org.apache.oozie.service.Services.setService(Services.java:372)
at org.apache.oozie.service.Services.loadServices(Services.java:305)
at org.apache.oozie.service.Services.init(Services.java:213)
at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4709)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:802)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:944)
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:505)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1068)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1060)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
at org.apache.catalina.core.StandardService.start(StandardService.java:525)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:759)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
我不确定是哪部分配置有问题，是什么原因导致的。我将不胜感激任何帮助。谢谢！
我只想更新一下。问题是 CDH 5.x 并不常见的错误。 由于与 Oozie 和 Hive 使用的 Jackson 版本不兼容，并且根据将 jars 加载到 Oozie 的类路径中的顺序，Oozie 可能无法启动并出现前面提到的异常。 如果您使用包裹，解决方案如下：
- 删除或移动 /opt/cloudera/parcels/CDH/lib/oozie/libserver/hive-exec.jar 和 /opt/cloudera/parcels/CDH/lib/oozie/libtools/hive-exec.jar.
- 从 Cloudera 存储库下载 hive-exec--core.jar 并将其放入 /opt/cloudera/parcels/CDH/lib/oozie/libserver/ 和 /opt/cloudera/parcels/CDH/lib/oozie/libtools/.
- 从 maven 仓库下载 kryo-2.22.jar 并将其放入 /opt/cloudera/parcels/CDH/lib/oozie/libserver/ 和 /opt/cloudera/parcels/CDH/lib/oozie/libtools/.
haproxy 配置和 Oozie HA 启用现在应该很简单了。