EC2 集群中由 Infinispan 支持的 Hibernate Search 中的重复记录错误
Duplicate record errors in Hibernate Search backed by Infinispan in an EC2 cluster
我们有一个在 EC2 集群中运行的应用程序（目前有 2 个节点用于测试）。为了搜索领域模型，我们使用 Hibernate Search，并且由于应用程序在集群上运行，我们使用 Infinispan 作为 Lucene 目录。为了在重启后存活下来，我们在 MySQL 上使用 JDBC 缓存存储，两个节点访问相同的 MySQL 表。为了考虑添加和删除节点，我们使用 "jgroups" 后端进行 Hibernate Search worker 配置。
我们的问题是关于我们在尝试重建整个实体索引时收到的重复记录异常。我们收到类似堆栈跟踪的错误：
ERROR [AsyncStoreProcessor-LuceneIndexesData-0] [2016-06-21 17:01:59] org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore - ISPN008024: Error while storing string key to database; key: '_d.fdt|0|1048576|com.model.SomeModel'
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '_d.fdt|0|1048576|com.model.SomeModel' for key 'PRIMARY'
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1041)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4190)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4122)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2570)
at com.mysql.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:1399)
at com.mysql.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:857)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2460)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2377)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2361)
at com.zaxxer.hikari.proxy.PreparedStatementJavassistProxy.executeUpdate(PreparedStatementJavassistProxy.java)
at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore.write(JdbcStringBasedStore.java:174)
at org.infinispan.persistence.async.AsyncCacheWriter.applyModificationsSync(AsyncCacheWriter.java:158)
at org.infinispan.persistence.async.AsyncCacheWriter$AsyncStoreProcessor.retryWork(AsyncCacheWriter.java:330)
at org.infinispan.persistence.async.AsyncCacheWriter$AsyncStoreProcessor.run(AsyncCacheWriter.java:312)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
当我们检查数据库时，有一个条目带有这个ID。当我们用单个节点尝试时，没有错误。所以我们的猜测是两个节点都试图将缓存条目写入数据库。 可能导致此问题的原因是什么？据我所知，jgroups 后端应该会阻止它。
我们使用 hibernate 4.3.9.Final、hibernate-search 5.2.1.Final、infinispan 7.2.5.Final 和 jgroups 3.6.8.Final。 Infinispan配置如下：
<?xml version="1.0" encoding="UTF-8"?>
<infinispan xmlns="urn:infinispan:config:7.2"
xmlns:jdbc="urn:infinispan:config:store:jdbc:7.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
urn:infinispan:config:7.2 http://www.infinispan.org/schemas/infinispan-config-7.2.xsd
urn:infinispan:config:store:jdbc:7.2 http://www.infinispan.org/schemas/infinispan-cachestore-jdbc-config-7.2.xsd">
<jgroups>
<stack-file name="tcp" path="default-configs/default-jgroups-tcp.xml"/>
<stack-file name="ec2" path="search/infinispan-jgroups-ec2.xml"/>
</jgroups>
<cache-container name="HibernateSearch" default-cache="default" statistics="false" shutdown-hook="DONT_REGISTER">
<transport stack="${infinispan.transport:tcp}"/>
<!-- Duplicate domains are allowed so that multiple deployments with default configuration
of Hibernate Search applications work - if possible it would be better to use JNDI to share
the CacheManager across applications -->
<jmx duplicate-domains="true"/>
<!-- *************************************** -->
<!-- Cache to store Lucene's file metadata -->
<!-- *************************************** -->
<replicated-cache name="LuceneIndexesMetadata" mode="SYNC" remote-timeout="25000">
<transaction mode="NONE"/>
<state-transfer enabled="true" timeout="480000" await-initial-transfer="true"/>
<indexing index="NONE"/>
<locking striping="false" acquire-timeout="10000" concurrency-level="500" write-skew="false"/>
<eviction max-entries="-1" strategy="NONE"/>
<expiration max-idle="-1"/>
<persistence passivation="false">
<jdbc:string-keyed-jdbc-store preload="true" fetch-state="true" read-only="false" purge="false">
<jdbc:data-source jndi-url="java:comp/env/jdbc/..."/>
<jdbc:string-keyed-table drop-on-exit="false" create-on-start="true" prefix="ISPN_STRING_TABLE">
<jdbc:id-column name="ID" type="VARCHAR(255)"/>
<jdbc:data-column name="METADATA" type="BLOB"/>
<jdbc:timestamp-column name="TIMESTAMP" type="BIGINT"/>
</jdbc:string-keyed-table>
<property name="key2StringMapper">org.infinispan.lucene.LuceneKey2StringMapper</property>
<write-behind/>
</jdbc:string-keyed-jdbc-store>
</persistence>
</replicated-cache>
<!-- **************************** -->
<!-- Cache to store Lucene data -->
<!-- **************************** -->
<distributed-cache name="LuceneIndexesData" mode="SYNC" remote-timeout="25000">
<transaction mode="NONE"/>
<state-transfer enabled="true" timeout="480000" await-initial-transfer="true"/>
<indexing index="NONE"/>
<locking striping="false" acquire-timeout="10000" concurrency-level="500" write-skew="false"/>
<eviction max-entries="-1" strategy="NONE"/>
<expiration max-idle="-1"/>
<persistence passivation="false">
<jdbc:string-keyed-jdbc-store preload="true" fetch-state="true" read-only="false" purge="false">
<jdbc:data-source jndi-url="java:comp/env/jdbc/..."/>
<jdbc:string-keyed-table drop-on-exit="false" create-on-start="true" prefix="ISPN_STRING_TABLE">
<jdbc:id-column name="ID" type="VARCHAR(255)"/>
<jdbc:data-column name="DATA" type="MEDIUMBLOB"/>
<jdbc:timestamp-column name="TIMESTAMP" type="BIGINT"/>
</jdbc:string-keyed-table>
<property name="key2StringMapper">org.infinispan.lucene.LuceneKey2StringMapper</property>
<write-behind/>
</jdbc:string-keyed-jdbc-store>
</persistence>
</distributed-cache>
<!-- ***************************** -->
<!-- Cache to store Lucene locks -->
<!-- ***************************** -->
<replicated-cache name="LuceneIndexesLocking" mode="SYNC" remote-timeout="25000">
<transaction mode="NONE"/>
<state-transfer enabled="true" timeout="480000" await-initial-transfer="true"/>
<indexing index="NONE"/>
<locking striping="false" acquire-timeout="10000" concurrency-level="500" write-skew="false"/>
<eviction max-entries="-1" strategy="NONE"/>
<expiration max-idle="-1"/>
</replicated-cache>
</cache-container>
</infinispan>
各自的Hibernate配置如下（通过Spring完成）：
<bean id="emf" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="dataSource" ref="dataSource"/>
<property name="persistenceUnitName" value="..."/>
<property name="packagesToScan" value="com...."/>
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="generateDdl" value="false"/>
<property name="showSql" value="false"/>
<property name="databasePlatform" value="org.hibernate.dialect.MySQL5Dialect"/>
<property name="database" value="MYSQL"/>
</bean>
</property>
<property name="jpaPropertyMap">
<map>
<entry key="hibernate.default_batch_fetch_size" value="50"/>
<entry key="hibernate.multiTenancy" value="SCHEMA"/>
<entry key="hibernate.multi_tenant_connection_provider" value-ref="connectionProvider"/>
<entry key="hibernate.tenant_identifier_resolver" value-ref="tenantIdentifierResolver"/>
<entry key="hibernate.cache.use_second_level_cache" value="true"/>
<entry key="hibernate.cache.region.factory_class" value="com.hazelcast.hibernate.HazelcastCacheRegionFactory"/>
<entry key="hibernate.cache.hazelcast.use_native_client" value="true"/>
<entry key="hibernate.cache.hazelcast.native_client_address" value="127.0.0.1"/>
<entry key="hibernate.cache.hazelcast.native_client_group" value="dev"/>
<entry key="hibernate.cache.hazelcast.native_client_password" value="dev-pass"/>
<entry key="hibernate.connection.characterEncoding" value="UTF-8"/>
<entry key="hibernate.connection.useUnicode" value="true"/>
<entry key="hibernate.search.default.directory_provider" value="infinispan"/>
<entry key="hibernate.search.default.locking_cachename" value="LuceneIndexesLocking"/>
<entry key="hibernate.search.default.data_cachename" value="LuceneIndexesData"/>
<entry key="hibernate.search.default.metadata_cachename" value="LuceneIndexesMetadata"/>
<entry key="hibernate.search.default.chunk_size" value="1048576"/>
<entry key="hibernate.search.infinispan.configuration_resourcename" value="search/hibernatesearch-infinispan.xml"/>
<entry key="hibernate.search.default.worker.backend" value="jgroups"/>
<entry key="hibernate.search.services.jgroups.configurationFile" value="search/infinispan-jgroups-ec2.xml"/>
</map>
</property>
</bean>
您对 JGroups 后端 的用途是正确的，并且您的 Hibernate 配置看起来是正确的。
问题出在 Infinispan 中 CacheStore 组件的配置上，它们有一个“shared”属性，其默认值为 假.
<jdbc:string-keyed-jdbc-store
preload="true"
fetch-state="true"
read-only="false"
purge="false"
shared="true" <!-- FIX
如您所料，每个 Infinispan 节点都将在 "each" CacheStore 实例上重写相同的条目，因为它没有猜到您的每个节点实际上都连接到同一个数据库。
将 shared 属性设置为 true 应该确保 Infinispan 的核心将在节点之间进行协调，以便一个（并且只有一个）节点将写条目。
我们有一个在 EC2 集群中运行的应用程序（目前有 2 个节点用于测试）。为了搜索领域模型，我们使用 Hibernate Search，并且由于应用程序在集群上运行，我们使用 Infinispan 作为 Lucene 目录。为了在重启后存活下来，我们在 MySQL 上使用 JDBC 缓存存储，两个节点访问相同的 MySQL 表。为了考虑添加和删除节点，我们使用 "jgroups" 后端进行 Hibernate Search worker 配置。
我们的问题是关于我们在尝试重建整个实体索引时收到的重复记录异常。我们收到类似堆栈跟踪的错误：
ERROR [AsyncStoreProcessor-LuceneIndexesData-0] [2016-06-21 17:01:59] org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore - ISPN008024: Error while storing string key to database; key: '_d.fdt|0|1048576|com.model.SomeModel'
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '_d.fdt|0|1048576|com.model.SomeModel' for key 'PRIMARY'
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1041)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4190)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4122)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2570)
at com.mysql.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:1399)
at com.mysql.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:857)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2460)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2377)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2361)
at com.zaxxer.hikari.proxy.PreparedStatementJavassistProxy.executeUpdate(PreparedStatementJavassistProxy.java)
at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore.write(JdbcStringBasedStore.java:174)
at org.infinispan.persistence.async.AsyncCacheWriter.applyModificationsSync(AsyncCacheWriter.java:158)
at org.infinispan.persistence.async.AsyncCacheWriter$AsyncStoreProcessor.retryWork(AsyncCacheWriter.java:330)
at org.infinispan.persistence.async.AsyncCacheWriter$AsyncStoreProcessor.run(AsyncCacheWriter.java:312)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
当我们检查数据库时，有一个条目带有这个ID。当我们用单个节点尝试时，没有错误。所以我们的猜测是两个节点都试图将缓存条目写入数据库。 可能导致此问题的原因是什么？据我所知，jgroups 后端应该会阻止它。
我们使用 hibernate 4.3.9.Final、hibernate-search 5.2.1.Final、infinispan 7.2.5.Final 和 jgroups 3.6.8.Final。 Infinispan配置如下：
<?xml version="1.0" encoding="UTF-8"?>
<infinispan xmlns="urn:infinispan:config:7.2"
xmlns:jdbc="urn:infinispan:config:store:jdbc:7.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
urn:infinispan:config:7.2 http://www.infinispan.org/schemas/infinispan-config-7.2.xsd
urn:infinispan:config:store:jdbc:7.2 http://www.infinispan.org/schemas/infinispan-cachestore-jdbc-config-7.2.xsd">
<jgroups>
<stack-file name="tcp" path="default-configs/default-jgroups-tcp.xml"/>
<stack-file name="ec2" path="search/infinispan-jgroups-ec2.xml"/>
</jgroups>
<cache-container name="HibernateSearch" default-cache="default" statistics="false" shutdown-hook="DONT_REGISTER">
<transport stack="${infinispan.transport:tcp}"/>
<!-- Duplicate domains are allowed so that multiple deployments with default configuration
of Hibernate Search applications work - if possible it would be better to use JNDI to share
the CacheManager across applications -->
<jmx duplicate-domains="true"/>
<!-- *************************************** -->
<!-- Cache to store Lucene's file metadata -->
<!-- *************************************** -->
<replicated-cache name="LuceneIndexesMetadata" mode="SYNC" remote-timeout="25000">
<transaction mode="NONE"/>
<state-transfer enabled="true" timeout="480000" await-initial-transfer="true"/>
<indexing index="NONE"/>
<locking striping="false" acquire-timeout="10000" concurrency-level="500" write-skew="false"/>
<eviction max-entries="-1" strategy="NONE"/>
<expiration max-idle="-1"/>
<persistence passivation="false">
<jdbc:string-keyed-jdbc-store preload="true" fetch-state="true" read-only="false" purge="false">
<jdbc:data-source jndi-url="java:comp/env/jdbc/..."/>
<jdbc:string-keyed-table drop-on-exit="false" create-on-start="true" prefix="ISPN_STRING_TABLE">
<jdbc:id-column name="ID" type="VARCHAR(255)"/>
<jdbc:data-column name="METADATA" type="BLOB"/>
<jdbc:timestamp-column name="TIMESTAMP" type="BIGINT"/>
</jdbc:string-keyed-table>
<property name="key2StringMapper">org.infinispan.lucene.LuceneKey2StringMapper</property>
<write-behind/>
</jdbc:string-keyed-jdbc-store>
</persistence>
</replicated-cache>
<!-- **************************** -->
<!-- Cache to store Lucene data -->
<!-- **************************** -->
<distributed-cache name="LuceneIndexesData" mode="SYNC" remote-timeout="25000">
<transaction mode="NONE"/>
<state-transfer enabled="true" timeout="480000" await-initial-transfer="true"/>
<indexing index="NONE"/>
<locking striping="false" acquire-timeout="10000" concurrency-level="500" write-skew="false"/>
<eviction max-entries="-1" strategy="NONE"/>
<expiration max-idle="-1"/>
<persistence passivation="false">
<jdbc:string-keyed-jdbc-store preload="true" fetch-state="true" read-only="false" purge="false">
<jdbc:data-source jndi-url="java:comp/env/jdbc/..."/>
<jdbc:string-keyed-table drop-on-exit="false" create-on-start="true" prefix="ISPN_STRING_TABLE">
<jdbc:id-column name="ID" type="VARCHAR(255)"/>
<jdbc:data-column name="DATA" type="MEDIUMBLOB"/>
<jdbc:timestamp-column name="TIMESTAMP" type="BIGINT"/>
</jdbc:string-keyed-table>
<property name="key2StringMapper">org.infinispan.lucene.LuceneKey2StringMapper</property>
<write-behind/>
</jdbc:string-keyed-jdbc-store>
</persistence>
</distributed-cache>
<!-- ***************************** -->
<!-- Cache to store Lucene locks -->
<!-- ***************************** -->
<replicated-cache name="LuceneIndexesLocking" mode="SYNC" remote-timeout="25000">
<transaction mode="NONE"/>
<state-transfer enabled="true" timeout="480000" await-initial-transfer="true"/>
<indexing index="NONE"/>
<locking striping="false" acquire-timeout="10000" concurrency-level="500" write-skew="false"/>
<eviction max-entries="-1" strategy="NONE"/>
<expiration max-idle="-1"/>
</replicated-cache>
</cache-container>
</infinispan>
各自的Hibernate配置如下（通过Spring完成）：
<bean id="emf" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="dataSource" ref="dataSource"/>
<property name="persistenceUnitName" value="..."/>
<property name="packagesToScan" value="com...."/>
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="generateDdl" value="false"/>
<property name="showSql" value="false"/>
<property name="databasePlatform" value="org.hibernate.dialect.MySQL5Dialect"/>
<property name="database" value="MYSQL"/>
</bean>
</property>
<property name="jpaPropertyMap">
<map>
<entry key="hibernate.default_batch_fetch_size" value="50"/>
<entry key="hibernate.multiTenancy" value="SCHEMA"/>
<entry key="hibernate.multi_tenant_connection_provider" value-ref="connectionProvider"/>
<entry key="hibernate.tenant_identifier_resolver" value-ref="tenantIdentifierResolver"/>
<entry key="hibernate.cache.use_second_level_cache" value="true"/>
<entry key="hibernate.cache.region.factory_class" value="com.hazelcast.hibernate.HazelcastCacheRegionFactory"/>
<entry key="hibernate.cache.hazelcast.use_native_client" value="true"/>
<entry key="hibernate.cache.hazelcast.native_client_address" value="127.0.0.1"/>
<entry key="hibernate.cache.hazelcast.native_client_group" value="dev"/>
<entry key="hibernate.cache.hazelcast.native_client_password" value="dev-pass"/>
<entry key="hibernate.connection.characterEncoding" value="UTF-8"/>
<entry key="hibernate.connection.useUnicode" value="true"/>
<entry key="hibernate.search.default.directory_provider" value="infinispan"/>
<entry key="hibernate.search.default.locking_cachename" value="LuceneIndexesLocking"/>
<entry key="hibernate.search.default.data_cachename" value="LuceneIndexesData"/>
<entry key="hibernate.search.default.metadata_cachename" value="LuceneIndexesMetadata"/>
<entry key="hibernate.search.default.chunk_size" value="1048576"/>
<entry key="hibernate.search.infinispan.configuration_resourcename" value="search/hibernatesearch-infinispan.xml"/>
<entry key="hibernate.search.default.worker.backend" value="jgroups"/>
<entry key="hibernate.search.services.jgroups.configurationFile" value="search/infinispan-jgroups-ec2.xml"/>
</map>
</property>
</bean>
您对 JGroups 后端 的用途是正确的，并且您的 Hibernate 配置看起来是正确的。
问题出在 Infinispan 中 CacheStore 组件的配置上，它们有一个“shared”属性，其默认值为 假.
<jdbc:string-keyed-jdbc-store
preload="true"
fetch-state="true"
read-only="false"
purge="false"
shared="true" <!-- FIX
如您所料，每个 Infinispan 节点都将在 "each" CacheStore 实例上重写相同的条目，因为它没有猜到您的每个节点实际上都连接到同一个数据库。
将 shared 属性设置为 true 应该确保 Infinispan 的核心将在节点之间进行协调，以便一个（并且只有一个）节点将写条目。