Apache Ignite 2.x - 我可以利用堆外存储的优势，并且仍然可以进行基于时间的驱逐（来自堆内和堆外存储）吗？

Question

我正在使用 Apache Ignite 2.8.1

我有一个特定的缓存，我需要将其存储在堆外（因为否则我会遇到内存不足的情况）。而且我还需要分发缓存。

但我还想要一个基于时间的逐出策略来从 on/off 内存缓存中逐出条目。

Apache Ignite 甚至可以做到这一点吗2.x？

在此处查看代码段。我尝试过以各种方式配置它，但除了手动操作外，没有任何东西可以从缓存中逐出这些条目。

当我运行下面的test方法时，所有条目都保留在缓存中。

import java.time.LocalDateTime;
import java.util.Arrays;
import java.util.concurrent.TimeUnit;

import javax.cache.expiry.CreatedExpiryPolicy;
import javax.cache.expiry.Duration;

import org.apache.ignite.Ignite;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.CacheAtomicityMode;
import org.apache.ignite.cache.CacheMode;
import org.apache.ignite.cache.eviction.lru.LruEvictionPolicyFactory;
import org.apache.ignite.configuration.CacheConfiguration;
import org.apache.ignite.configuration.DataRegionConfiguration;
import org.apache.ignite.configuration.DataStorageConfiguration;
import org.apache.ignite.configuration.IgniteConfiguration;
import org.apache.ignite.configuration.NearCacheConfiguration;
import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
    
public class IgniteCache {
    private static final String ORG_CACHE = IgniteCache.class.getSimpleName() + "Organizations";

    private static Ignite ignite;
    
    private static org.apache.ignite.IgniteCache cache;
    
    public void start() {

        IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
        DataStorageConfiguration dsCfg = new DataStorageConfiguration();
        DataRegionConfiguration dfltDataRegConf = new DataRegionConfiguration();
        dfltDataRegConf.setPersistenceEnabled(true);
        dsCfg.setDefaultDataRegionConfiguration(dfltDataRegConf);
        dsCfg.setStoragePath("E:\igniteStorage");
        igniteConfiguration.setDataStorageConfiguration(dsCfg);

        TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
        TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
        ipFinder.setAddresses(Arrays.asList("127.0.0.1:47500..47509"));
        tcpDiscoverySpi.setIpFinder(ipFinder);

        igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi);

        ignite = Ignition.start(igniteConfiguration);
        
        ignite.active(true);

        CacheConfiguration<Long, X12File> cacheCfg = new CacheConfiguration<>(ORG_CACHE);

        cacheCfg.setCacheMode(CacheMode.REPLICATED);
        cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
        cacheCfg.setBackups(1);
        
        cacheCfg.setEvictionPolicyFactory(new LruEvictionPolicyFactory<>(50000, 40, 5050)); // this doesn't do anything.
        cacheCfg.setNearConfiguration(
                new NearCacheConfiguration<Long, X12File>()
                .setNearEvictionPolicyFactory(new LruEvictionPolicyFactory<>(1_000_000))); // this doesn't do anything.
        
        cache = ignite.getOrCreateCache(cacheCfg);      
        
        for (long i = 0; i < 4_000_000; i++) {
            if (i > 0 && i % 10_000 == 0) {
                System.out.println("Done: " + i);
            }
            cache.withExpiryPolicy(new CreatedExpiryPolicy(new Duration(TimeUnit.SECONDS, 1))) // this expiry policy doesn't do anything
                .put(i, new X12File("x12file" + i, LocalDateTime.now().toString()));
        }
    }
    
    public void test() {
        System.out.println("Checking if cache entries are being properly evicted ...");

        int matches = 0;
        for (long i = 0; i < 4_000_000; i++) {
            if (cache.get(i) != null) {
                ++matches;
            }
        }
        System.out.println("Matches: " + matches);
    }   
}

在我的特定用例中，这只是 Apache Ignite 2.x 的缺点吗？

Answer 1

简而言之：

驱逐是 memory-based。删除一些记录，因为我的行太多或者它使用了太多 space
到期时间为 time-based。此记录在指定时间段内未被触及，因此应将其删除

所有缓存都是off-heap。您可以像这样为缓存配置 the expiry policy：

<bean class="org.apache.ignite.configuration.CacheConfiguration">
    <property name="name" value="myCache"/>
    <property name="expiryPolicyFactory">
        <bean class="javax.cache.expiry.CreatedExpiryPolicy" factory-method="factoryOf">
            <constructor-arg>
                <bean class="javax.cache.expiry.Duration">
                    <constructor-arg value="MINUTES"/>
                    <constructor-arg value="5"/>
                </bean>
            </constructor-arg>
        </bean>
   </property>
</bean>

（near/on-heap 缓存确实做了一些事情。只是不是你所期望的！它们都 on-堆 off-heap 缓存。）

Answer 2

据我从您的代码和解释中了解到，您似乎对驱逐政策和到期政策有些困惑。

逐出策略确定在 RAM 耗尽时应从 RAM 中删除哪些数据。
用于确定缓存条目的 TTL 的过期策略。

在您的情况下，您只需要过期政策。看我的例子：

private static final String ORG_CACHE = IgniteCache.class.getSimpleName() + "Organizations";

public static void main(String[] args) throws Exception {
    DataRegionConfiguration dfltDataRegConf = new DataRegionConfiguration();
    dfltDataRegConf.setPersistenceEnabled(true);

    DataStorageConfiguration dsCfg = new DataStorageConfiguration();
    dsCfg.setDefaultDataRegionConfiguration(dfltDataRegConf);
    dsCfg.setStoragePath("/home/kazakov/tmp");

    IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
    igniteConfiguration.setDataStorageConfiguration(dsCfg);

    TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
    TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
    ipFinder.setAddresses(Arrays.asList("127.0.0.1:47500..47509"));
    tcpDiscoverySpi.setIpFinder(ipFinder);

    igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi);

    try(Ignite ignite = Ignition.start(igniteConfiguration)) {
        ignite.active(true);

        CacheConfiguration<Long, X12File> cacheCfg = new CacheConfiguration<>(ORG_CACHE);

        cacheCfg.setCacheMode(CacheMode.REPLICATED);
        cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
        cacheCfg.setBackups(1);
        cacheCfg.setOnheapCacheEnabled(true);

        IgniteCache<Long, X12File> cache = ignite.getOrCreateCache(cacheCfg).withExpiryPolicy(new CreatedExpiryPolicy(new Duration(TimeUnit.SECONDS, 1)));

        for (long i = 0; i < 4_000_000; i++) {
            if (i > 0 && i % 10_000 == 0)
                System.out.println("Done: " + i);

            cache.put(i, new X12File("x12file" + i, LocalDateTime.now().toString()));
        }

        Thread.sleep(5000);

        int matches = 0;
        for (long i = 0; i < 4_000_000; i++) {
            if (cache.get(i) != null)
                ++matches;
        }
        System.out.println("Matches: " + matches);
    }
}

Apache Ignite 2.x - 我可以利用堆外存储的优势，并且仍然可以进行基于时间的驱逐（来自堆内和堆外存储）吗？

Apache Ignite 2.x - can I take advantage of offheap storage and also still have time based eviction (from the on & off heap storage)?

java

distributed-caching

ignite