点燃 caching/querying 花费太多时间

Question

我正在尝试采用 ignite 通过实现内存数据网格来解决我的一个需求

正确知道我正在使用第三个持久性 read/write 通过机制从我的 oracle 数据库中获取数据，在我的拓扑中我正在使用我激活的集群中的单个节点，该节点托管在具有 8G 内存的 VM 中和 120 G 硬盘。

我的节点正在使用具有 2G 堆上内存和 50G 堆外内存的本地存储，所有这些都启用了逐出和交换

DataStorageConfiguration dataStorageCfg = new DataStorageConfiguration();

DataRegionConfiguration dataRegionCfg = new DataRegionConfiguration();



// 2G initial size (RAM).
dataRegionCfg.setInitialSize(2L * 1024 * 1024 * 1024);

// 40 GB max size (RAM).
dataRegionCfg.setMaxSize(40L * 1024 * 1024 * 1024);

// Enabling RANDOM_LRU eviction for this region.
dataRegionCfg.setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU);

//dataRegionCfg.setPersistenceEnabled(true);
final String swapPath ="/opt/ignite/swap";

dataRegionCfg.setSwapPath(swapPath);

dataStorageCfg.setDefaultDataRegionConfiguration(dataRegionCfg);

cfg.setDataStorageConfiguration(dataStorageCfg);

在我的机器上缓存这个太花时间了

当缓存结束时，我的交换文件夹大约有 13G 关于我的 SQL 查询没有回复

我工具中的相同查询需要 1 分钟 45 秒才能响应，但使用 ignite 缓存查询方法不会响应，也不会抛出任何类型的错误或异常

SqlFieldsQuery sqlQuery;
FieldsQueryCursor<List<?>> queryCursor;
Iterator<List<?>> resultIt;

System.out.println(">>> All caches loaded! in : " + total + " ms");

System.out.println("---------------------------------------------- ");
System.out.println("---------------------------------------------- ");
System.out.println("---------------------------------------------- ");

System.out.println("\n \n \n ");
System.out.println("---------------------------------------------- ");
System.out.println("---------------------------------------------- ");
System.out.println("---------------------------------------------- ");

System.out.println("Checking join query POC first run");

start = System.currentTimeMillis();
sqlQuery = new SqlFieldsQuery(sql);
queryCursor = ignite.cache("MInoutlineCache").query(sqlQuery);
System.out.println("query result size is : "+queryCursor.getAll().size());

end = System.currentTimeMillis() - start;
total += end;

M我用ignite好吗？以单节点方式使用 ignite 是否有用，或者我打算在分区策略中构建具有大量节点的集群？

缓存行数为1000万行有没有另一种方法可以使用第三种持久性策略在我的上下文中实现良好的内存数据网格。

这个问题有很多问题对不起

注意： 我正在使用网格增益控制台生成我的配置我还将我的缓存模式名称更新为 public 以直接执行查询

这里是查询

 SELECT bp.name,
    CF.documentno,
    CF.MOVEMENTDATE,
    CF.m_product_id AS M_PRODUCT_ID,
    CF.product,
    CF.xx_lignegratuite,
    CF.m_attributesetinstance_id,
    CASE
      WHEN cf.isreturntrx='Y'
      THEN - CF.qtyentered
      ELSE CF.qtyentered
    END AS qtyentered,
    CF.discount,
    CF.DOCSTATUS,
    CF.ISRETURNTRX,
    CF.XX_REWARDAMT,
    CF.OperID,
    CF.clientId,
    CASE
      WHEN cf.xx_lignegratuite='N'
      THEN
        CASE
          WHEN cf.isreturntrx='Y'
          THEN             -cf.prixVente*cf.qtyentered* (1-(cf.discount/100))
          ELSE cf.prixVente*cf.qtyentered* (1-(cf.discount/100))
        END
      ELSE 0
    END AS totalline,
    CASE
      WHEN cf.XX_StartegicalProduct='Y'
      THEN (
        CASE
          WHEN cf.xx_lignegratuite='N'
          THEN
            CASE
              WHEN cf.isreturntrx='Y'
              THEN             -cf.prixVente*cf.qtyentered* (1-(cf.discount/100))
              ELSE cf.prixVente*cf.qtyentered* (1-(cf.discount/100))
            END
          ELSE 0
        END)
      ELSE 0
    END AS totallineStar,
    CASE
      WHEN cf.xx_lignegratuite='N'
      THEN
        CASE
          WHEN cf.typevente='W'
          THEN
            CASE
              WHEN cf.isreturntrx='Y'
              THEN                -(cf.XX_REWARDAMT/nb_doc)
              ELSE cf.XX_REWARDAMT/nb_doc
            END
          ELSE 0
        END
      ELSE 0
    END AS totalreward,
    CF.XX_StartegicalProduct,
    CF.SALESREP_ID,
    CF.C_DOCTYPE_ID,
    CF.AD_ORG_ID,
    CF.ad_orgtrx_id,
    CF.xx_laboratory_id,
    bp.c_bpartner_id,
    CF.nb_doc,
    CF.rate,
    CF.poste_id,
    CF.SalesRepTier_poste_id,
    CF.recSupr,
    CF.recSupr_poste_id,
    (SELECT Objectif_CA_oper
    FROM c_bpartner
    WHERE issalesrep ='Y'
    AND isemployee   ='Y'
    AND c_bpartner_id=CF.SalesRepTier
    ) AS ObjectifOp,
    (SELECT Objectif_CA_oper
    FROM c_bpartner
    WHERE issalesrep ='Y'
    AND isemployee   ='Y'
    AND c_bpartner_id=CF.SalesRepTier_poste_id
    ) AS ObjectifOp_poste_id,
    CASE
      WHEN cf.ISRETURNTRX='Y'
      THEN              -cf.QTYENTERED*prixRevient
      ELSE cf.QTYENTERED*prixRevient
    END AS consomation
  FROM
    (SELECT i.documentno,
      i.MOVEMENTDATE,
      p.m_product_id,
      p.name AS product,
      ol.xx_lignegratuite,
      il.m_attributesetinstance_id,
      il.qtyentered,
      ol.discount,
      i.DOCSTATUS,
      i.isreturntrx,
      ol.XX_REWARDAMT,
      i.C_BPartner_ID AS clientId,
      (SELECT u.C_BPARTNER_ID FROM AD_User u WHERE u.AD_User_ID = i.SALESREP_ID
      ) AS OperID,
      (SELECT ai.Valuenumber
      FROM M_AttributeInstance ai
      INNER JOIN M_Attribute a
      ON (ai.M_Attribute_ID             =a.M_Attribute_ID
      AND a.IsInstanceAttribute         ='Y')
      WHERE ai.M_AttributeSetInstance_ID=il.m_attributesetinstance_id
      AND a.Name                        ='Prix Vente'
      ) AS prixVente,
      (SELECT ai.Valuenumber
      FROM M_AttributeInstance ai
      INNER JOIN M_Attribute a
      ON (ai.M_Attribute_ID             =a.M_Attribute_ID
      AND a.IsInstanceAttribute         ='Y')
      WHERE ai.M_AttributeSetInstance_ID=il.m_attributesetinstance_id
      AND a.Name                        ='Prix Revient'
      ) AS prixRevient,
      (SELECT ai.Valuenumber
      FROM M_AttributeInstance ai
      INNER JOIN M_Attribute a
      ON (ai.M_Attribute_ID             =a.M_Attribute_ID
      AND a.IsInstanceAttribute         ='Y')
      WHERE ai.M_AttributeSetInstance_ID=il.m_attributesetinstance_id
      AND a.Name                        ='Fournisseur'
      ) AS Fournisseur,
      XX_StartegicalProduct,
      i.SALESREP_ID,
      i.C_DOCTYPE_ID,
      i.AD_ORG_ID,
      (SELECT o.AD_ORGTRX_ID
      FROM c_order o
      WHERE i.c_order_id=o.c_order_id
      ) AS ad_orgtrx_id,
      p.xx_laboratory_id,
      lt.rate,
      --COUNT(*) over (partition BY il.c_orderline_id) AS nb_doc,
      (
      SELECT COUNT(*)
      FROM m_inoutline ill
      WHERE ill.c_orderline_id=il.c_orderline_id
      )               AS nb_doc,
      ol.type         AS typevente,
      bpl.salesrep_id AS poste_id,
      (SELECT u.c_bpartner_id FROM AD_User u WHERE u.AD_User_ID = i.salesrep_id
      ) AS SalesRepTier,
      (SELECT u.c_bpartner_id FROM AD_User u WHERE u.AD_User_ID = bpl.salesrep_id
      ) AS SalesRepTier_poste_id,
      (SELECT u.XX_RecSupervisor_ID FROM AD_User u WHERE u.AD_User_ID=i.SALESREP_ID
      ) AS recSupr,
      (SELECT u.XX_RecSupervisor_ID
      FROM AD_User u
      WHERE u.AD_User_ID=bpl.salesrep_id
      ) AS recSupr_poste_id
    FROM m_inoutline il
    INNER JOIN m_inout i
    ON il.m_inout_id=i.m_inout_id
    INNER JOIN c_orderline ol
    ON ol.c_orderline_id=il.c_orderline_id
    INNER JOIN m_product p
    ON p.m_product_id=il.m_product_id
    INNER JOIN C_Bpartner bpl
    ON (bpl.c_bpartner_id=i.c_bpartner_id)
    LEFT OUTER JOIN xx_listetauxvaleur lt
    ON p.xx_listetauxvaleur_id = lt.xx_listetauxvaleur_id
    WHERE i.issotrx            ='Y'
      --AND p.m_attributeset_id   IS NOT NULL
    AND il.movementqty<>0
    ) CF
  LEFT OUTER JOIN c_bpartner bp
  ON (CF.Fournisseur=bp.c_bpartner_id)
  ORDER BY bp.name,
    documentno;

Answer 1

我在你的陈述中看到了很多连接。您确定您在这些连接中存在的所有字段上都有索引吗？我认为在执行此操作之前需要进行大量优化。

我还建议取消交换，因为这是一个过时的功能，其性能影响尚不清楚。

如果您希望在执行多个连接时在单个节点上优于 Oracle，这也不会发生。

Answer 2

考虑应能回答您的大部分问题的一般性能提示： https://www.gridgain.com/docs/latest/perf-troubleshooting-guide/general-perf-tips

要获得最佳性能，您需要将整个数据集保存在 RAM 中。如果资源不足，请考虑 Ignite 本机持久性而不是交换存储。

点燃 caching/querying 花费太多时间

Ignite caching/querying taking too much time

ignite

gridgain