如何提高Liquibase在Greenplum数据库上的性能

Question

我正在使用 postgresql 8.4-701.jdbc4 评估 Liquibase (3.3.2) 针对 Greenplum（类固醇上的 Postgres）数据库的使用，并且运行在尝试添加一个变更集时遇到性能问题(create table) 针对包含许多分区 table 的现有架构。我目前在第三个小时，不知道这需要多长时间。 JdbcSnapshot 进程是一次性的吗？或者我会在部署每个变更集时遇到这种类型的性能吗？

要应用更改，我正在执行：

mvn resources:resources liquibase:update

pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.cobalt.bi</groupId>
    <artifactId>liquibase</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>liquibase</name>
    <description>liquibase</description>

    <dependencies>

<dependency>
    <groupId>postgresql</groupId>
    <artifactId>postgresql</artifactId>
    <version>9.1-901.jdbc4</version>
</dependency>


        <dependency>
            <groupId>com.oracle</groupId>
            <artifactId>ojdbc14</artifactId>
            <version>10.2.0.4.0</version>
        </dependency>

        <dependency>
            <groupId>org.liquibase</groupId>
            <artifactId>liquibase-core</artifactId>
            <version>3.3.2</version>
            <scope>test</scope>
        </dependency>

    </dependencies>

    <build>
          <plugins>

            <plugin>
                <groupId>org.liquibase</groupId>
                <artifactId>liquibase-maven-plugin</artifactId>
                <version>3.0.2</version>
                <configuration>
                  <promptOnNonLocalDatabase>false</promptOnNonLocalDatabase>
                  <propertyFileWillOverride>true</propertyFileWillOverride>
                  <propertyFile>target/classes/liquibase.properties</propertyFile>
                </configuration>

                <executions>
                  <execution>
                    <phase>process-resources</phase>
                    <goals>
                      <goal>update</goal>
                    </goals>
                  </execution>
                </executions>
            </plugin>
          </plugins>

          <filters>
            <filter>${user.home}/.liquibase.properties</filter>
          </filters>

          <resources>
            <resource>
              <directory>src/main/resources</directory>
              <filtering>true</filtering>
            </resource>
          </resources>

    </build>

</project>

$HOME/.liquibase.properties:

db.fmXX.user=szalwinb
db.fmXX.database=fmd1
db.fmXX.host=host-name-for-gp-db
db.fmXX.port=5432

liquibase.verbose: true
liquibase.dropFirst: false
liquibase.changeLogFile: src/main/resources/db/master.xml
liquibase.contexts:
liquibase.driver: org.postgresql.Driver
liquibase.url: jdbc:postgresql://${db.fmXX.host}:${db.fmXX.port}/${db.fmXX.database}
liquibase.username: ${db.fmXX.user}
liquibase.password: ${db.fmXX.user}
liquibase.defaultCatalogName: ${db.fmXX.user}
liquibase.defaultSchemaName: ${db.fmXX.user}
liquibase.logging: debug

使用 JConsole 查看堆栈跟踪，我看到：

Name: main
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:152)
java.net.SocketInputStream.read(SocketInputStream.java:122)
org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:135)
org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:104)
org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
org.postgresql.core.PGStream.ReceiveChar(PGStream.java:259)
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1620)
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
   - locked org.postgresql.core.v3.QueryExecutorImpl@11d1532d
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353)
org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:252)
org.postgresql.jdbc2.AbstractJdbc2DatabaseMetaData.getTables(AbstractJdbc2DatabaseMetaData.java:2062)
liquibase.snapshot.JdbcDatabaseSnapshot$CachingDatabaseMetaData.getTables(JdbcDatabaseSnapshot.java:173)
liquibase.snapshot.jvm.TableSnapshotGenerator.snapshotObject(TableSnapshotGenerator.java:33)
liquibase.snapshot.jvm.JdbcSnapshotGenerator.snapshot(JdbcSnapshotGenerator.java:59)
liquibase.snapshot.SnapshotGeneratorChain.snapshot(SnapshotGeneratorChain.java:42)
liquibase.snapshot.jvm.JdbcSnapshotGenerator.snapshot(JdbcSnapshotGenerator.java:62)
liquibase.snapshot.SnapshotGeneratorChain.snapshot(SnapshotGeneratorChain.java:42)
liquibase.snapshot.jvm.JdbcSnapshotGenerator.snapshot(JdbcSnapshotGenerator.java:62)
liquibase.snapshot.SnapshotGeneratorChain.snapshot(SnapshotGeneratorChain.java:42)
liquibase.snapshot.jvm.JdbcSnapshotGenerator.snapshot(JdbcSnapshotGenerator.java:62)
liquibase.snapshot.SnapshotGeneratorChain.snapshot(SnapshotGeneratorChain.java:42)
liquibase.snapshot.jvm.JdbcSnapshotGenerator.snapshot(JdbcSnapshotGenerator.java:62)
liquibase.snapshot.SnapshotGeneratorChain.snapshot(SnapshotGeneratorChain.java:42)
liquibase.snapshot.jvm.JdbcSnapshotGenerator.snapshot(JdbcSnapshotGenerator.java:62)
liquibase.snapshot.SnapshotGeneratorChain.snapshot(SnapshotGeneratorChain.java:42)
liquibase.snapshot.jvm.JdbcSnapshotGenerator.snapshot(JdbcSnapshotGenerator.java:62)
liquibase.snapshot.SnapshotGeneratorChain.snapshot(SnapshotGeneratorChain.java:42)
liquibase.snapshot.DatabaseSnapshot.include(DatabaseSnapshot.java:75)
liquibase.snapshot.DatabaseSnapshot.replaceObject(DatabaseSnapshot.java:144)
liquibase.snapshot.DatabaseSnapshot.replaceObject(DatabaseSnapshot.java:157)
liquibase.snapshot.DatabaseSnapshot.includeNestedObjects(DatabaseSnapshot.java:107)
liquibase.snapshot.DatabaseSnapshot.include(DatabaseSnapshot.java:94)
liquibase.snapshot.SnapshotGeneratorFactory.createSnapshot(SnapshotGeneratorFactory.java:128)
liquibase.snapshot.SnapshotGeneratorFactory.createSnapshot(SnapshotGeneratorFactory.java:121)
liquibase.snapshot.SnapshotGeneratorFactory.createSnapshot(SnapshotGeneratorFactory.java:109)
liquibase.snapshot.SnapshotGeneratorFactory.has(SnapshotGeneratorFactory.java:99)
liquibase.snapshot.SnapshotGeneratorFactory.hasDatabaseChangeLogLockTable(SnapshotGeneratorFactory.java:171)
liquibase.database.AbstractJdbcDatabase.hasDatabaseChangeLogLockTable(AbstractJdbcDatabase.java:749)
liquibase.database.AbstractJdbcDatabase.checkDatabaseChangeLogLockTable(AbstractJdbcDatabase.java:775)
liquibase.lockservice.LockServiceImpl.acquireLock(LockServiceImpl.java:95)
liquibase.lockservice.LockServiceImpl.waitForLock(LockServiceImpl.java:62)
liquibase.Liquibase.update(Liquibase.java:123)
org.liquibase.maven.plugins.LiquibaseUpdate.doUpdate(LiquibaseUpdate.java:31)
org.liquibase.maven.plugins.AbstractLiquibaseUpdateMojo.performLiquibaseTask(AbstractLiquibaseUpdateMojo.java:24)
org.liquibase.maven.plugins.AbstractLiquibaseMojo.execute(AbstractLiquibaseMojo.java:328)
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:347)
org.apache.maven.DefaultMaven.execute(DefaultMaven.java:154)
org.apache.maven.cli.MavenCli.execute(MavenCli.java:582)
org.apache.maven.cli.MavenCli.doMain(MavenCli.java:214)
org.apache.maven.cli.MavenCli.main(MavenCli.java:158)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)

检查数据库服务器日志，我看到每个 table.

都会执行此查询

2015-02-16 10:14:41.232988 PST,"szalwinb","fmd1",p21417,th632498080,"172.30.133.118","60261",2015-02-16 07:05:46 PST,93508959,con1317174,cmd41041,seg-1,,dx1748316,x93508959,sx1,"LOG","00000","execute <unnamed>: SELECT NULL AS TABLE_CAT, n.nspname AS TABLE_SCHEM, c.relname AS TABLE_NAME,  CASE n.nspname ~ '^pg_' OR n.nspname = 'information_schema'  WHEN true THEN CASE  WHEN n.nspname = 'pg_catalog' OR n.nspname = 'information_schema' THEN CASE c.relkind   WHEN 'r' THEN 'SYSTEM TABLE'   WHEN 'v' THEN 'SYSTEM VIEW'   WHEN 'i' THEN 'SYSTEM INDEX'   ELSE NULL   END  WHEN n.nspname = 'pg_toast' THEN CASE c.relkind   WHEN 'r' THEN 'SYSTEM TOAST TABLE'   WHEN 'i' THEN 'SYSTEM TOAST INDEX'   ELSE NULL   END  ELSE CASE c.relkind   WHEN 'r' THEN 'TEMPORARY TABLE'   WHEN 'i' THEN 'TEMPORARY INDEX'   ELSE NULL   END  END  WHEN false THEN CASE c.relkind  WHEN 'r' THEN 'TABLE'  WHEN 'i' THEN 'INDEX'  WHEN 'S' THEN 'SEQUENCE'  WHEN 'v' THEN 'VIEW'  ELSE NULL  END  ELSE NULL  END  AS TABLE_TYPE, d.description AS REMARKS  FROM pg_catalog.pg_namespace n, pg_catalog.pg_class c  LEFT JOIN pg_catalog.pg_description d ON (c.oid = d.objoid AND d.objsubid = 0)  LEFT JOIN pg_catalog.pg_class dc ON (d.classoid=dc.oid AND dc.relname='pg_class')  LEFT JOIN pg_catalog.pg_namespace dn ON (dn.oid=dc.relnamespace AND dn.nspname='pg_catalog')  WHERE c.relnamespace = n.oid  AND n.nspname LIKE 'szalwin'  AND c.relname LIKE 'customers_1_prt_p_20140708'  AND (false  OR ( c.relkind = 'r' AND n.nspname !~ '^pg_' AND n.nspname <> 'information_schema' ) )  ORDER BY TABLE_TYPE,TABLE_SCHEM,TABLE_NAME",,,,,,"SELECT NULL AS TABLE_CAT, n.nspname AS TABLE_SCHEM, c.relname AS TABLE_NAME,  CASE n.nspname ~ '^pg_' OR n.nspname = 'information_schema'  WHEN true THEN CASE  WHEN n.nspname = 'pg_catalog' OR n.nspname = 'information_schema' THEN CASE c.relkind   WHEN 'r' THEN 'SYSTEM TABLE'   WHEN 'v' THEN 'SYSTEM VIEW'   WHEN 'i' THEN 'SYSTEM INDEX'   ELSE NULL   END  WHEN n.nspname = 'pg_toast' THEN CASE c.relkind   WHEN 'r' THEN 'SYSTEM TOAST TABLE'   WHEN 'i' THEN 'SYSTEM TOAST INDEX'   ELSE NULL   END  ELSE CASE c.relkind   WHEN 'r' THEN 'TEMPORARY TABLE'   WHEN 'i' THEN 'TEMPORARY INDEX'   ELSE NULL   END  END  WHEN false THEN CASE c.relkind  WHEN 'r' THEN 'TABLE'  WHEN 'i' THEN 'INDEX'  WHEN 'S' THEN 'SEQUENCE'  WHEN 'v' THEN 'VIEW'  ELSE NULL  END  ELSE NULL  END  AS TABLE_TYPE, d.description AS REMARKS  FROM pg_catalog.pg_namespace n, pg_catalog.pg_class c  LEFT JOIN pg_catalog.pg_description d ON (c.oid = d.objoid AND d.objsubid = 0)  LEFT JOIN pg_catalog.pg_class dc ON (d.classoid=dc.oid AND dc.relname='pg_class')  LEFT JOIN pg_catalog.pg_namespace dn ON (dn.oid=dc.relnamespace AND dn.nspname='pg_catalog')  WHERE c.relnamespace = n.oid  AND n.nspname LIKE 'szalwinb'  AND c.relname LIKE 'customers_1_prt_p_20140708'  AND (false  OR ( c.relkind = 'r' AND n.nspname !~ '^pg_' AND n.nspname <> 'information_schema' ) )  ORDER BY TABLE_TYPE,TABLE_SCHEM,TABLE_NAME ",0,,"postgres.c",2625,

Answer 1

Liquibase 会在启动时做一个快照以确定数据库更新日志 table 是否存在以及它是否被正确定义。但是，快照逻辑应该仅限于查看那个 table。它不应该螺旋上升到快照一切。

这可能是特定于 postgres 或什至特定于 greenplumb 的问题。我创建了 https://liquibase.jira.com/browse/CORE-2259 来进一步研究它。

Answer 2

操作错误。 pom 的依赖项部分请求一个版本的 liquibase，而 build/plugin 部分请求一个不同的早期版本。我更新了 pom.xml 以在两个部分中使用相同的版本并且一切正常。

[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:55 min
[INFO] Finished at: 2015-02-25T21:26:33-08:00
[INFO] Final Memory: 12M/648M
[INFO] ------------------------------------------------------------------------

如何提高Liquibase在Greenplum数据库上的性能

How to improve performance of Liquibase on Greenplum database

postgresql

liquibase

greenplum