Spark with Pureconfig - 正确的maven shade插件配置

Question

我遇到了与这里描述的完全相同的问题：。上述问题的唯一答案似乎是合理的，但我正在使用 Maven 而不是 sbt，并且我未能将发布的解决方案从 sbt 转换为 Maven。

我试过类似下面的方法：

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.0.0</version>
<executions>
    <execution>
        <phase>package</phase>
        <goals>
            <goal>shade</goal>
        </goals>
    </execution>
</executions>
<configuration>
    <createDependencyReducedPom>false</createDependencyReducedPom>
    <relocations>
        <relocation>
            <pattern>com.chuusai:shapeless_2.11:2.3.2</pattern>
            <shadedPattern>com.matek.shaded.com.chuusai:shapeless_2.11:2.3.2</shadedPattern>
        </relocation>
        <relocation>
            <pattern>com.chuusai:shapeless_2.11:2.0.0</pattern>
            <shadedPattern>com.matek.shaded.com.chuusai:shapeless_2.11:2.0.0</shadedPattern>
        </relocation>
        <relocation>
            <pattern>com.github.pureconfig</pattern>
            <shadedPattern>com.matek.shaded.com.github.pureconfig</shadedPattern>
            <excludes>
                <exclude>com.chuusai:shapeless_2.11:2.3.2</exclude>
            </excludes>
            <includes>
                <include>com.matek.shaded.com.chuusai:shapeless_2.11:2.3.2</include>
            </includes>
        </relocation>
    </relocations>
</configuration>

但毫不奇怪，这不起作用（我什至不确定它是否正确）。如何指定 maven shade 插件配置以使其与 spark submit 一起工作？

Answer 1

我设法解决了这个问题。这实际上是我的错误，模式只是 shapeless 而不是 com.chuusai.shapeless。这有效：

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.0.0</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <createDependencyReducedPom>false</createDependencyReducedPom>
                    <relocations>
                        <relocation>
                            <pattern>shapeless</pattern>
                            <shadedPattern>com.matek.shaded.shapeless</shadedPattern>
                        </relocation>
                    </relocations>
                </configuration>
            </plugin>

Spark with Pureconfig - 正确的maven shade插件配置

Spark with Pureconfig - proper maven shade plugin configuration

scala

maven

apache-spark

sbt

maven-shade-plugin