Redshift：表信息查询无法通过 spark 工作

Question

我正在尝试运行使用数据块从 spark 代码中查询此查询：

select * from svv_table_info

但我收到此错误消息：

Exception in thread "main" java.sql.SQLException: Amazon Invalid operation: Specified types or functions (one per INFO message) not supported on Redshift tables.;

我为什么会收到这个有什么意见吗？

Answer 1

Postgres 系统类型 OID 中的视图 returns table_id。

psql=# \d+ svv_table_info
    Column     |     Type      | Modifiers | Storage  | Description
---------------+---------------+-----------+----------+-------------
 database      | text          |           | extended |
 schema        | text          |           | extended |
 table_id      | oid           |           | plain    |
 table         | text          |           | extended |
 encoded       | text          |           | extended |
 diststyle     | text          |           | extended |
 sortkey1      | text          |           | extended |
 max_varchar   | integer       |           | plain    |
 sortkey1_enc  | character(32) |           | extended |
 sortkey_num   | integer       |           | plain    |
 size          | bigint        |           | plain    |
 pct_used      | numeric(10,4) |           | main     |
 empty         | bigint        |           | plain    |
 unsorted      | numeric(5,2)  |           | main     |
 stats_off     | numeric(5,2)  |           | main     |
 tbl_rows      | numeric(38,0) |           | main     |
 skew_sortkey1 | numeric(19,2) |           | main     |
 skew_rows     | numeric(19,2) |           | main     |

您可以将其转换为 INTEGER，Spark 应该能够处理它。

SELECT database,schema,table_id::INT
      ,"table",encoded,diststyle,sortkey1
      ,max_varchar,sortkey1_enc,sortkey_num
      ,size,pct_used,empty,unsorted,stats_off
      ,tbl_rows,skew_sortkey1,skew_rows 
FROM svv_table_info;

Redshift：表信息查询无法通过 spark 工作

Redshift: tables info query not working via spark

scala

amazon-redshift

apache-spark

databricks