使用 spark sql 在 sql 服务器上执行查询
execute query on sqlserver using spark sql
我正在尝试使用 spark sql.
获取 sql 服务器中架构中所有 table 的行数和列数
当我使用 sqoop 执行以下查询时,它给出了正确的结果。
sqoop eval --connect "jdbc:sqlserver://<hostname>;database=<dbname>" \
--username=<username> --password=<pwd> \
--query """SELECT
ta.name TableName ,
pa.rows RowCnt,
COUNT(ins.COLUMN_NAME) ColCnt FROM <db>.sys.tables ta INNER JOIN
<db>.sys.partitions pa ON pa.OBJECT_ID = ta.OBJECT_ID INNER JOIN
<db>.sys.schemas sc ON ta.schema_id = sc.schema_id join
<db>.INFORMATION_SCHEMA.COLUMNS ins on ins.TABLE_SCHEMA =sc.name and ins.TABLE_NAME=ta.name
WHERE ta.is_ms_shipped = 0 AND pa.index_id IN (1,0) and sc.name ='<schema>' GROUP BY sc.name, ta.name, pa.rows order by
TableName"""
但是当我尝试从 spark sql 执行相同的查询时,我收到一个错误“com.microsoft.sqlserver.jdbc.SQLServerException:关键字 'WHERE' 附近的语法不正确”
如果有人知道这个错误,请帮助我。
下面是我执行的spark sql命令
spark-shell --jars "/var/lib/sqoop/sqljdbc42.jar"
sqlContext.read.format("jdbc").option("url", "jdbc:sqlserver://<hostname>;database=<dbname>;user=<user>;password=<pwd>").option("dbtable", """(SELECT
ta.name TableName ,pa.rows RowCnt,
COUNT(ins.COLUMN_NAME) ColCnt FROM <db>.sys.tables ta INNER JOIN
<db>.sys.partitions pa ON pa.OBJECT_ID = ta.OBJECT_ID INNER JOIN
<db>.sys.schemas sc ON ta.schema_id = sc.schema_id join
<db>.INFORMATION_SCHEMA.COLUMNS ins on ins.TABLE_SCHEMA =sc.name and ins.TABLE_NAME=ta.name
WHERE ta.is_ms_shipped = 0 AND pa.index_id IN (1,0) and sc.name ='<schema>' GROUP BY sc.name,ta.name, pa.rows)""").option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver").load()
预期输出:
表名、行数、列数
table一、62、30
table B, 3846, 76
您的 Spark SQL 命令中的问题与 dbTable
选项有关。
dbTable
接受在 SQL 查询的 FROM
子句中有效的任何内容。例如,除了完整的 table 之外,您还可以在括号中使用子查询。但是,在括号中使用子查询时,它应该有一个别名。因此,您的命令应修改为,
sqlContext
.read
.format("jdbc")
.option("url", "jdbc:sqlserver://<hostname>;database=<dbname>;user=<user>;password=<pwd>")
.option("dbtable",
"""(SELECT
ta.name TableName ,
pa.rows RowCnt,
COUNT(ins.COLUMN_NAME) ColCnt
FROM <db>.sys.tables ta
INNER JOIN
<db>.sys.partitions pa
ON pa.OBJECT_ID = ta.OBJECT_ID
INNER JOIN
<db>.sys.schemas sc
ON ta.schema_id = sc.schema_id
JOIN
<db>.INFORMATION_SCHEMA.COLUMNS ins
ON ins.TABLE_SCHEMA = sc.name and ins.TABLE_NAME = ta.name
WHERE ta.is_ms_shipped = 0
AND pa.index_id IN (1,0)
AND sc.name ='<schema>'
GROUP BY sc.name,ta.name, pa.rows) as TEMP""")
.option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver")
.load()
只是预感。希望这对您有所帮助!
我正在尝试使用 spark sql.
获取 sql 服务器中架构中所有 table 的行数和列数当我使用 sqoop 执行以下查询时,它给出了正确的结果。
sqoop eval --connect "jdbc:sqlserver://<hostname>;database=<dbname>" \
--username=<username> --password=<pwd> \
--query """SELECT
ta.name TableName ,
pa.rows RowCnt,
COUNT(ins.COLUMN_NAME) ColCnt FROM <db>.sys.tables ta INNER JOIN
<db>.sys.partitions pa ON pa.OBJECT_ID = ta.OBJECT_ID INNER JOIN
<db>.sys.schemas sc ON ta.schema_id = sc.schema_id join
<db>.INFORMATION_SCHEMA.COLUMNS ins on ins.TABLE_SCHEMA =sc.name and ins.TABLE_NAME=ta.name
WHERE ta.is_ms_shipped = 0 AND pa.index_id IN (1,0) and sc.name ='<schema>' GROUP BY sc.name, ta.name, pa.rows order by
TableName"""
但是当我尝试从 spark sql 执行相同的查询时,我收到一个错误“com.microsoft.sqlserver.jdbc.SQLServerException:关键字 'WHERE' 附近的语法不正确” 如果有人知道这个错误,请帮助我。
下面是我执行的spark sql命令 spark-shell --jars "/var/lib/sqoop/sqljdbc42.jar"
sqlContext.read.format("jdbc").option("url", "jdbc:sqlserver://<hostname>;database=<dbname>;user=<user>;password=<pwd>").option("dbtable", """(SELECT
ta.name TableName ,pa.rows RowCnt,
COUNT(ins.COLUMN_NAME) ColCnt FROM <db>.sys.tables ta INNER JOIN
<db>.sys.partitions pa ON pa.OBJECT_ID = ta.OBJECT_ID INNER JOIN
<db>.sys.schemas sc ON ta.schema_id = sc.schema_id join
<db>.INFORMATION_SCHEMA.COLUMNS ins on ins.TABLE_SCHEMA =sc.name and ins.TABLE_NAME=ta.name
WHERE ta.is_ms_shipped = 0 AND pa.index_id IN (1,0) and sc.name ='<schema>' GROUP BY sc.name,ta.name, pa.rows)""").option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver").load()
预期输出:
表名、行数、列数
table一、62、30
table B, 3846, 76
您的 Spark SQL 命令中的问题与 dbTable
选项有关。
dbTable
接受在 SQL 查询的 FROM
子句中有效的任何内容。例如,除了完整的 table 之外,您还可以在括号中使用子查询。但是,在括号中使用子查询时,它应该有一个别名。因此,您的命令应修改为,
sqlContext
.read
.format("jdbc")
.option("url", "jdbc:sqlserver://<hostname>;database=<dbname>;user=<user>;password=<pwd>")
.option("dbtable",
"""(SELECT
ta.name TableName ,
pa.rows RowCnt,
COUNT(ins.COLUMN_NAME) ColCnt
FROM <db>.sys.tables ta
INNER JOIN
<db>.sys.partitions pa
ON pa.OBJECT_ID = ta.OBJECT_ID
INNER JOIN
<db>.sys.schemas sc
ON ta.schema_id = sc.schema_id
JOIN
<db>.INFORMATION_SCHEMA.COLUMNS ins
ON ins.TABLE_SCHEMA = sc.name and ins.TABLE_NAME = ta.name
WHERE ta.is_ms_shipped = 0
AND pa.index_id IN (1,0)
AND sc.name ='<schema>'
GROUP BY sc.name,ta.name, pa.rows) as TEMP""")
.option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver")
.load()
只是预感。希望这对您有所帮助!