SSIS 无法保存包并重新启动 Visual Studio
SSIS failing to save packages and reboots Visual Studio
这是我第一次使用 SSIS，请多多包涵...
我正在使用 SSIS 将 tables 从 Oracle 迁移到 SSMS，我正在尝试传输一些非常大的 tables（超过 5000 万行）。当我只是想保存包时，SSIS 现在完全冻结并重新启动 VS（甚至 运行ning 它）。它不断返回内存不足的错误，但是，我在远程服务器上工作，该服务器的 RAM 远远超过 运行 此程序包所需的内存。
尝试保存时出现错误消息
我唯一能想到的是，当这个包试图 运行 时，我的以太网 Kbps 在包启动时就飙升了。可能需要更新我的管道？
以太网图
此外，我最大的 table 将在导入时因 BYTE 大小而失败（同样，几乎没有用完服务器上的所有内存）。我们正在使用 ODBC 源，因为这是我们能够让其他大型 table 上传超过 100 万行的唯一方法。
我已经尝试创建一个临时缓冲文件来帮助缓解内存压力，但没有任何改变。我已将 AutoAdjustBufferSize 更改为 True，结果没有变化。还更改了 DefaultBufferMaxRows 和 DefaultBufferSize.. 没有变化。
错误 运行 大 TABLE:
Information: 0x4004300C at SRC_STG_TABLENAME, SSIS.Pipeline: Execute
phase is beginning.
Information: 0x4004800D at SRC_STG_TABLENAME: The buffer manager
failed a memory allocation call for 810400000 bytes, but was unable
to swap out any buffers to relieve memory pressure. 2 buffers were
considered and 2 were locked.
Either not enough memory is available to the pipeline because not
enough are installed, other processes were using it, or too many
buffers are locked.
Information: 0x4004800F at SRC_STG_TABLENAME: Buffer manager
allocated 1548 megabyte(s) in 2 physical buffer(s).
Information: 0x40048010 at SRC_STG_TABLENAME: Component "ODBC
Source" (60) owns 775 megabyte(s) physical buffer.
Information: 0x4004800D at SRC_STG_TABLENAME: The buffer manager
failed a memory allocation call for 810400000 bytes, but was unable
to swap out any buffers to relieve memory pressure. 2 buffers were
considered and 2 were locked.
Either not enough memory is available to the pipeline because not
enough are installed, other processes were using it, or too many
buffers are locked.
Information: 0x4004800F at SRC_STG_TABLENAME: Buffer manager
allocated 1548 megabyte(s) in 2 physical buffer(s).
Information: 0x40048010 at SRC_STG_TABLENAME: Component "ODBC
Source" (60) owns 775 megabyte(s) physical buffer.
Information: 0x4004800D at SRC_STG_TABLENAME: The buffer manager
failed a memory allocation call for 810400000 bytes, but was unable
to swap out any buffers to relieve memory pressure. 2 buffers were
considered and 2 were locked.
Either not enough memory is available to the pipeline because not
enough are installed, other processes were using it, or too many
buffers are locked.
Information: 0x4004800F at SRC_STG_TABLENAME: Buffer manager
allocated 1548 megabyte(s) in 2 physical buffer(s).
Information: 0x40048010 at SRC_STG_TABLENAME: Component "ODBC
Source" (60) owns 775 megabyte(s) physical buffer.
Error: 0xC0047012 at SRC_STG_TABLENAME: A buffer failed while
allocating 810400000 bytes.
Error: 0xC0047011 at SRC_STG_TABLENAME: The system reports 26
percent memory load. There are 68718940160 bytes of physical memory
with 50752466944 bytes free. There are 4294836224 bytes of virtual
memory with 914223104 bytes free. The paging file has 84825067520
bytes with 61915041792 bytes free.
Information: 0x4004800F at SRC_STG_TABLENAME: Buffer manager
allocated 1548 megabyte(s) in 2 physical buffer(s).
Information: 0x40048010 at SRC_STG_TABLENAME: Component "ODBC
Source" (60) owns 775 megabyte(s) physical buffer.
Error: 0x279 at SRC_STG_TABLENAME, ODBC Source [60]: Failed to add
row to output buffer.
Error: 0x384 at SRC_STG_TABLENAME, ODBC Source [60]: Open Database
Connectivity (ODBC) error occurred.
Error: 0xC0047038 at SRC_STG_TABLENAME, SSIS.Pipeline: SSIS Error
Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on ODBC Source
returned error code 0x80004005. The component returned a failure code
when the pipeline engine called PrimeOutput(). The meaning of the
failure code is defined by the component, but the error is fatal and
the pipeline stopped executing. There may be error messages posted
before this with more information about the failure.
这真是耽误了我的工作。帮助！
您是否运行并行处理您的包裹？如果是，就转意甲。
您也可以尝试使用模运算将这个大 table 分成子集。看那个例子：
http://henkvandervalk.com/reading-as-fast-as-possible-from-a-table-with-ssis-part-ii
（例子中他是运行并联的，但是你可以把这个串联起来）
此外，如果您在 运行 正在 SQL 服务器实例的计算机上 运行 安装 SSIS 程序包，当您 运行 程序包时，将 SQL 服务器实例的最大服务器内存选项设置为较小的值。
这将增加可用内存。
我建议分块读取数据：
与其加载整个 table，不如尝试将数据分成块并将它们导入到 SQL 服务器。一段时间后，我 ，我会尝试重现它以适应 Oracle 语法：
分步指南
在此示例中，每个块包含 10000 行。
- 声明 2 个
Int32 类型的变量（@[User::RowCount] 和 @[User::IncrementValue]）
- 添加执行
select Count(*)命令的Execute SQL Task并将结果集存储到变量@[User::RowCount]
- 添加具有以下首选项的 For 循环：
- 在 for 循环容器内添加一个
Data flow task
- 在数据流任务中添加一个
ODBC Source 和 OLEDB Destination
- 在 ODBC Source select
SQL Command 选项中编写一个 SELECT * FROM TABLE 查询 *（仅检索元数据`
- 映射源和目标之间的列
- 返回
Control flow并点击Data flow task然后按F4查看属性window
在属性 window 中转到表达式并将以下表达式分配给 [ODBC Source].[SQLCommand] 属性：（有关更多信息，请参阅 How to pass SSIS variables in ODBC SQLCommand expression?)
"SELECT * FROM MYTABLE ORDER BY ID_COLUMN
OFFSET " + (DT_WSTR,50)@[User::IncrementValue] + "FETCH NEXT 10000 ROWS ONLY;"
其中 MYTABLE 是源 table 名称，IDCOLUMN 是您的主键或标识列。
控制流截图
参考文献
- ODBC Source - SQL Server
- How to pass SSIS variables in ODBC SQLCommand expression?
- HOW TO USE SSIS ODBC SOURCE AND DIFFERENCE BETWEEN OLE DB AND ODBC?
- How do I limit the number of rows returned by an Oracle query after ordering?
更新 1 - 其他可能的解决方法
在搜索类似问题时，我发现了一些您可以尝试的其他解决方法：
(1) 更改SQL服务器最大内存
SSIS: The Buffer Manager Failed a Memory Allocation Call
sp_configure 'show advanced options', 1;
GO
RECONFIGURE;
GO
sp_configure 'max server memory', 4096;
GO
RECONFIGURE;
GO
(2) 启用命名管道
-
- 进入控制面板->管理工具->计算机管理
- 在 SQL 实例的协议上 -> 设置命名管道 =
Enabled
- 重新启动 SQL 实例服务
- 之后尝试导入数据，它现在将分块获取数据，而不是一次获取所有数据。希望这对你们有用并节省你们的时间。
(3) 如果使用 SQL Server 2008 安装修补程序
更新 2 - 了解错误
在下面的MSDN link中，错误原因描述如下：
Virtual memory is a superset of physical memory. Processes in Windows typically do not specify which they are to use, as that would (greatly) inhibit how Windows can multitask. SSIS allocates virtual memory. If Windows is able to, all of these allocations are held in physical memory, where access is faster. However, if SSIS requests more memory than is physically available, then that virtual memory spills to disk, making the package operate orders of magnitude slower. And in worst cases, if there is not enough virtual memory in the system, then the package will fail.
这是我第一次使用 SSIS，请多多包涵... 我正在使用 SSIS 将 tables 从 Oracle 迁移到 SSMS，我正在尝试传输一些非常大的 tables（超过 5000 万行）。当我只是想保存包时，SSIS 现在完全冻结并重新启动 VS（甚至 运行ning 它）。它不断返回内存不足的错误，但是，我在远程服务器上工作，该服务器的 RAM 远远超过 运行 此程序包所需的内存。
尝试保存时出现错误消息
我唯一能想到的是，当这个包试图 运行 时，我的以太网 Kbps 在包启动时就飙升了。可能需要更新我的管道？
以太网图
此外，我最大的 table 将在导入时因 BYTE 大小而失败（同样，几乎没有用完服务器上的所有内存）。我们正在使用 ODBC 源，因为这是我们能够让其他大型 table 上传超过 100 万行的唯一方法。
我已经尝试创建一个临时缓冲文件来帮助缓解内存压力，但没有任何改变。我已将 AutoAdjustBufferSize 更改为 True，结果没有变化。还更改了 DefaultBufferMaxRows 和 DefaultBufferSize.. 没有变化。
错误 运行 大 TABLE:
Information: 0x4004300C at SRC_STG_TABLENAME, SSIS.Pipeline: Execute phase is beginning.
Information: 0x4004800D at SRC_STG_TABLENAME: The buffer manager failed a memory allocation call for 810400000 bytes, but was unable to swap out any buffers to relieve memory pressure. 2 buffers were considered and 2 were locked.
Either not enough memory is available to the pipeline because not enough are installed, other processes were using it, or too many buffers are locked.
Information: 0x4004800F at SRC_STG_TABLENAME: Buffer manager allocated 1548 megabyte(s) in 2 physical buffer(s).
Information: 0x40048010 at SRC_STG_TABLENAME: Component "ODBC Source" (60) owns 775 megabyte(s) physical buffer.
Information: 0x4004800D at SRC_STG_TABLENAME: The buffer manager failed a memory allocation call for 810400000 bytes, but was unable to swap out any buffers to relieve memory pressure. 2 buffers were considered and 2 were locked.
Either not enough memory is available to the pipeline because not enough are installed, other processes were using it, or too many buffers are locked.
Information: 0x4004800F at SRC_STG_TABLENAME: Buffer manager allocated 1548 megabyte(s) in 2 physical buffer(s).
Information: 0x40048010 at SRC_STG_TABLENAME: Component "ODBC Source" (60) owns 775 megabyte(s) physical buffer.
Information: 0x4004800D at SRC_STG_TABLENAME: The buffer manager failed a memory allocation call for 810400000 bytes, but was unable to swap out any buffers to relieve memory pressure. 2 buffers were considered and 2 were locked.
Either not enough memory is available to the pipeline because not enough are installed, other processes were using it, or too many buffers are locked.
Information: 0x4004800F at SRC_STG_TABLENAME: Buffer manager allocated 1548 megabyte(s) in 2 physical buffer(s).
Information: 0x40048010 at SRC_STG_TABLENAME: Component "ODBC Source" (60) owns 775 megabyte(s) physical buffer.
Error: 0xC0047012 at SRC_STG_TABLENAME: A buffer failed while allocating 810400000 bytes.
Error: 0xC0047011 at SRC_STG_TABLENAME: The system reports 26 percent memory load. There are 68718940160 bytes of physical memory with 50752466944 bytes free. There are 4294836224 bytes of virtual memory with 914223104 bytes free. The paging file has 84825067520 bytes with 61915041792 bytes free.
Information: 0x4004800F at SRC_STG_TABLENAME: Buffer manager allocated 1548 megabyte(s) in 2 physical buffer(s).
Information: 0x40048010 at SRC_STG_TABLENAME: Component "ODBC Source" (60) owns 775 megabyte(s) physical buffer.
Error: 0x279 at SRC_STG_TABLENAME, ODBC Source [60]: Failed to add row to output buffer.
Error: 0x384 at SRC_STG_TABLENAME, ODBC Source [60]: Open Database Connectivity (ODBC) error occurred.
Error: 0xC0047038 at SRC_STG_TABLENAME, SSIS.Pipeline: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on ODBC Source returned error code 0x80004005. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. There may be error messages posted before this with more information about the failure.
这真是耽误了我的工作。帮助！
您是否运行并行处理您的包裹？如果是，就转意甲。
您也可以尝试使用模运算将这个大 table 分成子集。看那个例子：
http://henkvandervalk.com/reading-as-fast-as-possible-from-a-table-with-ssis-part-ii
（例子中他是运行并联的，但是你可以把这个串联起来）
此外，如果您在 运行 正在 SQL 服务器实例的计算机上 运行 安装 SSIS 程序包，当您 运行 程序包时，将 SQL 服务器实例的最大服务器内存选项设置为较小的值。 这将增加可用内存。
我建议分块读取数据：
与其加载整个 table，不如尝试将数据分成块并将它们导入到 SQL 服务器。一段时间后，我
分步指南
在此示例中，每个块包含 10000 行。
- 声明 2 个
Int32类型的变量（@[User::RowCount]和@[User::IncrementValue]） - 添加执行
select Count(*)命令的Execute SQL Task并将结果集存储到变量@[User::RowCount]
- 添加具有以下首选项的 For 循环：
- 在 for 循环容器内添加一个
Data flow task - 在数据流任务中添加一个
ODBC Source和OLEDB Destination - 在 ODBC Source select
SQL Command选项中编写一个SELECT * FROM TABLE查询 *（仅检索元数据` - 映射源和目标之间的列
- 返回
Control flow并点击Data flow task然后按F4查看属性window 在属性 window 中转到表达式并将以下表达式分配给
[ODBC Source].[SQLCommand]属性：（有关更多信息，请参阅 How to pass SSIS variables in ODBC SQLCommand expression?)"SELECT * FROM MYTABLE ORDER BY ID_COLUMN OFFSET " + (DT_WSTR,50)@[User::IncrementValue] + "FETCH NEXT 10000 ROWS ONLY;"
其中 MYTABLE 是源 table 名称，IDCOLUMN 是您的主键或标识列。
控制流截图
参考文献
- ODBC Source - SQL Server
- How to pass SSIS variables in ODBC SQLCommand expression?
- HOW TO USE SSIS ODBC SOURCE AND DIFFERENCE BETWEEN OLE DB AND ODBC?
- How do I limit the number of rows returned by an Oracle query after ordering?
更新 1 - 其他可能的解决方法
在搜索类似问题时，我发现了一些您可以尝试的其他解决方法：
(1) 更改SQL服务器最大内存
SSIS: The Buffer Manager Failed a Memory Allocation Call
sp_configure 'show advanced options', 1; GO RECONFIGURE; GO sp_configure 'max server memory', 4096; GO RECONFIGURE; GO
(2) 启用命名管道
-
- 进入控制面板->管理工具->计算机管理
- 在 SQL 实例的协议上 -> 设置命名管道 =
Enabled - 重新启动 SQL 实例服务
- 之后尝试导入数据，它现在将分块获取数据，而不是一次获取所有数据。希望这对你们有用并节省你们的时间。
(3) 如果使用 SQL Server 2008 安装修补程序
更新 2 - 了解错误
在下面的MSDN link中，错误原因描述如下：
Virtual memory is a superset of physical memory. Processes in Windows typically do not specify which they are to use, as that would (greatly) inhibit how Windows can multitask. SSIS allocates virtual memory. If Windows is able to, all of these allocations are held in physical memory, where access is faster. However, if SSIS requests more memory than is physically available, then that virtual memory spills to disk, making the package operate orders of magnitude slower. And in worst cases, if there is not enough virtual memory in the system, then the package will fail.