当使用 SQL 服务器代理启动时，SSIS 包无法使用 C# 脚本任务处理所有行
SSIS package fails to process all rows with C# Script task when started with SQL Server Agent
我需要构建一个 SSIS 包来发送 HTML 格式的电子邮件，然后将电子邮件保存为 tiff 文件。我创建了一个脚本任务来处理必要的记录，然后将 HTML 代码转换为 tiff。我已将该过程拆分为单独的包，电子邮件发送工作正常，将 HTML 转换为 tiff 是导致问题的原因。
当 运行 手动打包时，它将毫无问题地处理所有文件。我的测试目前大约有 315 个文件，这需要能够处理至少 1,000 个，同时能够一次发送多达 10,000 个。问题是当我将包设置为使用 SQL 服务器代理执行时，它在 207 个文件处停止。包已部署到 SSIS 目录中的 SQL Server 2019
到目前为止我尝试了什么
我开始将脚本放置在 SSIS 包中并部署到服务器并从一个步骤调用包（99.999999% 的时间适用于所有包）尝试了 32 位和 64 位运行时。在查看执行报告时，从来没有任何错误消息只是意外终止。单击目录并执行包时，它将处理所有文件。 SQL 服务器代理正在使用代理，我还使用我的管理员凭据创建了另一个代理帐户来测试该帐户是否存在任何问题。
创建了另一个包来调用包并使用执行包任务调用第一个包，结果相同 207 个文件。将执行进程任务更改为执行 SQL 任务，并尝试创建用于手动启动目录 207 文件中的程序包的脚本。尝试通过其他 SSIS 包和 SQL 服务器代理从命令行直接执行脚本，结果相同 207 个文件。如果我直接在 SQL Server Agent 之外尝试这些方法中的任何一个，进程运行没有问题。
我将脚本任务转换为控制台应用程序，它可以处理所有文件。当从 SQL 服务器代理的任何方法调用可执行文件时，它再次停止在 207 个文件处。
我咨询了公司的 DBA 和系统团队，他们没有发现任何可能导致此错误的原因。似乎存在某种类型的限制，无论执行方法如何 SQL Server Agent 都不允许。我提到过查看第三方应用程序，但被告知不行。
我在下面包含了我能够拼凑的代码。我是一名 SQL 开发人员，因此 C# 不在我的知识库之内。有没有一种方法可以优化代码，使其只使用一个线程或在每个字母之间进行清理。在某些时候可能需要创建超过一万个字母。
更新
我已将代码替换为新的更新代码。电子邮件和图像创建都包括在内，因为这是最终产品必须做的。发送电子邮件时，有一个主要和次要电子邮件地址，并且根据使用的电子邮件地址，它会更改电子邮件正文包含的内容。在查看代码时，有一段 try catch 会在指示时发送到主节点，如果失败则发送到辅助节点。我猜想有一种更简洁的方法来完成该部分，但这是我在 SQL 中处理其他所有内容时的第一个程序。
感谢您的所有建议和帮助。
更新代码
using System;
using System.Data;
using System.Windows.Forms;
using System.Data.SqlClient;
using System.Threading;
using System.Threading.Tasks;
using System.Drawing;
using System.IO;
using System.Net.Mail;
using System.Configuration;
using System.Diagnostics;
namespace DocCreator
{
class Program
{
static void Main(string[] args)
{
var connSSIS = ConfigurationManager.ConnectionStrings["ssis_ssrs"].ConnectionString;
int logid = 0;
int count = 0;
string previous = "";
try
{
var connDataExtract = ConfigurationManager.ConnectionStrings["dataExtract"].ConnectionString;
string archiveFolder = @"Folder Path";
string project = "Mobile Pay";
string logProc = "[dbo].[usp_EmailSentandError]";
int rowCount = 0;
DataTable dt = new DataTable();
using (SqlConnection connLog = new SqlConnection(connDataExtract))
{
connLog.Open();
SqlCommand command = new SqlCommand("etl.usp_GetEmailListID", connLog)
{
CommandType = CommandType.StoredProcedure
};
command.Parameters.Add("@Project", SqlDbType.NText).Value = project;
command.Parameters.Add("@NullValue", SqlDbType.Int).Value = 0;
using (SqlDataReader dr = command.ExecuteReader())
{
dt.Load(dr);
}
connLog.Close();
}
foreach (DataRow dr in dt.Rows)
//Parallel.ForEach(dt.AsEnumerable(), dr =>
{
try
{
var emailID = dr["Email_ID"];
var usePrimary = dr["UsePrimaryEmail"];
try
{
if ((bool)usePrimary)
{
try
{
var dp = GetDataPoints(connDataExtract, (int)emailID, (bool)usePrimary);
string indexfqdn = Path.Combine(archiveFolder, dp.IndexFile);
string filefqdn = Path.Combine(archiveFolder, dp.ArchiveFileName);
string mailBody = GetEmailBody(connDataExtract, dp.SqlProc, (int)emailID, dp.EmailBody_id);
SendEmail(dp.EmailFrom, dp.EmailSubject, dp.Email, mailBody);
Archive(mailBody, filefqdn, dp.FileWidth, dp.FileHeight, dp.IndexFileInsert, indexfqdn, dp.ArchiveFile);
LogEmail(connDataExtract, logProc, (int)emailID, dp.EmailBody_id, 1, 1, "", 0);
rowCount++;
}
catch (Exception e)
{
try
{
var dp = GetDataPoints(connDataExtract, (int)emailID, false);
string indexfqdn = Path.Combine(archiveFolder, dp.IndexFile);
string filefqdn = Path.Combine(archiveFolder, dp.ArchiveFileName);
string mailBody = GetEmailBody(connDataExtract, dp.SqlProc, (int)emailID, dp.EmailBody_id);
SendEmail(dp.EmailFrom, dp.EmailSubject, dp.Email, mailBody);
Archive(mailBody, filefqdn, dp.FileWidth, dp.FileHeight, dp.IndexFileInsert, indexfqdn, dp.ArchiveFile);
LogEmail(connDataExtract, logProc, (int)emailID, dp.EmailBody_id, 0, 1, e.Message.ToString(), 1);
rowCount++;
}
catch (Exception e2)
{
LogEmail(connDataExtract, logProc, (int)emailID, 0, 0, 0, e2.Message.ToString(), 1);
Console.Clear();
Console.WriteLine(e2.Message);
Console.ReadLine();
}
}
}
else
{
try
{
var dp = GetDataPoints(connDataExtract, (int)emailID, (bool)usePrimary);
string indexfqdn = Path.Combine(archiveFolder, dp.IndexFile);
string filefqdn = Path.Combine(archiveFolder, dp.ArchiveFileName);
string mailBody = GetEmailBody(connDataExtract, dp.SqlProc, (int)emailID, dp.EmailBody_id);
SendEmail(dp.EmailFrom, dp.EmailSubject, dp.Email, mailBody);
Archive(mailBody, filefqdn, dp.FileWidth, dp.FileHeight, dp.IndexFileInsert, indexfqdn, dp.ArchiveFile);
LogEmail(connDataExtract, logProc, (int)emailID, dp.EmailBody_id, 0, 1, "", 0);
rowCount++;
}
catch (Exception e)
{
LogEmail(connDataExtract, logProc, (int)emailID, 0, 0, 0, e.Message.ToString(), 1);
Console.Clear();
Console.WriteLine(e.Message);
Console.ReadLine();
}
}
}
catch (Exception e)
{
LogEmail(connDataExtract, logProc, (int)emailID, 0, 0, 0, e.Message.ToString(), 1);
Console.Clear();
Console.WriteLine(e.Message);
Console.ReadLine();
}
}
catch (Exception e2)
{
//Console.WriteLine(e2.Message);
using (SqlConnection connLog2 = new SqlConnection(connSSIS))
{
connLog2.Open();
SqlCommand command2 = new SqlCommand("dbo.usp_InsertssisScriptTaskLog", connLog2)
{
CommandType = CommandType.StoredProcedure
};
command2.Parameters.Add("@PackageLogID", SqlDbType.Int).Value = ((ulong)logid);
command2.Parameters.Add("@ErrorMessage", SqlDbType.NText).Value = e2.Message.ToString();
command2.ExecuteNonQuery();
connLog2.Close();
}
}
count++;
int directThreadsCount = Process.GetCurrentProcess().Threads.Count;
Console.Clear();
Console.WriteLine(previous + Environment.NewLine+ count + " Memory usage: " + GC.GetTotalMemory(false) + " " + GC.GetTotalMemory(true) + " Threads: " + directThreadsCount);
previous = count + " Memory usage: " + GC.GetTotalMemory(false) + " " + GC.GetTotalMemory(true) + " Threads: " + directThreadsCount;
GC.Collect(1);
}
Console.WriteLine("Files have been created");
}
catch (Exception e)
{
//Console.WriteLine(e.Message);
using (SqlConnection connLog = new SqlConnection(connSSIS))
{
connLog.Open();
SqlCommand command = new SqlCommand("dbo.usp_InsertssisScriptTaskLog", connLog)
{
CommandType = CommandType.StoredProcedure
};
command.Parameters.Add("@PackageLogID", SqlDbType.Int).Value = ((ulong)logid);
command.Parameters.Add("@ErrorMessage", SqlDbType.NText).Value = e.Message.ToString();
command.ExecuteNonQuery();
connLog.Close();
}
}
}
public static
(int EmailBody_id, bool ArchiveFile, int FileHeight, int FileWidth, string IndexFile, string IndexFileInsert, string Email, string ArchiveFileName, string EmailFrom, string EmailSubject, string SqlProc)
GetDataPoints(string connDataExtract, int Email_ID, bool UsePrimary)
{
string dataExtract = connDataExtract;
int emailID = Email_ID;
bool usePri = UsePrimary;
using (SqlConnection connLog = new SqlConnection(dataExtract))
{
connLog.Open();
SqlCommand command = new SqlCommand("etl.usp_EmailGetDataPoints", connLog)
{
CommandType = CommandType.StoredProcedure
};
command.Parameters.Add("@Email_ID", SqlDbType.Int).Value = emailID;
command.Parameters.Add("@UsePrimary", SqlDbType.Bit).Value = usePri;
SqlDataReader sqlDataReader;
int EmailBody_id = 0;
bool ArchiveFile = true;
int FileHeight = 0;
int FileWidth = 0;
string IndexFile = "";
string IndexFileInsert = "";
string Email = "";
string ArchiveFileName = "";
string EmailFrom = "";
string EmailSubject = "";
string SqlProc = "";
sqlDataReader = command.ExecuteReader();
while (sqlDataReader.Read())
{
EmailBody_id = (int)sqlDataReader.GetValue(sqlDataReader.GetOrdinal("EmailBody_ID"));
ArchiveFile = (bool)sqlDataReader.GetValue(sqlDataReader.GetOrdinal("ArchiveFile"));
FileHeight = (int)sqlDataReader.GetValue(sqlDataReader.GetOrdinal("FileHeight"));
FileWidth = (int)sqlDataReader.GetValue(sqlDataReader.GetOrdinal("FileWidth"));
IndexFile = sqlDataReader.GetValue(sqlDataReader.GetOrdinal("IndexFile")).ToString();
IndexFileInsert = sqlDataReader.GetValue(sqlDataReader.GetOrdinal("IndexFileInsert")).ToString();
Email = sqlDataReader.GetValue(sqlDataReader.GetOrdinal("Email")).ToString();
ArchiveFileName = sqlDataReader.GetValue(sqlDataReader.GetOrdinal("ArchiveFileName")).ToString();
EmailFrom = sqlDataReader.GetValue(sqlDataReader.GetOrdinal("EmailFrom")).ToString();
EmailSubject = sqlDataReader.GetValue(sqlDataReader.GetOrdinal("EmailSubject")).ToString();
SqlProc = sqlDataReader.GetValue(sqlDataReader.GetOrdinal("SqlProc")).ToString();
}
connLog.Close();
return (
EmailBody_id,
ArchiveFile,
FileHeight,
FileWidth,
IndexFile,
IndexFileInsert,
Email,
ArchiveFileName,
EmailFrom,
EmailSubject,
SqlProc);
}
}
public static string GetEmailBody(string connDataExtract, string SQLProc, int Email_ID, int EmailBodyID)
{
string dataExtract = connDataExtract;
string proc = SQLProc;
int emailID = Email_ID;
int bodyID = EmailBodyID;
string MailBody;
using (SqlConnection connLog = new SqlConnection(dataExtract))
{
connLog.Open();
SqlCommand command = new SqlCommand(proc, connLog)
{
CommandType = CommandType.StoredProcedure
};
command.Parameters.Add("@Email_ID", SqlDbType.Int).Value = ((ulong)emailID);
command.Parameters.Add("@EmailBody_ID", SqlDbType.Int).Value = ((ulong)bodyID);
SqlDataReader dataReader;
string Output = "";
dataReader = command.ExecuteReader();
while (dataReader.Read())
{
Output = dataReader.GetValue(0).ToString();
}
connLog.Close();
MailBody = Output;
return MailBody;
}
}
public static void SendEmail(string emailFrom, string emailSubject, string emailTo, string mailBody)
{
string from = emailFrom;
string subject = emailSubject;
string to = emailTo;
string source = mailBody;
using (MailMessage myHtmlFormattedMail = new MailMessage())
{
MailAddress fromMail = new MailAddress(from);
myHtmlFormattedMail.From = fromMail;
myHtmlFormattedMail.Subject = subject;
myHtmlFormattedMail.Body = source;
foreach (var address in to.Split(new[] { ";" }, StringSplitOptions.RemoveEmptyEntries))
{
myHtmlFormattedMail.To.Add(address);
}
myHtmlFormattedMail.IsBodyHtml = true;
SmtpClient mySmtpClient = new SmtpClient();
mySmtpClient.Send(myHtmlFormattedMail);
}
}
public static void IndexFile(string indexFileInsert, string indexfqdn)
{
string insert = indexFileInsert;
string fqdn = indexfqdn;
try
{
if (!File.Exists(fqdn))
{
using (StreamWriter sw = File.CreateText(fqdn))
{
sw.WriteLine(insert);
}
}
using (StreamWriter sw = File.AppendText(fqdn))
{
sw.WriteLine(insert);
}
}
catch { }
}
public static void LogEmail(string databaseServer, string logProc, int email_ID, int emailBodyID, int primaryEmailUsed, int emailSent, string errorMessage, int errorExists)
{
string dataExtract = databaseServer;
string proc = logProc;
int emailID = email_ID;
int bodyID = emailBodyID;
int usePri = primaryEmailUsed;
int sent = emailSent;
string error = errorMessage;
int exists = errorExists;
using (SqlConnection connLog = new SqlConnection(dataExtract))
{
connLog.Open();
SqlCommand command = new SqlCommand(proc, connLog)
{
CommandType = CommandType.StoredProcedure
};
command.Parameters.Add("@Email_ID", SqlDbType.Int).Value = ((ulong)emailID);
command.Parameters.Add("@EmailBody_ID", SqlDbType.Int).Value = ((ulong)bodyID);
command.Parameters.Add("@PrimaryEmailUsed", SqlDbType.Int).Value = ((ulong)usePri);
command.Parameters.Add("@EmailSent", SqlDbType.Int).Value = ((ulong)sent);
command.Parameters.Add("@ErrorMessage", SqlDbType.NText).Value = error;
command.Parameters.Add("@ErrorExists", SqlDbType.Int).Value = ((ulong)exists);
command.ExecuteNonQuery();
connLog.Close();
}
}
public static void StartBrowser(string mailBody, string file, int width, int height, string fileInsert, string indexFile)
{
try
{
string source = mailBody;
string fqdn = file;
int w = width;
int h = height;
string insert = fileInsert;
string indexFQDN = indexFile;
IndexFile(insert, indexFQDN);
using (WebBrowser wb = new WebBrowser())
{
wb.ScrollBarsEnabled = false;
wb.Width = w;
wb.Height = h;
wb.Visible = false;
wb.DocumentCompleted +=
(sender, e) => WebBrowser_DocumentCompleted(sender, e, fqdn);
wb.DocumentText = source;
Application.Run();
}
}
finally
{
Application.Exit();
}
}
public static void WebBrowser_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e, string file)
{
string fqdn = file;
var webBrowser = (WebBrowser)sender;
using (Bitmap bitmap = new Bitmap(webBrowser.Width, webBrowser.Height))
{
webBrowser
.DrawToBitmap(bitmap, new Rectangle(0, 0, bitmap.Width, bitmap.Height));
bitmap.Save(fqdn, System.Drawing.Imaging.ImageFormat.Jpeg);
}
}
static void Wait(int milliseconds)
{
using (System.Windows.Forms.Timer timer1 = new System.Windows.Forms.Timer())
{
if (milliseconds == 0 || milliseconds < 0) return;
// Console.WriteLine("start wait timer");
timer1.Interval = milliseconds;
timer1.Enabled = true;
timer1.Start();
timer1.Tick += (s, e) =>
{
timer1.Enabled = false;
timer1.Stop();
// Console.WriteLine("stop wait timer");
};
while (timer1.Enabled)
{
System.Windows.Forms.Application.DoEvents();
}
}
}
public static void Archive(string emailBody, string file, int width, int height, string fileInsert, string indexFile, bool archiveFile)
{
string source = emailBody;
string fqdn = file;
int w = width;
int h = height;
string insert = fileInsert;
string indexFQDN = indexFile;
bool archive = archiveFile;
if (archive)
{
Thread tr = new Thread(() => StartBrowser(source, fqdn, w, h, insert, indexFQDN))
{
Name = "Fred",
IsBackground = true
};
tr.SetApartmentState(ApartmentState.STA);
tr.Start();
int wc = 800;
while (!File.Exists(file) || wc <= 0)
{
Wait(50);
wc--;
};
tr.Abort();
}
}
}
}
我已经解决了这个问题，它满足了我项目的需要。可能有更好的解决方案，但这确实有效。使用上面的代码我创建了一个 executable 文件并将结果集限制为前 100 个。创建了一个带有 For 循环的 ssis 包，该循环从暂存 table 中进行记录计数并启动 executable 文件。我进行了几次测试，并且能够超过项目要求的 10,000 个限制。
我需要构建一个 SSIS 包来发送 HTML 格式的电子邮件，然后将电子邮件保存为 tiff 文件。我创建了一个脚本任务来处理必要的记录，然后将 HTML 代码转换为 tiff。我已将该过程拆分为单独的包，电子邮件发送工作正常，将 HTML 转换为 tiff 是导致问题的原因。
当 运行 手动打包时，它将毫无问题地处理所有文件。我的测试目前大约有 315 个文件，这需要能够处理至少 1,000 个，同时能够一次发送多达 10,000 个。问题是当我将包设置为使用 SQL 服务器代理执行时，它在 207 个文件处停止。包已部署到 SSIS 目录中的 SQL Server 2019
到目前为止我尝试了什么
我开始将脚本放置在 SSIS 包中并部署到服务器并从一个步骤调用包（99.999999% 的时间适用于所有包）尝试了 32 位和 64 位运行时。在查看执行报告时，从来没有任何错误消息只是意外终止。单击目录并执行包时，它将处理所有文件。 SQL 服务器代理正在使用代理，我还使用我的管理员凭据创建了另一个代理帐户来测试该帐户是否存在任何问题。
创建了另一个包来调用包并使用执行包任务调用第一个包，结果相同 207 个文件。将执行进程任务更改为执行 SQL 任务，并尝试创建用于手动启动目录 207 文件中的程序包的脚本。尝试通过其他 SSIS 包和 SQL 服务器代理从命令行直接执行脚本，结果相同 207 个文件。如果我直接在 SQL Server Agent 之外尝试这些方法中的任何一个，进程运行没有问题。
我将脚本任务转换为控制台应用程序，它可以处理所有文件。当从 SQL 服务器代理的任何方法调用可执行文件时，它再次停止在 207 个文件处。
我咨询了公司的 DBA 和系统团队，他们没有发现任何可能导致此错误的原因。似乎存在某种类型的限制，无论执行方法如何 SQL Server Agent 都不允许。我提到过查看第三方应用程序，但被告知不行。
我在下面包含了我能够拼凑的代码。我是一名 SQL 开发人员，因此 C# 不在我的知识库之内。有没有一种方法可以优化代码，使其只使用一个线程或在每个字母之间进行清理。在某些时候可能需要创建超过一万个字母。
更新
我已将代码替换为新的更新代码。电子邮件和图像创建都包括在内，因为这是最终产品必须做的。发送电子邮件时，有一个主要和次要电子邮件地址，并且根据使用的电子邮件地址，它会更改电子邮件正文包含的内容。在查看代码时，有一段 try catch 会在指示时发送到主节点，如果失败则发送到辅助节点。我猜想有一种更简洁的方法来完成该部分，但这是我在 SQL 中处理其他所有内容时的第一个程序。
感谢您的所有建议和帮助。
更新代码
using System;
using System.Data;
using System.Windows.Forms;
using System.Data.SqlClient;
using System.Threading;
using System.Threading.Tasks;
using System.Drawing;
using System.IO;
using System.Net.Mail;
using System.Configuration;
using System.Diagnostics;
namespace DocCreator
{
class Program
{
static void Main(string[] args)
{
var connSSIS = ConfigurationManager.ConnectionStrings["ssis_ssrs"].ConnectionString;
int logid = 0;
int count = 0;
string previous = "";
try
{
var connDataExtract = ConfigurationManager.ConnectionStrings["dataExtract"].ConnectionString;
string archiveFolder = @"Folder Path";
string project = "Mobile Pay";
string logProc = "[dbo].[usp_EmailSentandError]";
int rowCount = 0;
DataTable dt = new DataTable();
using (SqlConnection connLog = new SqlConnection(connDataExtract))
{
connLog.Open();
SqlCommand command = new SqlCommand("etl.usp_GetEmailListID", connLog)
{
CommandType = CommandType.StoredProcedure
};
command.Parameters.Add("@Project", SqlDbType.NText).Value = project;
command.Parameters.Add("@NullValue", SqlDbType.Int).Value = 0;
using (SqlDataReader dr = command.ExecuteReader())
{
dt.Load(dr);
}
connLog.Close();
}
foreach (DataRow dr in dt.Rows)
//Parallel.ForEach(dt.AsEnumerable(), dr =>
{
try
{
var emailID = dr["Email_ID"];
var usePrimary = dr["UsePrimaryEmail"];
try
{
if ((bool)usePrimary)
{
try
{
var dp = GetDataPoints(connDataExtract, (int)emailID, (bool)usePrimary);
string indexfqdn = Path.Combine(archiveFolder, dp.IndexFile);
string filefqdn = Path.Combine(archiveFolder, dp.ArchiveFileName);
string mailBody = GetEmailBody(connDataExtract, dp.SqlProc, (int)emailID, dp.EmailBody_id);
SendEmail(dp.EmailFrom, dp.EmailSubject, dp.Email, mailBody);
Archive(mailBody, filefqdn, dp.FileWidth, dp.FileHeight, dp.IndexFileInsert, indexfqdn, dp.ArchiveFile);
LogEmail(connDataExtract, logProc, (int)emailID, dp.EmailBody_id, 1, 1, "", 0);
rowCount++;
}
catch (Exception e)
{
try
{
var dp = GetDataPoints(connDataExtract, (int)emailID, false);
string indexfqdn = Path.Combine(archiveFolder, dp.IndexFile);
string filefqdn = Path.Combine(archiveFolder, dp.ArchiveFileName);
string mailBody = GetEmailBody(connDataExtract, dp.SqlProc, (int)emailID, dp.EmailBody_id);
SendEmail(dp.EmailFrom, dp.EmailSubject, dp.Email, mailBody);
Archive(mailBody, filefqdn, dp.FileWidth, dp.FileHeight, dp.IndexFileInsert, indexfqdn, dp.ArchiveFile);
LogEmail(connDataExtract, logProc, (int)emailID, dp.EmailBody_id, 0, 1, e.Message.ToString(), 1);
rowCount++;
}
catch (Exception e2)
{
LogEmail(connDataExtract, logProc, (int)emailID, 0, 0, 0, e2.Message.ToString(), 1);
Console.Clear();
Console.WriteLine(e2.Message);
Console.ReadLine();
}
}
}
else
{
try
{
var dp = GetDataPoints(connDataExtract, (int)emailID, (bool)usePrimary);
string indexfqdn = Path.Combine(archiveFolder, dp.IndexFile);
string filefqdn = Path.Combine(archiveFolder, dp.ArchiveFileName);
string mailBody = GetEmailBody(connDataExtract, dp.SqlProc, (int)emailID, dp.EmailBody_id);
SendEmail(dp.EmailFrom, dp.EmailSubject, dp.Email, mailBody);
Archive(mailBody, filefqdn, dp.FileWidth, dp.FileHeight, dp.IndexFileInsert, indexfqdn, dp.ArchiveFile);
LogEmail(connDataExtract, logProc, (int)emailID, dp.EmailBody_id, 0, 1, "", 0);
rowCount++;
}
catch (Exception e)
{
LogEmail(connDataExtract, logProc, (int)emailID, 0, 0, 0, e.Message.ToString(), 1);
Console.Clear();
Console.WriteLine(e.Message);
Console.ReadLine();
}
}
}
catch (Exception e)
{
LogEmail(connDataExtract, logProc, (int)emailID, 0, 0, 0, e.Message.ToString(), 1);
Console.Clear();
Console.WriteLine(e.Message);
Console.ReadLine();
}
}
catch (Exception e2)
{
//Console.WriteLine(e2.Message);
using (SqlConnection connLog2 = new SqlConnection(connSSIS))
{
connLog2.Open();
SqlCommand command2 = new SqlCommand("dbo.usp_InsertssisScriptTaskLog", connLog2)
{
CommandType = CommandType.StoredProcedure
};
command2.Parameters.Add("@PackageLogID", SqlDbType.Int).Value = ((ulong)logid);
command2.Parameters.Add("@ErrorMessage", SqlDbType.NText).Value = e2.Message.ToString();
command2.ExecuteNonQuery();
connLog2.Close();
}
}
count++;
int directThreadsCount = Process.GetCurrentProcess().Threads.Count;
Console.Clear();
Console.WriteLine(previous + Environment.NewLine+ count + " Memory usage: " + GC.GetTotalMemory(false) + " " + GC.GetTotalMemory(true) + " Threads: " + directThreadsCount);
previous = count + " Memory usage: " + GC.GetTotalMemory(false) + " " + GC.GetTotalMemory(true) + " Threads: " + directThreadsCount;
GC.Collect(1);
}
Console.WriteLine("Files have been created");
}
catch (Exception e)
{
//Console.WriteLine(e.Message);
using (SqlConnection connLog = new SqlConnection(connSSIS))
{
connLog.Open();
SqlCommand command = new SqlCommand("dbo.usp_InsertssisScriptTaskLog", connLog)
{
CommandType = CommandType.StoredProcedure
};
command.Parameters.Add("@PackageLogID", SqlDbType.Int).Value = ((ulong)logid);
command.Parameters.Add("@ErrorMessage", SqlDbType.NText).Value = e.Message.ToString();
command.ExecuteNonQuery();
connLog.Close();
}
}
}
public static
(int EmailBody_id, bool ArchiveFile, int FileHeight, int FileWidth, string IndexFile, string IndexFileInsert, string Email, string ArchiveFileName, string EmailFrom, string EmailSubject, string SqlProc)
GetDataPoints(string connDataExtract, int Email_ID, bool UsePrimary)
{
string dataExtract = connDataExtract;
int emailID = Email_ID;
bool usePri = UsePrimary;
using (SqlConnection connLog = new SqlConnection(dataExtract))
{
connLog.Open();
SqlCommand command = new SqlCommand("etl.usp_EmailGetDataPoints", connLog)
{
CommandType = CommandType.StoredProcedure
};
command.Parameters.Add("@Email_ID", SqlDbType.Int).Value = emailID;
command.Parameters.Add("@UsePrimary", SqlDbType.Bit).Value = usePri;
SqlDataReader sqlDataReader;
int EmailBody_id = 0;
bool ArchiveFile = true;
int FileHeight = 0;
int FileWidth = 0;
string IndexFile = "";
string IndexFileInsert = "";
string Email = "";
string ArchiveFileName = "";
string EmailFrom = "";
string EmailSubject = "";
string SqlProc = "";
sqlDataReader = command.ExecuteReader();
while (sqlDataReader.Read())
{
EmailBody_id = (int)sqlDataReader.GetValue(sqlDataReader.GetOrdinal("EmailBody_ID"));
ArchiveFile = (bool)sqlDataReader.GetValue(sqlDataReader.GetOrdinal("ArchiveFile"));
FileHeight = (int)sqlDataReader.GetValue(sqlDataReader.GetOrdinal("FileHeight"));
FileWidth = (int)sqlDataReader.GetValue(sqlDataReader.GetOrdinal("FileWidth"));
IndexFile = sqlDataReader.GetValue(sqlDataReader.GetOrdinal("IndexFile")).ToString();
IndexFileInsert = sqlDataReader.GetValue(sqlDataReader.GetOrdinal("IndexFileInsert")).ToString();
Email = sqlDataReader.GetValue(sqlDataReader.GetOrdinal("Email")).ToString();
ArchiveFileName = sqlDataReader.GetValue(sqlDataReader.GetOrdinal("ArchiveFileName")).ToString();
EmailFrom = sqlDataReader.GetValue(sqlDataReader.GetOrdinal("EmailFrom")).ToString();
EmailSubject = sqlDataReader.GetValue(sqlDataReader.GetOrdinal("EmailSubject")).ToString();
SqlProc = sqlDataReader.GetValue(sqlDataReader.GetOrdinal("SqlProc")).ToString();
}
connLog.Close();
return (
EmailBody_id,
ArchiveFile,
FileHeight,
FileWidth,
IndexFile,
IndexFileInsert,
Email,
ArchiveFileName,
EmailFrom,
EmailSubject,
SqlProc);
}
}
public static string GetEmailBody(string connDataExtract, string SQLProc, int Email_ID, int EmailBodyID)
{
string dataExtract = connDataExtract;
string proc = SQLProc;
int emailID = Email_ID;
int bodyID = EmailBodyID;
string MailBody;
using (SqlConnection connLog = new SqlConnection(dataExtract))
{
connLog.Open();
SqlCommand command = new SqlCommand(proc, connLog)
{
CommandType = CommandType.StoredProcedure
};
command.Parameters.Add("@Email_ID", SqlDbType.Int).Value = ((ulong)emailID);
command.Parameters.Add("@EmailBody_ID", SqlDbType.Int).Value = ((ulong)bodyID);
SqlDataReader dataReader;
string Output = "";
dataReader = command.ExecuteReader();
while (dataReader.Read())
{
Output = dataReader.GetValue(0).ToString();
}
connLog.Close();
MailBody = Output;
return MailBody;
}
}
public static void SendEmail(string emailFrom, string emailSubject, string emailTo, string mailBody)
{
string from = emailFrom;
string subject = emailSubject;
string to = emailTo;
string source = mailBody;
using (MailMessage myHtmlFormattedMail = new MailMessage())
{
MailAddress fromMail = new MailAddress(from);
myHtmlFormattedMail.From = fromMail;
myHtmlFormattedMail.Subject = subject;
myHtmlFormattedMail.Body = source;
foreach (var address in to.Split(new[] { ";" }, StringSplitOptions.RemoveEmptyEntries))
{
myHtmlFormattedMail.To.Add(address);
}
myHtmlFormattedMail.IsBodyHtml = true;
SmtpClient mySmtpClient = new SmtpClient();
mySmtpClient.Send(myHtmlFormattedMail);
}
}
public static void IndexFile(string indexFileInsert, string indexfqdn)
{
string insert = indexFileInsert;
string fqdn = indexfqdn;
try
{
if (!File.Exists(fqdn))
{
using (StreamWriter sw = File.CreateText(fqdn))
{
sw.WriteLine(insert);
}
}
using (StreamWriter sw = File.AppendText(fqdn))
{
sw.WriteLine(insert);
}
}
catch { }
}
public static void LogEmail(string databaseServer, string logProc, int email_ID, int emailBodyID, int primaryEmailUsed, int emailSent, string errorMessage, int errorExists)
{
string dataExtract = databaseServer;
string proc = logProc;
int emailID = email_ID;
int bodyID = emailBodyID;
int usePri = primaryEmailUsed;
int sent = emailSent;
string error = errorMessage;
int exists = errorExists;
using (SqlConnection connLog = new SqlConnection(dataExtract))
{
connLog.Open();
SqlCommand command = new SqlCommand(proc, connLog)
{
CommandType = CommandType.StoredProcedure
};
command.Parameters.Add("@Email_ID", SqlDbType.Int).Value = ((ulong)emailID);
command.Parameters.Add("@EmailBody_ID", SqlDbType.Int).Value = ((ulong)bodyID);
command.Parameters.Add("@PrimaryEmailUsed", SqlDbType.Int).Value = ((ulong)usePri);
command.Parameters.Add("@EmailSent", SqlDbType.Int).Value = ((ulong)sent);
command.Parameters.Add("@ErrorMessage", SqlDbType.NText).Value = error;
command.Parameters.Add("@ErrorExists", SqlDbType.Int).Value = ((ulong)exists);
command.ExecuteNonQuery();
connLog.Close();
}
}
public static void StartBrowser(string mailBody, string file, int width, int height, string fileInsert, string indexFile)
{
try
{
string source = mailBody;
string fqdn = file;
int w = width;
int h = height;
string insert = fileInsert;
string indexFQDN = indexFile;
IndexFile(insert, indexFQDN);
using (WebBrowser wb = new WebBrowser())
{
wb.ScrollBarsEnabled = false;
wb.Width = w;
wb.Height = h;
wb.Visible = false;
wb.DocumentCompleted +=
(sender, e) => WebBrowser_DocumentCompleted(sender, e, fqdn);
wb.DocumentText = source;
Application.Run();
}
}
finally
{
Application.Exit();
}
}
public static void WebBrowser_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e, string file)
{
string fqdn = file;
var webBrowser = (WebBrowser)sender;
using (Bitmap bitmap = new Bitmap(webBrowser.Width, webBrowser.Height))
{
webBrowser
.DrawToBitmap(bitmap, new Rectangle(0, 0, bitmap.Width, bitmap.Height));
bitmap.Save(fqdn, System.Drawing.Imaging.ImageFormat.Jpeg);
}
}
static void Wait(int milliseconds)
{
using (System.Windows.Forms.Timer timer1 = new System.Windows.Forms.Timer())
{
if (milliseconds == 0 || milliseconds < 0) return;
// Console.WriteLine("start wait timer");
timer1.Interval = milliseconds;
timer1.Enabled = true;
timer1.Start();
timer1.Tick += (s, e) =>
{
timer1.Enabled = false;
timer1.Stop();
// Console.WriteLine("stop wait timer");
};
while (timer1.Enabled)
{
System.Windows.Forms.Application.DoEvents();
}
}
}
public static void Archive(string emailBody, string file, int width, int height, string fileInsert, string indexFile, bool archiveFile)
{
string source = emailBody;
string fqdn = file;
int w = width;
int h = height;
string insert = fileInsert;
string indexFQDN = indexFile;
bool archive = archiveFile;
if (archive)
{
Thread tr = new Thread(() => StartBrowser(source, fqdn, w, h, insert, indexFQDN))
{
Name = "Fred",
IsBackground = true
};
tr.SetApartmentState(ApartmentState.STA);
tr.Start();
int wc = 800;
while (!File.Exists(file) || wc <= 0)
{
Wait(50);
wc--;
};
tr.Abort();
}
}
}
}
我已经解决了这个问题，它满足了我项目的需要。可能有更好的解决方案，但这确实有效。使用上面的代码我创建了一个 executable 文件并将结果集限制为前 100 个。创建了一个带有 For 循环的 ssis 包，该循环从暂存 table 中进行记录计数并启动 executable 文件。我进行了几次测试，并且能够超过项目要求的 10,000 个限制。