将文件写出内存c#

Question

我在使用 c# windows 表单时遇到一些问题。
我的目标是将一个大文件（可能> 5GB）切成文件，每个文件包含一百万行。
根据下面的代码，我不知道为什么会内存不足。

谢谢。

StreamReader readfile = new StreamReader(...);
StreamWriter writefile = new StreamWriter(...);    
string content;
while ((content = readfile.ReadLine()) != null)
{
    writefile.Write(content + "\r\n");
    i++;
    if (i % 1000000 == 0)
    {
        index++;
        writefile.Close();
        writefile.Dispose();
        writefile = new StreamWriter(...);  
    }
    label5.Text = i.ToString();
    label5.Update();
}

Answer 1

错误可能在

label5.Text = i.ToString();
label5.Update();

只是为了做一个测试，我写了类似的东西：

for (int i = 0; i < int.MaxValue; i++)
{
    label1.Text = i.ToString();
    label1.Update();
}

应用程序在 16000-18000 左右冻结（Windows 7 Pro SP1 x64，应用程序运行 x86 和 x64）。

可能发生的情况是，通过运行您在应用程序主线程中的长时间操作，您拖延了 window 的消息队列，并在某个时刻冻结。您可以通过添加

来看到这是问题所在

Application.DoEvents();

而不是

label5.Update();

但即使这是错误的解决方案。正确的解决方案是使用 Invoke 方法（因为您在辅助线程上），

将复制移动到另一个线程并每 x 毫秒更新一次控件

例如：

public void Copy(string source, string dest)
{
    const int updateMilliseconds = 100;

    int index = 0;
    int i = 0;
    StreamWriter writefile = null;

    try
    {
        using (StreamReader readfile = new StreamReader(source))
        {
            writefile = new StreamWriter(dest + index);

            // Initial value "back in time". Forces initial update
            int milliseconds = unchecked(Environment.TickCount - updateMilliseconds);

            string content;
            while ((content = readfile.ReadLine()) != null)
            {
                writefile.Write(content);
                writefile.Write("\r\n"); // Splitted to remove a string concatenation
                i++;

                if (i % 1000000 == 0)
                {
                    index++;
                    writefile.Dispose();
                    writefile = new StreamWriter(dest + index);

                    // Force update
                    milliseconds = unchecked(milliseconds - updateMilliseconds);
                }

                int milliseconds2 = Environment.TickCount;

                int diff = unchecked(milliseconds2 - milliseconds);

                if (diff >= updateMilliseconds)
                {
                    milliseconds = milliseconds2;
                    Invoke((Action)(() => label5.Text = string.Format("File {0}, line {1}", index, i)));
                }
            }
        }
    }
    finally
    {
        if (writefile != null)
        {
            writefile.Dispose();
        }
    }

    // Last update
    Invoke((Action)(() => label5.Text = string.Format("File {0}, line {1} Finished", index, i)));
}

并调用它：

var thread = new Thread(() => Copy(@"C:\Temp\lst.txt", @"C:\Temp\output"));
thread.Start();

注意它是如何每100毫秒写入一次label5，在开始时加上一次（通过设置初始值milliseconds "back in time"），每次输出文件是更改（通过设置 milliseconds "back in time" 的值）并在处理完所有内容后。

可以使用 BackgroundWorker class 编写更正确的示例，该示例明确存在于此场景中。它有一个事件 ProgressChanged，可以订阅该事件以更新 window。

像这样：

private void button1_Click(object sender, EventArgs e)
{
    BackgroundWorker backgroundWorker = new BackgroundWorker();
    backgroundWorker.WorkerReportsProgress = true;
    backgroundWorker.ProgressChanged += backgroundWorker_ProgressChanged;
    backgroundWorker.RunWorkerCompleted += backgroundWorker_RunWorkerCompleted;
    backgroundWorker.DoWork += backgroundWorker_DoWork;
    backgroundWorker.RunWorkerAsync(new string[] { @"C:\Temp\lst.txt", @"C:\Temp\output" });
}

private void backgroundWorker_DoWork(object sender, DoWorkEventArgs e)
{
    BackgroundWorker worker = sender as BackgroundWorker;

    string[] arguments = (string[])e.Argument;
    string source = arguments[0];
    string dest = arguments[1];

    const int updateMilliseconds = 100;

    int index = 0;
    int i = 0;
    StreamWriter writefile = null;

    try
    {
        using (StreamReader readfile = new StreamReader(source))
        {
            writefile = new StreamWriter(dest + index);

            // Initial value "back in time". Forces initial update
            int milliseconds = unchecked(Environment.TickCount - updateMilliseconds);

            string content;
            while ((content = readfile.ReadLine()) != null)
            {
                writefile.Write(content);
                writefile.Write("\r\n"); // Splitted to remove a string concatenation
                i++;

                if (i % 1000000 == 0)
                {
                    index++;
                    writefile.Dispose();
                    writefile = new StreamWriter(dest + index);

                    // Force update
                    milliseconds = unchecked(milliseconds - updateMilliseconds);
                }

                int milliseconds2 = Environment.TickCount;

                int diff = unchecked(milliseconds2 - milliseconds);

                if (diff >= updateMilliseconds)
                {
                    milliseconds = milliseconds2;
                    worker.ReportProgress(0, new int[] { index, i });
                }
            }
        }
    }
    finally
    {
        if (writefile != null)
        {
            writefile.Dispose();
        }
    }

    // For the RunWorkerCompleted
    e.Result = new int[] { index, i };
}

void backgroundWorker_ProgressChanged(object sender, ProgressChangedEventArgs e)
{
    int[] state = (int[])e.UserState;
    label5.Text = string.Format("File {0}, line {1}", state[0], state[1]);
}

void backgroundWorker_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e)
{
    int[] state = (int[])e.Result;
    label5.Text = string.Format("File {0}, line {1} Finished", state[0], state[1]);
}

将文件写出内存c#

write file out of memory c#

c#

out-of-memory

streamwriter

streamreader