尝试读取 700k+ 的数据并发生错误 "GC Overhead Limit Exceeded"

Question

好吧，我需要帮助来检查我的代码，因为我在编程方面还是个新手（目前是我计算机科学文凭课程的第二年）。当我尝试运行下面的代码时，出现标题 GC Overhead Limit Exceeded 中的错误。

这段代码的简要说明，我正在尝试从 CSV 文件中读取数据，然后将其传输到数据库。仅供参考，实际上我需要阅读 10 个 tables/CSV 个文件，但在此我将展示这个 table Tickets 因为错误仅在我尝试阅读 table/file。其他 table 有数百个 rows/data，而 table Tickets 有 735,504 个 rows/data。此外，在错误发生前的代码运行 6 小时后，我成功读取了 450,028 条数据。

我该如何解决这个错误？可以修改什么来改进我的代码？如果你们能帮助我，我真的很感激:)

public class Demo2 {
    public static void main(String[] args) {    

        String url = "jdbc:mysql://localhost:3306/database";
        String username = "root";
        String password = "password";
        
        try {
            //Connect to the database
            Connection connection = DriverManager.getConnection(url, username, password);
            
            //Test on one table only
            String tableName = "Tickets";
            
            System.out.println("Connecting to TABLE " +tableName +"...");
            readCSVFile(tableName, connection);
            
            System.out.println();
            System.out.println("THE END");

            connection.close();//close connection to the database
        }
        catch (SQLException e) {
            System.out.println("ERROR at main(): SQLException!!");
            e.printStackTrace();
        }
    }
    
    static int countNewRow = 0;
    static int countUpdatedRow = 0;
    
    //Method to read the CSV File
    static void readCSVFile(String tableName, Connection conn) {
        
        //Read CSV File
        try {
            String path = tableName +".csv";
            
            BufferedReader br = new BufferedReader(new FileReader(path));
            br.readLine();//skip the first line
            String inData;
            
            //Read The Remaining Line
            while((inData=br.readLine()) != null)
            {
                String[] rowData = inData.split(",");
                ArrayList <String> rowDataList = new ArrayList<String>();
                
                for (int i=0; i<rowData.length; i++)
                    rowDataList.add(rowData[i]);
                
                
                //To combine String that starts and ends with "
                for(int i=0; i<rowDataList.size(); i++) {
                    if (rowDataList.get(i).charAt(0) == '"') {
                        String string1 = rowDataList.get(i).substring(1, rowDataList.get(i).length());
                        String string2 = rowDataList.get(i+1).substring(0, rowDataList.get(i+1).length()-1);
                        String combined = string1 +"," +string2;
                        
                        rowDataList.set(i, combined); 
                        rowDataList.remove(i+1);
                        
                        break;
                    }
                }
                
                
                //Remove the RM
                for(int i=0; i<rowDataList.size(); i++) {
                    if (rowDataList.get(i).startsWith("RM")) {
                        String string = rowDataList.get(i).substring(2);
                        rowDataList.set(i, string);
                    }
                }
                
                //This is just to keep track of the data that has been read
                System.out.println("[" +rowDataList.get(0) +"]");
                
                //Transfer the data to the database
                insertToDatabase(conn, tableName, rowDataList);
            }
            
            System.out.println("New Row Added : " +countNewRow);
            System.out.println("Updated Row   : " +countUpdatedRow);
            System.out.println("== Process Completed ==");
            br.close();
        }
        catch (FileNotFoundException e) {
            System.out.println("ERROR at readCSVFile(): FileNotFoundException!!");
            e.printStackTrace();
        }
        catch (IOException e) {
            System.out.println("ERROR at readCSVFile(): IOException!!");
            e.printStackTrace();
        }
        catch (SQLException e) {
            System.out.println("ERROR at readCSVFile(): SQLException!!");
            e.printStackTrace();
        }
        catch (ParseException e) {
            System.out.println("ERROR at readCSVFile(): ParseException!!");
            e.printStackTrace();
        }
    }
    
    
    static void insertToDatabase(Connection connection, String tableName, ArrayList <String> rowDataList) throws SQLException, ParseException {
        
        String tableIdName = tableName;
        if (tableIdName.charAt(tableIdName.length()-1) == 's')
            tableIdName = tableIdName.substring(0, tableIdName.length()-1);
        
        //To read row
        String rowID = rowDataList.get(0);
        String selectSQL = "SELECT * FROM " +tableName +" "
                          +"WHERE " +tableIdName +"_ID = " +rowID;
        
        Statement statement = connection.createStatement();
        ResultSet result = statement.executeQuery(selectSQL);
        
        boolean value = result.next();
        
        //INSERT @ UPDATE row
        if (value == true) { //Update Row if the data is already existed
            updateStatementt(tableName, connection, rowDataList);
            countUpdatedRow++;
        }
        else { //Insert New Row
            insertStatementt(tableName, connection, rowDataList);
            countNewRow++;
        }
    }
    
    //Method to insert data to the database
    static void insertStatementt(String tableType, Connection conn, ArrayList <String> rowDataList) throws SQLException, ParseException {
        
        //Generate Question Mark
        String generateQuestionMark = null;
        
        if(rowDataList.size() == 1)
            generateQuestionMark = "?";
        else
            generateQuestionMark = "?, ";
        
        for(int i=1; i<rowDataList.size(); i++) {
            if(i!=rowDataList.size()-1)
                generateQuestionMark += "?, ";
            else
                generateQuestionMark += "?";
        }
        
        //Insert sql
        String sql = "INSERT INTO " +tableType +" VALUES (" +generateQuestionMark +")";
        
        PreparedStatement insertStatement = conn.prepareStatement(sql);
        
        //Insert data

        //There are other 'if' and 'else if' statements here for other tables

        else if (tableType.equals("Tickets")) {
            int ticketID = Integer.parseInt(rowDataList.get(0));
            int movieId = Integer.parseInt(rowDataList.get(1));
            int theaterId = Integer.parseInt(rowDataList.get(2));
            String[] date = rowDataList.get(3).split("/");
            String dateString = date[2] +"-" +date[1] +"-" +date[0];
            Date showDate = Date.valueOf(dateString);
            int showTimeId = Integer.parseInt(rowDataList.get(4));
            int cptId = Integer.parseInt(rowDataList.get(5));
            int pcId = Integer.parseInt(rowDataList.get(6));
            float amountPaid = Float.parseFloat(rowDataList.get(7));
            int year = Integer.parseInt(rowDataList.get(8));
            String month = rowDataList.get(9);
            
            insertStatement.setInt(1, ticketID);
            insertStatement.setInt(2, movieId);
            insertStatement.setInt(3, theaterId);
            insertStatement.setDate(4, showDate);
            insertStatement.setInt(5, showTimeId);
            insertStatement.setInt(6, cptId);
            insertStatement.setInt(7, pcId);
            insertStatement.setFloat(8, amountPaid);
            insertStatement.setInt(9, year);
            insertStatement.setString(10, month);
        }
        
        insertStatement.executeUpdate();        
        insertStatement.close();
    }
    
    
    //Method to update the data from the database
    static void updateStatementt(String tableType, Connection conn, ArrayList <String> rowDataList) throws SQLException {
        Statement  statement = conn.createStatement();
        String sql = "UPDATE " +tableType;
                
        //There are other 'if' and 'else if' statements here for other tables
        
        else if (tableType.equals("Tickets")) {
            String[] date = rowDataList.get(3).split("/");
            String dateString = date[2] +"-" +date[1] +"-" +date[0];
            
            sql += " SET movie_id = " +rowDataList.get(1) +","
                +  " theater_id = " +rowDataList.get(2) +","
                +  " showdate = \"" +dateString +"\","
                +  " showtime_id = " +rowDataList.get(4) +","
                +  " costperticket_id = " +rowDataList.get(5) +","
                +  " personcategory_id = " +rowDataList.get(6) +","
                +  " amount_paid = " +rowDataList.get(7) +","
                +  " year = " +rowDataList.get(8) +","
                +  " month = \"" +rowDataList.get(9) +"\""
                +  " WHERE ticket_id = " +rowDataList.get(0);
        }
        
        statement.executeUpdate(sql);
    }
}

Answer 1

简而言之，读一行，随心所欲。您没有足够的内存来容纳所有 700k 行。

Answer 2

如果您真的想将所有这些数据读入 Java 堆，请增加堆大小，例如使用 -Xmx command-line 开关。由于文本数据在 JVM 中的编码方式，您可能需要比总数据大小所建议的更多的堆。

此外，您的代码中可能有一些地方可以减轻 JVM 内存管理系统的压力。例如，使用“+”连接字符串会产生大量临时数据，这会增加垃圾收集器的负载。使用 StringBuilder 组装字符串可能是一种简单的、较少 resource-hungry 的替代方法。

Answer 3

您应该为更新语句添加 statement.close()。

尝试读取 700k+ 的数据并发生错误 "GC Overhead Limit Exceeded"

Trying to read 700k+ of data and the Error "GC Overhead Limit Exceeded" occurred

java

out-of-memory