如何使用 Java 在 mongodb 中上传 json 文件?
How to upload a json file in mongodb using Java?
我正在尝试使用 Java 将一个大的 JSON 文件 (newclicklogs.json) 上传到 mongodb。这是我的 JSON 文件的样子:
{"preview":false,"result":{"search_term":"rania","request_time":"Sat Apr 01 12:47:04 -0400 2017","request_ip":"127.0.0.1","stats_type":"stats","upi":"355658761","unit":"DR","job_title":"Communications Officer","vpu":"INP","organization":"73","city":"Wash","country":"DC","title":"Tom","url":"www.demo.com","tab_name":"People-Tab","page_name":"PEOPLE","result_number":"5","page_num":"0","session_id":"df234f468cb3fe8be","total_results":"5","filter":"qterm=rina","_time":"2017-04-01T12:47:04.000-0400"}}
{"preview"......}
{"preview"......}
....
这是我的 Java 代码:
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.commons.io.FileUtils;
import org.bson.Document;
import com.mongodb.DBObject;
import com.mongodb.MongoClient;
public class Main {
public static void main(String[] args) throws IOException {
String jsonString = FileUtils.readFileToString(new File("data/newclicklogs.json"), "UTF-8");
Document doc = Document.parse(jsonString);
List<Document> list = new ArrayList<>();
list.add(doc);
new MongoClient().getDatabase("test2").getCollection("collection1").insertMany(list);
}
}
当我查询我的 mongodb 集合时,只添加了一个文档。如何将文件中的所有文档添加到 mongodb 集合中。我是 mongodb 的新手。任何帮助表示赞赏。
您应该尝试使用带缓冲的批量写入 reader。
下面的代码将从文件中读取 json 数据,每次一行(文档),将 json 解析为 Document
并在将其写入数据库之前批量请求。
MongoClient client = new MongoClient("localhost", 27017);
MongoDatabase database = client.getDatabase("test2");
MongoCollection<Document> collection = database.getCollection("collection1");
int count = 0;
int batch = 100;
List<InsertOneModel<Document>> docs = new ArrayList<>();
try (BufferedReader br = new BufferedReader(new FileReader("data/newclicklogs.json"))) {
String line;
while ((line = br.readLine()) != null) {
docs.add(new InsertOneModel<>(Document.parse(line)));
count++;
if (count == batch) {
collection.bulkWrite(docs, new BulkWriteOptions().ordered(false));
docs.clear();
count = 0;
}
}
}
if (count > 0) {
collection.bulkWrite(docs, new BulkWriteOptions().ordered(false));
}
当您 运行 Document.parse
整个 json 时,您实质上是通过覆盖所有以前的文档来将文档减少到最后一个文档。
这里有更多内容
http://mongodb.github.io/mongo-java-driver/3.4/driver/tutorials/bulk-writes/
我正在尝试使用 Java 将一个大的 JSON 文件 (newclicklogs.json) 上传到 mongodb。这是我的 JSON 文件的样子:
{"preview":false,"result":{"search_term":"rania","request_time":"Sat Apr 01 12:47:04 -0400 2017","request_ip":"127.0.0.1","stats_type":"stats","upi":"355658761","unit":"DR","job_title":"Communications Officer","vpu":"INP","organization":"73","city":"Wash","country":"DC","title":"Tom","url":"www.demo.com","tab_name":"People-Tab","page_name":"PEOPLE","result_number":"5","page_num":"0","session_id":"df234f468cb3fe8be","total_results":"5","filter":"qterm=rina","_time":"2017-04-01T12:47:04.000-0400"}}
{"preview"......}
{"preview"......}
....
这是我的 Java 代码:
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.commons.io.FileUtils;
import org.bson.Document;
import com.mongodb.DBObject;
import com.mongodb.MongoClient;
public class Main {
public static void main(String[] args) throws IOException {
String jsonString = FileUtils.readFileToString(new File("data/newclicklogs.json"), "UTF-8");
Document doc = Document.parse(jsonString);
List<Document> list = new ArrayList<>();
list.add(doc);
new MongoClient().getDatabase("test2").getCollection("collection1").insertMany(list);
}
}
当我查询我的 mongodb 集合时,只添加了一个文档。如何将文件中的所有文档添加到 mongodb 集合中。我是 mongodb 的新手。任何帮助表示赞赏。
您应该尝试使用带缓冲的批量写入 reader。
下面的代码将从文件中读取 json 数据,每次一行(文档),将 json 解析为 Document
并在将其写入数据库之前批量请求。
MongoClient client = new MongoClient("localhost", 27017);
MongoDatabase database = client.getDatabase("test2");
MongoCollection<Document> collection = database.getCollection("collection1");
int count = 0;
int batch = 100;
List<InsertOneModel<Document>> docs = new ArrayList<>();
try (BufferedReader br = new BufferedReader(new FileReader("data/newclicklogs.json"))) {
String line;
while ((line = br.readLine()) != null) {
docs.add(new InsertOneModel<>(Document.parse(line)));
count++;
if (count == batch) {
collection.bulkWrite(docs, new BulkWriteOptions().ordered(false));
docs.clear();
count = 0;
}
}
}
if (count > 0) {
collection.bulkWrite(docs, new BulkWriteOptions().ordered(false));
}
当您 运行 Document.parse
整个 json 时,您实质上是通过覆盖所有以前的文档来将文档减少到最后一个文档。
这里有更多内容
http://mongodb.github.io/mongo-java-driver/3.4/driver/tutorials/bulk-writes/