从 Lua table 创建 .CSV 文件
Creating a .CSV file from a Lua table
我正在尝试从 lua table 创建一个 .csv 文件。我已经在网上和这个论坛上阅读了一些文档......但似乎无法理解。我认为这是因为 lua table 的格式 - 自己看一下。
这个脚本全部来自一个很棒的 open-source 软件，叫做 NeuralTalk2。该软件的要点是为图像添加标题。您可以在该页面上阅读更多信息。
无论如何，让我向您介绍第一段代码：一个函数，它接受 lua table 并将其写入 .json 文件。它是这样的：
function utils.write_json(path, j)
-- API reference http://www.kyne.com.au/~mark/software/lua-cjson-manual.html#encode
cjson.encode_sparse_array(true, 2, 10)
local text = cjson.encode(j)
local file = io.open(path, 'w')
file:write(text)
file:close()
end
代码编译后，.json 文件如下所示：
[{"caption":"a view of a UNK UNK in a cloudy sky","image_id":"0001"},{"caption":"a view of a UNK UNK in a cloudy sky","image_id":"0002"}]
它会持续更长的时间，但一般来说，"caption" 后跟一些文本，"image_id" 后跟图像 ID。
当我在终端上打印 table 时，它看起来像这样：
{
1681 :
{
caption : "a person holding a cell phone in their hand"
image_id : "1681"
}
1682 :
{
caption : "a person is taking a picture of a mirror"
image_id : "1682"
}
}
它前后都有内容...我只是向您展示 table 的一般格式。
您可能想知道 table 是如何定义的...我不确定脚本中是否有非常明确的定义。分享给大家看看，定义它的文件依赖了很多其他文件，所以很乱。
我希望从终端输出中，您可以大致了解 table 的结构，并从中了解 table 的结构。我想将它输出到一个 .csv 文件，看起来像这样
image_id captions
1 xxxx
2 xxxx
3 xxxx
我该怎么做...？不确定，鉴于 lua table...
的格式
这是定义它的脚本。具体来说，它在最后定义，但同样，不确定它是否有太大帮助。
require 'torch'
require 'nn'
require 'nngraph'
-- exotics
require 'loadcaffe'
-- local imports
local utils = require 'misc.utils'
require 'misc.DataLoader'
require 'misc.DataLoaderRaw'
require 'misc.LanguageModel'
local net_utils = require 'misc.net_utils'
local csv_utils = require 'misc.csv_utils'
-------------------------------------------------------------------------------
-- Input arguments and options
-------------------------------------------------------------------------------
cmd = torch.CmdLine()
cmd:text()
cmd:text('Train an Image Captioning model')
cmd:text()
cmd:text('Options')
-- Input paths
cmd:option('-model','','path to model to evaluate')
-- Basic options
cmd:option('-batch_size', 1, 'if > 0 then overrule, otherwise load from checkpoint.')
cmd:option('-num_images', 100, 'how many images to use when periodically evaluating the loss? (-1 = all)')
cmd:option('-language_eval', 0, 'Evaluate language as well (1 = yes, 0 = no)? BLEU/CIDEr/METEOR/ROUGE_L? requires coco-caption code from Github.')
cmd:option('-dump_images', 1, 'Dump images into vis/imgs folder for vis? (1=yes,0=no)')
cmd:option('-dump_json', 1, 'Dump json with predictions into vis folder? (1=yes,0=no)')
cmd:option('-dump_path', 0, 'Write image paths along with predictions into vis json? (1=yes,0=no)')
-- Sampling options
cmd:option('-sample_max', 1, '1 = sample argmax words. 0 = sample from distributions.')
cmd:option('-beam_size', 2, 'used when sample_max = 1, indicates number of beams in beam search. Usually 2 or 3 works well. More is not better. Set this to 1 for faster runtime but a bit worse performance.')
cmd:option('-temperature', 1.0, 'temperature when sampling from distributions (i.e. when sample_max = 0). Lower = "safer" predictions.')
-- For evaluation on a folder of images:
cmd:option('-image_folder', '', 'If this is nonempty then will predict on the images in this folder path')
cmd:option('-image_root', '', 'In case the image paths have to be preprended with a root path to an image folder')
-- For evaluation on MSCOCO images from some split:
cmd:option('-input_h5','','path to the h5file containing the preprocessed dataset. empty = fetch from model checkpoint.')
cmd:option('-input_json','','path to the json file containing additional info and vocab. empty = fetch from model checkpoint.')
cmd:option('-split', 'test', 'if running on MSCOCO images, which split to use: val|test|train')
cmd:option('-coco_json', '', 'if nonempty then use this file in DataLoaderRaw (see docs there). Used only in MSCOCO test evaluation, where we have a specific json file of only test set images.')
-- misc
cmd:option('-backend', 'cudnn', 'nn|cudnn')
cmd:option('-id', 'evalscript', 'an id identifying this run/job. used only if language_eval = 1 for appending to intermediate files')
cmd:option('-seed', 123, 'random number generator seed to use')
cmd:option('-gpuid', 0, 'which gpu to use. -1 = use CPU')
cmd:text()
-------------------------------------------------------------------------------
-- Basic Torch initializations
-------------------------------------------------------------------------------
local opt = cmd:parse(arg)
torch.manualSeed(opt.seed)
torch.setdefaulttensortype('torch.FloatTensor') -- for CPU
if opt.gpuid >= 0 then
require 'cutorch'
require 'cunn'
if opt.backend == 'cudnn' then require 'cudnn' end
cutorch.manualSeed(opt.seed)
cutorch.setDevice(opt.gpuid + 1) -- note +1 because lua is 1-indexed
end
-------------------------------------------------------------------------------
-- Load the model checkpoint to evaluate
-------------------------------------------------------------------------------
assert(string.len(opt.model) > 0, 'must provide a model')
local checkpoint = torch.load(opt.model)
-- override and collect parameters
if string.len(opt.input_h5) == 0 then opt.input_h5 = checkpoint.opt.input_h5 end
if string.len(opt.input_json) == 0 then opt.input_json = checkpoint.opt.input_json end
if opt.batch_size == 0 then opt.batch_size = checkpoint.opt.batch_size end
local fetch = {'rnn_size', 'input_encoding_size', 'drop_prob_lm', 'cnn_proto', 'cnn_model', 'seq_per_img'}
for k,v in pairs(fetch) do
opt[v] = checkpoint.opt[v] -- copy over options from model
end
local vocab = checkpoint.vocab -- ix -> word mapping
-------------------------------------------------------------------------------
-- Create the Data Loader instance
-------------------------------------------------------------------------------
local loader
if string.len(opt.image_folder) == 0 then
loader = DataLoader{h5_file = opt.input_h5, json_file = opt.input_json}
else
loader = DataLoaderRaw{folder_path = opt.image_folder, coco_json = opt.coco_json}
end
-------------------------------------------------------------------------------
-- Load the networks from model checkpoint
-------------------------------------------------------------------------------
local protos = checkpoint.protos
protos.expander = nn.FeatExpander(opt.seq_per_img)
protos.crit = nn.LanguageModelCriterion()
protos.lm:createClones() -- reconstruct clones inside the language model
if opt.gpuid >= 0 then for k,v in pairs(protos) do v:cuda() end end
-------------------------------------------------------------------------------
-- Evaluation fun(ction)
-------------------------------------------------------------------------------
local function eval_split(split, evalopt)
local verbose = utils.getopt(evalopt, 'verbose', true)
local num_images = utils.getopt(evalopt, 'num_images', true)
protos.cnn:evaluate()
protos.lm:evaluate()
loader:resetIterator(split) -- rewind iteator back to first datapoint in the split
local n = 0
local loss_sum = 0
local loss_evals = 0
local predictions = {}
while true do
-- fetch a batch of data
local data = loader:getBatch{batch_size = opt.batch_size, split = split, seq_per_img = opt.seq_per_img}
data.images = net_utils.prepro(data.images, false, opt.gpuid >= 0) -- preprocess in place, and don't augment
n = n + data.images:size(1)
-- forward the model to get loss
local feats = protos.cnn:forward(data.images)
-- evaluate loss if we have the labels
local loss = 0
if data.labels then
local expanded_feats = protos.expander:forward(feats)
local logprobs = protos.lm:forward{expanded_feats, data.labels}
loss = protos.crit:forward(logprobs, data.labels)
loss_sum = loss_sum + loss
loss_evals = loss_evals + 1
end
-- forward the model to also get generated samples for each image
local sample_opts = { sample_max = opt.sample_max, beam_size = opt.beam_size, temperature = opt.temperature }
local seq = protos.lm:sample(feats, sample_opts)
local sents = net_utils.decode_sequence(vocab, seq)
for k=1,#sents do
local entry = {image_id = data.infos[k].id, caption = sents[k]}
if opt.dump_path == 1 then
entry.file_name = data.infos[k].file_path
end
table.insert(predictions, entry)
if opt.dump_images == 1 then
-- dump the raw image to vis/ folder
local cmd = 'cp "' .. path.join(opt.image_root, data.infos[k].file_path) .. '" vis/imgs/img' .. #predictions .. '.jpg' -- bit gross
print(cmd)
os.execute(cmd) -- dont think there is cleaner way in Lua
end
if verbose then
print(string.format('image %s: %s', entry.image_id, entry.caption))
end
end
-- if we wrapped around the split or used up val imgs budget then bail
local ix0 = data.bounds.it_pos_now
local ix1 = math.min(data.bounds.it_max, num_images)
if verbose then
print(string.format('evaluating performance... %d/%d (%f)', ix0-1, ix1, loss))
end
if data.bounds.wrapped then break end -- the split ran out of data, lets break out
if num_images >= 0 and n >= num_images then break end -- we've used enough images
end
local lang_stats
if opt.language_eval == 1 then
lang_stats = net_utils.language_eval(predictions, opt.id)
end
return loss_sum/loss_evals, predictions, lang_stats
end
local loss, split_predictions, lang_stats = eval_split(opt.split, {num_images = opt.num_images})
print('loss: ', loss)
if lang_stats then
print(lang_stats)
end
if opt.dump_json == 1 then
-- dump the json
print(split_predictions)
utils.write_json('vis/vis.json', split_predictions)
csv_utils.write('vis/vis.csv', split_predictions, ";")
end
{
1681 :
{
caption : "a person holding a cell phone in their hand"
image_id : "1681"
}
1682 :
{
caption : "a person is taking a picture of a mirror"
image_id : "1682"
}
}
每个{}表示一个table。冒号前面的数字或文本是一个键，冒号后面的东西是存储在该键下的 table 中的值。
让我们创建一个 table 结构，它会产生类似于上面的输出：
local myTable = {}
myTable[1681] = {caption = "a person holding a cell phone in their hand",
image_id = "1681"}
myTable[1682] = {caption = "a person is taking a picture of a mirror",
image_id = "1682"}
不确定您的问题出在哪里。我认为创建所需的 csv 文件相当简单。您所需要的只是一个循环，为每个 table 条目创建一个新行并添加相应值的 image_id （或键）和标题
一行看起来像：
local nextLine = myTable[1681].image_id .. "," .. myTable[1681].caption .. "\n"
当然这不是很漂亮，您可以使用循环来获取其中的所有元素 table 但我想我也应该为您留一些工作 ;)
如果有人想知道，我很久以前就找到了解决方案。
function nt2_write(path, data, sep)
sep = sep or ','
local file = assert(io.open(path, "w"))
file:write('Image ID' .. "," .. 'Caption')
file:write('\n')
for k, v in pairs(data) do
file:write(v["image_id"] .. "," .. v["caption"])
file:write('\n')
end
file:close()
end
当然，您可能需要更改字符串值，但是是的。编程愉快。
我正在尝试从 lua table 创建一个 .csv 文件。我已经在网上和这个论坛上阅读了一些文档......但似乎无法理解。我认为这是因为 lua table 的格式 - 自己看一下。
这个脚本全部来自一个很棒的 open-source 软件，叫做 NeuralTalk2。该软件的要点是为图像添加标题。您可以在该页面上阅读更多信息。
无论如何，让我向您介绍第一段代码：一个函数，它接受 lua table 并将其写入 .json 文件。它是这样的：
function utils.write_json(path, j)
-- API reference http://www.kyne.com.au/~mark/software/lua-cjson-manual.html#encode
cjson.encode_sparse_array(true, 2, 10)
local text = cjson.encode(j)
local file = io.open(path, 'w')
file:write(text)
file:close()
end
代码编译后，.json 文件如下所示：
[{"caption":"a view of a UNK UNK in a cloudy sky","image_id":"0001"},{"caption":"a view of a UNK UNK in a cloudy sky","image_id":"0002"}]
它会持续更长的时间，但一般来说，"caption" 后跟一些文本，"image_id" 后跟图像 ID。
当我在终端上打印 table 时，它看起来像这样：
{
1681 :
{
caption : "a person holding a cell phone in their hand"
image_id : "1681"
}
1682 :
{
caption : "a person is taking a picture of a mirror"
image_id : "1682"
}
}
它前后都有内容...我只是向您展示 table 的一般格式。
您可能想知道 table 是如何定义的...我不确定脚本中是否有非常明确的定义。分享给大家看看，定义它的文件依赖了很多其他文件，所以很乱。
我希望从终端输出中，您可以大致了解 table 的结构，并从中了解 table 的结构。我想将它输出到一个 .csv 文件，看起来像这样
image_id captions
1 xxxx
2 xxxx
3 xxxx
我该怎么做...？不确定，鉴于 lua table...
的格式这是定义它的脚本。具体来说，它在最后定义，但同样，不确定它是否有太大帮助。
require 'torch'
require 'nn'
require 'nngraph'
-- exotics
require 'loadcaffe'
-- local imports
local utils = require 'misc.utils'
require 'misc.DataLoader'
require 'misc.DataLoaderRaw'
require 'misc.LanguageModel'
local net_utils = require 'misc.net_utils'
local csv_utils = require 'misc.csv_utils'
-------------------------------------------------------------------------------
-- Input arguments and options
-------------------------------------------------------------------------------
cmd = torch.CmdLine()
cmd:text()
cmd:text('Train an Image Captioning model')
cmd:text()
cmd:text('Options')
-- Input paths
cmd:option('-model','','path to model to evaluate')
-- Basic options
cmd:option('-batch_size', 1, 'if > 0 then overrule, otherwise load from checkpoint.')
cmd:option('-num_images', 100, 'how many images to use when periodically evaluating the loss? (-1 = all)')
cmd:option('-language_eval', 0, 'Evaluate language as well (1 = yes, 0 = no)? BLEU/CIDEr/METEOR/ROUGE_L? requires coco-caption code from Github.')
cmd:option('-dump_images', 1, 'Dump images into vis/imgs folder for vis? (1=yes,0=no)')
cmd:option('-dump_json', 1, 'Dump json with predictions into vis folder? (1=yes,0=no)')
cmd:option('-dump_path', 0, 'Write image paths along with predictions into vis json? (1=yes,0=no)')
-- Sampling options
cmd:option('-sample_max', 1, '1 = sample argmax words. 0 = sample from distributions.')
cmd:option('-beam_size', 2, 'used when sample_max = 1, indicates number of beams in beam search. Usually 2 or 3 works well. More is not better. Set this to 1 for faster runtime but a bit worse performance.')
cmd:option('-temperature', 1.0, 'temperature when sampling from distributions (i.e. when sample_max = 0). Lower = "safer" predictions.')
-- For evaluation on a folder of images:
cmd:option('-image_folder', '', 'If this is nonempty then will predict on the images in this folder path')
cmd:option('-image_root', '', 'In case the image paths have to be preprended with a root path to an image folder')
-- For evaluation on MSCOCO images from some split:
cmd:option('-input_h5','','path to the h5file containing the preprocessed dataset. empty = fetch from model checkpoint.')
cmd:option('-input_json','','path to the json file containing additional info and vocab. empty = fetch from model checkpoint.')
cmd:option('-split', 'test', 'if running on MSCOCO images, which split to use: val|test|train')
cmd:option('-coco_json', '', 'if nonempty then use this file in DataLoaderRaw (see docs there). Used only in MSCOCO test evaluation, where we have a specific json file of only test set images.')
-- misc
cmd:option('-backend', 'cudnn', 'nn|cudnn')
cmd:option('-id', 'evalscript', 'an id identifying this run/job. used only if language_eval = 1 for appending to intermediate files')
cmd:option('-seed', 123, 'random number generator seed to use')
cmd:option('-gpuid', 0, 'which gpu to use. -1 = use CPU')
cmd:text()
-------------------------------------------------------------------------------
-- Basic Torch initializations
-------------------------------------------------------------------------------
local opt = cmd:parse(arg)
torch.manualSeed(opt.seed)
torch.setdefaulttensortype('torch.FloatTensor') -- for CPU
if opt.gpuid >= 0 then
require 'cutorch'
require 'cunn'
if opt.backend == 'cudnn' then require 'cudnn' end
cutorch.manualSeed(opt.seed)
cutorch.setDevice(opt.gpuid + 1) -- note +1 because lua is 1-indexed
end
-------------------------------------------------------------------------------
-- Load the model checkpoint to evaluate
-------------------------------------------------------------------------------
assert(string.len(opt.model) > 0, 'must provide a model')
local checkpoint = torch.load(opt.model)
-- override and collect parameters
if string.len(opt.input_h5) == 0 then opt.input_h5 = checkpoint.opt.input_h5 end
if string.len(opt.input_json) == 0 then opt.input_json = checkpoint.opt.input_json end
if opt.batch_size == 0 then opt.batch_size = checkpoint.opt.batch_size end
local fetch = {'rnn_size', 'input_encoding_size', 'drop_prob_lm', 'cnn_proto', 'cnn_model', 'seq_per_img'}
for k,v in pairs(fetch) do
opt[v] = checkpoint.opt[v] -- copy over options from model
end
local vocab = checkpoint.vocab -- ix -> word mapping
-------------------------------------------------------------------------------
-- Create the Data Loader instance
-------------------------------------------------------------------------------
local loader
if string.len(opt.image_folder) == 0 then
loader = DataLoader{h5_file = opt.input_h5, json_file = opt.input_json}
else
loader = DataLoaderRaw{folder_path = opt.image_folder, coco_json = opt.coco_json}
end
-------------------------------------------------------------------------------
-- Load the networks from model checkpoint
-------------------------------------------------------------------------------
local protos = checkpoint.protos
protos.expander = nn.FeatExpander(opt.seq_per_img)
protos.crit = nn.LanguageModelCriterion()
protos.lm:createClones() -- reconstruct clones inside the language model
if opt.gpuid >= 0 then for k,v in pairs(protos) do v:cuda() end end
-------------------------------------------------------------------------------
-- Evaluation fun(ction)
-------------------------------------------------------------------------------
local function eval_split(split, evalopt)
local verbose = utils.getopt(evalopt, 'verbose', true)
local num_images = utils.getopt(evalopt, 'num_images', true)
protos.cnn:evaluate()
protos.lm:evaluate()
loader:resetIterator(split) -- rewind iteator back to first datapoint in the split
local n = 0
local loss_sum = 0
local loss_evals = 0
local predictions = {}
while true do
-- fetch a batch of data
local data = loader:getBatch{batch_size = opt.batch_size, split = split, seq_per_img = opt.seq_per_img}
data.images = net_utils.prepro(data.images, false, opt.gpuid >= 0) -- preprocess in place, and don't augment
n = n + data.images:size(1)
-- forward the model to get loss
local feats = protos.cnn:forward(data.images)
-- evaluate loss if we have the labels
local loss = 0
if data.labels then
local expanded_feats = protos.expander:forward(feats)
local logprobs = protos.lm:forward{expanded_feats, data.labels}
loss = protos.crit:forward(logprobs, data.labels)
loss_sum = loss_sum + loss
loss_evals = loss_evals + 1
end
-- forward the model to also get generated samples for each image
local sample_opts = { sample_max = opt.sample_max, beam_size = opt.beam_size, temperature = opt.temperature }
local seq = protos.lm:sample(feats, sample_opts)
local sents = net_utils.decode_sequence(vocab, seq)
for k=1,#sents do
local entry = {image_id = data.infos[k].id, caption = sents[k]}
if opt.dump_path == 1 then
entry.file_name = data.infos[k].file_path
end
table.insert(predictions, entry)
if opt.dump_images == 1 then
-- dump the raw image to vis/ folder
local cmd = 'cp "' .. path.join(opt.image_root, data.infos[k].file_path) .. '" vis/imgs/img' .. #predictions .. '.jpg' -- bit gross
print(cmd)
os.execute(cmd) -- dont think there is cleaner way in Lua
end
if verbose then
print(string.format('image %s: %s', entry.image_id, entry.caption))
end
end
-- if we wrapped around the split or used up val imgs budget then bail
local ix0 = data.bounds.it_pos_now
local ix1 = math.min(data.bounds.it_max, num_images)
if verbose then
print(string.format('evaluating performance... %d/%d (%f)', ix0-1, ix1, loss))
end
if data.bounds.wrapped then break end -- the split ran out of data, lets break out
if num_images >= 0 and n >= num_images then break end -- we've used enough images
end
local lang_stats
if opt.language_eval == 1 then
lang_stats = net_utils.language_eval(predictions, opt.id)
end
return loss_sum/loss_evals, predictions, lang_stats
end
local loss, split_predictions, lang_stats = eval_split(opt.split, {num_images = opt.num_images})
print('loss: ', loss)
if lang_stats then
print(lang_stats)
end
if opt.dump_json == 1 then
-- dump the json
print(split_predictions)
utils.write_json('vis/vis.json', split_predictions)
csv_utils.write('vis/vis.csv', split_predictions, ";")
end
{
1681 :
{
caption : "a person holding a cell phone in their hand"
image_id : "1681"
}
1682 :
{
caption : "a person is taking a picture of a mirror"
image_id : "1682"
}
}
每个{}表示一个table。冒号前面的数字或文本是一个键，冒号后面的东西是存储在该键下的 table 中的值。
让我们创建一个 table 结构，它会产生类似于上面的输出：
local myTable = {}
myTable[1681] = {caption = "a person holding a cell phone in their hand",
image_id = "1681"}
myTable[1682] = {caption = "a person is taking a picture of a mirror",
image_id = "1682"}
不确定您的问题出在哪里。我认为创建所需的 csv 文件相当简单。您所需要的只是一个循环，为每个 table 条目创建一个新行并添加相应值的 image_id （或键）和标题
一行看起来像：
local nextLine = myTable[1681].image_id .. "," .. myTable[1681].caption .. "\n"
当然这不是很漂亮，您可以使用循环来获取其中的所有元素 table 但我想我也应该为您留一些工作 ;)
如果有人想知道，我很久以前就找到了解决方案。
function nt2_write(path, data, sep)
sep = sep or ','
local file = assert(io.open(path, "w"))
file:write('Image ID' .. "," .. 'Caption')
file:write('\n')
for k, v in pairs(data) do
file:write(v["image_id"] .. "," .. v["caption"])
file:write('\n')
end
file:close()
end
当然，您可能需要更改字符串值，但是是的。编程愉快。