正确评估测试数据集

Question

我使用 huggingface 库训练了一个机器翻译模型：

def compute_metrics(eval_preds):
    preds, labels = eval_preds
    if isinstance(preds, tuple):
        preds = preds[0]
    decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)

    # Replace -100 in the labels as we can't decode them.
    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

    # Some simple post-processing
    decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

    result = metric.compute(predictions=decoded_preds, references=decoded_labels)
    result = {"bleu": result["score"]}

    prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
    result["gen_len"] = np.mean(prediction_lens)
    result = {k: round(v, 4) for k, v in result.items()}
    return result


trainer = Seq2SeqTrainer(
    model,
    args,
    train_dataset=tokenized_datasets['train'],
    eval_dataset=tokenized_datasets['test'],
    data_collator=data_collator,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics
)
trainer.train()

model_dir = './models/'
trainer.save_model(model_dir)

以上代码取自此Google Colab notebook。训练结束后，我可以看到训练好的模型被保存到文件夹 models 并计算了指标。现在我想加载经过训练的模型并对新数据集进行预测，这是我尝试过的：

dataset = load_dataset('csv', data_files='data/training_data.csv')
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

# Tokenize the test dataset
tokenized_datasets = train_test.map(preprocess_function_v2, batched=True)
test_dataset = tokenized_datasets['test']
model = AutoModelForSeq2SeqLM.from_pretrained('models')
model(test_dataset)

它抛出以下错误：

*** AttributeError: 'Dataset' object has no attribute 'size'

我也尝试了evaluate()函数，但是它说：

*** torch.nn.modules.module.ModuleAttributeError: 'MarianMTModel' object has no attribute 'evaluate'

而函数eval只打印模型的配置。在新数据集上评估训练模型性能的正确方法是什么？

Answer 1

原来可以使用以下代码生成预测：

inputs = tokenizer(
        questions,
        max_length=max_input_length,
        truncation=True,
        return_tensors='pt',
        padding=True).to('cuda')
translation = model.generate(**inputs)

正确评估测试数据集

Properly evaluate a test dataset

huggingface-transformers

transformer

huggingface-datasets