难以预测的抱脸分类

Question

我正在微调 longformer，然后使用 TextClassificationPipeline 和 model(**inputs) 方法进行预测。我不确定为什么会得到不同的结果

import pandas as pd
import datasets
from transformers import LongformerTokenizerFast, LongformerForSequenceClassification, Trainer, TrainingArguments, LongformerConfig
import torch.nn as nn
import torch
from torch.utils.data import DataLoader#Dataset, 
import numpy as np
from sklearn.metrics import accuracy_score, precision_recall_fscore_support
from tqdm import tqdm
#import wandb
import os
from datasets import Dataset
from transformers import TextClassificationPipeline, AutoTokenizer, AutoModelForSequenceClassification

tokenizer = LongformerTokenizerFast.from_pretrained('folder_path/', max_length = maximum_len)

正在从保存的位置加载经过微调的模型。使用原始分词器

saved_location='c:/xyz'
model_saved=AutoModelForSequenceClassification.from_pretrained(saved_location)
pipe = TextClassificationPipeline(model=model_saved, tokenizer=tokenizer, device=0)#tokenizer_saved, padding=True, truncation=True)
prediction = pipe(["The text to predict"], return_all_scores=True)
prediction
[[{'label': 'LABEL_0', 'score': 0.7107483148574829},
  {'label': 'LABEL_1', 'score': 0.2892516553401947}]]

第二种方法

inputs = tokenizer("The text to predict", return_tensors="pt").to(device)
outputs = model_saved(**inputs)#, labels=labels)
print (outputs['logits'])
#tensor([[ 0.4552, -0.4438]], device='cuda:0', grad_fn=<AddmmBackward0>)
torch.sigmoid(outputs['logits'])
#tensor([[0.6119, 0.3908]], device='cuda:0', grad_fn=<SigmoidBackward0>)

AutoModelForSequenceClassificationreturns 概率 0.71 and 0.29。当我看第二种方法时。它 returns logits 0.4552, -0.4438 转换为概率 0.6119, 0.3908

#更新 1

cronoik 的回答中的第一个 link TextClassificationPipeline

function_to_apply (str, optional, defaults to "default") — The function to apply to the model outputs in order to retrieve the scores. Accepts four different values:
"default": if the model has a single label, will apply the sigmoid function on the output. If the model has several labels, will apply the softmax function on the output.
"sigmoid": Applies the sigmoid function on the output.
"softmax": Applies the softmax function on the output.
"none": Does not apply any function on the output.

因为这是一个二元分类问题（单标签）它不应该应用 sigmoid 吗？

Answer 1

我假设model.config.num_labels==2，如果是这样的话，TextClassificationPipeline applies softmax and not sigmoid to calculate the probabilities (code)。

import torch

logits = torch.tensor([ 0.4552, -0.4438])
print(torch.softmax(logits,0))

输出：

tensor([0.7107, 0.2893])

难以预测的抱脸分类

huggingface classification struggling with prediction

python

nlp

classification

huggingface-transformers