具有 L1 正则化的逻辑回归模型

Question

我正在尝试对逻辑模型应用 L1 正则化

class LogisticRegression(nn.Module):


    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(input_size, num_classes)
    def forward(self, x):
        x = x.reshape(-1, 784)
        output = self.linear(x)
        return output

    def training_step(self, batch):
        images, labels = batch 
        output = self(images)                 
        loss = F.cross_entropy(output, labels)
    
    
        acc = accuracy(output, labels)           
        return {'Training_loss': loss, 'Training_acc': acc}
      
    def training_epoch_end(self, outputs):
        batch_losses = [x['Training_loss'] for x in outputs]
        epoch_loss = torch.stack(batch_losses).mean()   
        batch_accs = [x['Training_acc'] for x in outputs]
        epoch_acc = torch.stack(batch_accs).mean()     
        return {'Training_loss': epoch_loss.item(), 'Training_acc': epoch_acc.item()}

    def epoch_end(self, epoch, result):
        print("Epoch [{}], Training_loss: {:.4f}, Training_acc: {:.4f}".format(epoch, result['Training_loss'], result['Training_acc']))
model = LogisticRegression()

但我认为我做错了准确度没有改变。

L1=0.2
def evaluate(model_b, trainloader):
    outputs = [model_b.training_step(batch) for batch in trainloader]
    return model_b.training_epoch_end(outputs)

def fit(epochs, lr, model_b, trainloader, opt_func=torch.optim.SGD):
    history = []
    optimizer = opt_func(model_b.parameters(), lr)
    for epoch in range(epochs):
        ##### Training Phase 
        for batch in trainloader:
            loss = model_b.training_step(batch)['Training_loss']
            
            loss_Lasso = loss  + 0.5 * L1 #  L1 reg
            
            loss_Lasso.backward()
            optimizer.step()
            optimizer.zero_grad()
        result = evaluate_b(model_b, trainloader)
        model_b.epoch_end(epoch, result)
        history.append(result)
    return history

任何人都可以帮助我解决我所缺少的以及如何真正应用 L1 正则化吗？还有，L1正则化是叫套索吗？

Answer 1

我相信 l1-norm 是一种套索正则化，是的，但是 there are others.

在您的代码段中，L1 被设置为常数，您应该测量模型参数的 l1-norm。然后像您一样将其与您的网络损失相加。在您的示例中只有一个图层，因此您只需要 self.linear 的参数。首先收集所有参数，然后用 torch.norm. You could also use nn.L1Loss.

测量总范数

params = torch.cat([x.view(-1) for x in model.linear.parameters()])
L1 = lamb*torch.norm(params, p=1)

其中 lamb 是您的 lambda 正则化参数，model 从 LogisticRegression class.

具有 L1 正则化的逻辑回归模型

logistic regression model with L1 regularisations

python

pytorch

logistic-regression

regularized