Cut-Mix in Classification

작성일 | In

Cut Mix 방법 소개

png

식물 이미지로 침입종을 분류하는 이진 분류 대회
이미지 분류에서 Cut-Mix을 적용하여 모델 성능을 높일 수 있습니다.

Invasive Species Monitoring

Identify images of invasive hydrangea
https://www.kaggle.com/c/invasive-species-monitoring

평가 지표 : Area under ROC Curve

패키지 불러오기

필요한 라이브러라들을 불러옵니다.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
import numpy as np
import pandas as pd 
import glob
from torch.utils.data import DataLoader, Dataset
from sklearn.model_selection import StratifiedKFold
from torch.optim.lr_scheduler import ReduceLROnPlateau
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

from PIL import Image
import matplotlib.pyplot as plt
import time
import copy
import cv2
import random
from tqdm import tqdm

import torch.nn as nn
import torch
from torchvision import models, transforms
import torch.nn.functional as F

import warnings
warnings.filterwarnings('ignore')

import os

하이퍼 파라미터

하이퍼파라미터 값들이 있는 딕셔너리를 정해줍니다.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
args = {"TRAIN_LABEL_CSV" : './train_labels.csv',
        "TRAIN_PATH" : "../input/invasivespecies/train",
        "TEST_PATH" : "../input/invasivespecies/test",
        "RESIZE" : 224,#(224,224),
        "LEARNING_RATE" : 0.001,
        "WEIGHT_DECAY" : 0.003,
        "BATCH_SIZE" : 32,
        "FEATURE_EXTRACTING" : True,
        "NUM_EPOCHS" : 50,
        "MEAN" : (0.485, 0.456, 0.406),
        "STD" : (0.229, 0.224, 0.225),
        "BETA" : 1.0,
        "MODEL" : "efficient",
        "MODEL_PATH" : ".",
        "NUM_FOLDS" : 5,
        "DEVICE" : torch.device("cuda:0" if torch.cuda.is_available() else 'cpu')}

데이터 불러오기

데이터를 간단히 살펴보면, invasive(1) 데이터는 1,448개, non-invasive(0) 데이터는 847개 입니다. 또한 예측에 사용될 Test 데이터는 전체 1,531개 입니다. 학습 데이터 전체를 사용하기 위해, 5-Fold Stratify로 모델을 훈련합니다.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
def get_data(args):
    train = pd.read_csv(args["TRAIN_LABEL_CSV"])
    train_list = glob.glob(args["TRAIN_PATH"] + "/*")
    test_list = glob.glob(args["TEST_PATH"] + "/*")

    path_df = pd.DataFrame({"path" : train_list})
    path_df["name"] = path_df["path"].apply(lambda x : x.split("/")[-1].split(".")[0])
    path_df["name"] = path_df["name"].astype(np.int)
    train_df = pd.merge(train, path_df, on='name', how='left')
    
    print(f"Train 데이터 수 : {len(train_list)}\nTest 데이터 수 : {len(test_list)}\n")
    print(f"Target 데이터 수 : \n{train_df['invasive'].value_counts()}\n")
    
    return train_df, train_list, test_list

train_df, train_list, test_list = get_data(args)
train_df.head()

1
2
3
4
5
6
7
Train 데이터 수 : 2295
Test 데이터 수 : 1531

Target 데이터 수 : 
1    1448
0     847
Name: invasive, dtype: int64
name invasive path
0 1 0 ../input/invasivespecies/train/1.jpg
1 2 0 ../input/invasivespecies/train/2.jpg
2 3 1 ../input/invasivespecies/train/3.jpg
3 4 0 ../input/invasivespecies/train/4.jpg
4 5 1 ../input/invasivespecies/train/5.jpg

Invasive(1)/Non-invasive(0) 이미지 확인

침입종 수국(1)과 침입종이 아닌 식물(0)의 이미지입니다.

png

데이터 로더

데이터 로더 클래스를 정의합니다.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
class InvasiveDataset(Dataset):
    def __init__(self, dataframe, transform=None):
        super().__init__()
        self.df = dataframe
        self.file_list = dataframe["path"].values
        self.transform = transform
    def __getitem__(self, index):
        
        path = self.file_list[index]
        image_id = path.split("/")[-1].split(".")[0]
        image = Image.open(path)
        label = self.df[self.df['name']==int(image_id)]['invasive'].values # label in Series. Change to array.
        label = label.squeeze()
        if self.transform:
            image = self.transform(image)
        
        return image, label
        
    def __len__(self):
        return len(self.file_list)

Transforms 함수 정의

Train 데이터에 적용된 augmentation : RandomResizedCrop & RandomHorizontalFlip
Valid 데이터에 적용된 augmentation : Resize & CenterCrop

위 augmentation 적용 결과는 아래에서 확인할 수 있습니다.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
def get_train_transform(args):
    return transforms.Compose([
        transforms.RandomResizedCrop(args["RESIZE"], scale=(0.5, 1.0)),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize(mean=args["MEAN"], std=args["STD"])
    ])

def get_valid_transform(args):
    return transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(args["RESIZE"]),
        transforms.ToTensor(),
        transforms.Normalize(mean=args["MEAN"], std=args["STD"])
    ])

데이터 로더 적용 함수

데이터 로더를 적용하여 데이터셋을 train, valid로 나누고, 각각 배치 형태로 만들어 줍니다.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
def get_data_loader(train_df, valid_df=None):
    
    if valid_df is not None: # k-fold
        train_set = InvasiveDataset(train_df, transform=get_train_transform(args))
        train_loader = DataLoader(train_set, batch_size=args["BATCH_SIZE"], shuffle=True, drop_last=True, num_workers=1)
        valid_set = InvasiveDataset(valid_df, transform=get_valid_transform(args))
        valid_loader = DataLoader(valid_set, batch_size=1, shuffle=False, num_workers=1)     
    else:
        train_, valid_ = train_test_split(train_df, test_size=0.1, random_state=42, stratify=train_df["invasive"])
        train_set = InvasiveDataset(train_, transform=get_train_transform(args))
        train_loader = DataLoader(train_set, batch_size=args["BATCH_SIZE"], shuffle=True, drop_last=True, num_workers=1)
        valid_set = InvasiveDataset(valid_, transform=get_valid_transform(args))
        valid_loader = DataLoader(valid_set, batch_size=1, shuffle=False, num_workers=1)
    
    return train_loader, valid_loader

train_loader, valid_loader = get_data_loader(train_df)

1. Augmentation 확인하기(Train)

Train 데이터에 적용된 augmentation을 시각화했습니다.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
def plot_train_transform(train_list, args, num_images=3, fig_size=(20,15)):

    preprocess1 = transforms.Compose([
        transforms.RandomResizedCrop(args["RESIZE"], scale=(0.5, 1.0)),
    ])

    preprocess2 = transforms.Compose([
            transforms.RandomHorizontalFlip(p=1),
    ])    

    fig , axes = plt.subplots(num_images,4)
    fig.set_size_inches(fig_size)

    random_path = random.sample(train_list,num_images)


    for idx in range(num_images):
        img = Image.open(random_path[idx])
        img_origin = img.copy()
        img_aug1 = preprocess1(img)
        img_aug2 = preprocess2(img)
        img_aug3 = preprocess2(img_aug1)
        imgs = [img_origin, img_aug1, img_aug2, img_aug3]
        preprocess_name = ["Original Image", f"RandomResizedCrop(Resize:{args['RESIZE']})", "RandomHorizontalFlip"]
        for i in range(len(imgs)):

            axes[idx, i].imshow(imgs[i])
            if i ==len(imgs)-1:
                axes[idx, i].set_title(f"{preprocess_name[1]}+{preprocess_name[2]}\nSize : {imgs[i].size}")
            else:
                axes[idx, i].set_title(f"{preprocess_name[i]}\nSize : {imgs[i].size}")
            axes[idx, i].axis('off')
            
plot_train_transform(train_list, args, num_images=3, fig_size=(15,12))

png

Train셋에 적용된 augmentation을 시각화하였습니다.
먼저 RandomResizedCrop을 통해 이미지를 scale범위 내의 비율로 랜덤하게 크롭한 후, size(224,224)로 크기를 조정했습니다. 그 다음 RandomHorizontalFlip으로 랜덤하게 좌우반전을 주었습니다.

2. Augmentation 확인하기(Valid)

Valid 데이터에 적용된 augmentation을 시각화했습니다.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
def plot_valid_transform(train_list, args, num_images=3, fig_size=(20,15)):
    
    preprocess1= transforms.Compose([
        transforms.Resize(256),
    ])

    preprocess2 = transforms.Compose([
            transforms.CenterCrop(args["RESIZE"]),
    ])
    
    fig , axes = plt.subplots(num_images,4)
    fig.set_size_inches(fig_size)

    random_path = random.sample(train_list,num_images)

    for idx in range(num_images):
        img = Image.open(random_path[idx])
        img_origin = img.copy()
        img_aug1 = preprocess1(img)
        img_aug2 = preprocess2(img)
        img_aug3 = preprocess2(img_aug1)
        imgs = [img_origin, img_aug1, img_aug2, img_aug3]
        preprocess_name = ["Original Image", "Resize(256)", f"CenterCrop(Resize:{args['RESIZE']})"]
        for i in range(len(imgs)):

            axes[idx, i].imshow(imgs[i])
            if i == len(imgs)-1:
                axes[idx, i].set_title(f"{preprocess_name[1]}+{preprocess_name[2]}\nSize : {imgs[i].size}")
            else:
                axes[idx, i].set_title(f"{preprocess_name[i]}\nSize : {imgs[i].size}")
            axes[idx, i].axis('off')

plot_valid_transform(train_list, args, num_images=3, fig_size=(15,12))

png

Valid 데이터에는 Resize 와 CenterCrop이 적용되었습니다.
Resize에서는 가로세로 비율을 유지하며 이미지 크기를 조정하였습니다. Invasive 대회의 데이터셋의 이미지들은 주로 중앙쪽에 주요 정보(침입종 식물)가 분포되어 있었습니다.
이러한 점을 이용하여 CenterCrop으로 중간 부분 위주로 이미지를 잘라내어 모델이 주요한 정보에 더욱 집중하여 학습할 수 있도록 하였습니다.

학습 준비

모델, Optimizer, Criterion

모델은 resnet18을 사용했습니다. 이진 분류이므로, num_classes를 2로 설정합니다.
Criterion은 CrossEntropyLoss, Optimizer은 Adam을 사용합니다.

1
2
3
4
def set_parameter_requires_grad(model, feature_extracting):
    if feature_extracting:
        for param in model.parameters():
            param.requires_grad = False

1
2
3
4
5
6
7
8
9
10
def get_model(args):
    model = models.resnet18(pretrained=True)
    set_parameter_requires_grad(model, args["FEATURE_EXTRACTING"])
    model.fc = nn.Linear(512, 2)
    model = model.to(args["DEVICE"])

    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(params=model.parameters(), lr=args["LEARNING_RATE"], weight_decay=args["WEIGHT_DECAY"])
    
    return model, criterion, optimizer

한개의 배치에 대해 모델 성능 확인

전체 데이터를 학습하기 전, 우선 하나의 배치에 대해서 모델의 학습이 잘 이루어지는 확인합니다. 전체 데이터를 학습하게 되면, 모델이 잘 훈련되고 있는지 확인하기까지 시간이 많이 소요되므로 먼저 일부 데이터에 대해서만 학습을 진행해봅니다.

아래 출력되는 loss값을 보면, 모델 학습이 잘 이루어진 것을 확인할 수 있습니다.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
def check_batch(model, train_loader,criterion, optimizer, args):

    image, label = next(iter(train_loader))

    for i in range(30):
        data = image.to(args["DEVICE"])
        targets = label.to(args["DEVICE"])

        outputs = model(data)
        loss = criterion(outputs, targets)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        print(loss)
    return 
model, criterion, optimizer = get_model(args)
check_batch(model, train_loader, criterion, optimizer, args)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
tensor(0.7094, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6460, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.5871, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.5325, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.4823, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.4363, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.3945, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.3565, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.3224, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.2917, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.2642, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.2396, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.2177, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.1982, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.1808, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.1653, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.1514, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.1391, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.1280, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.1181, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.1092, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.1013, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.0941, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.0876, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.0818, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.0765, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.0717, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.0674, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.0635, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.0599, device='cuda:0', grad_fn=<NllLossBackward>)

Cut Mix

Cut Mix 설명

Cut mix은 주어진 이미지에 대해서 랜덤하게 패치를 만들어내어 다른 이미지에 합성하는 augmentation 기법 중 하나입니다.
Cut mix의 예시는 아래 4.결과에서 확인할 수 있습니다.
rand_bbox 함수는 패치를 생성하는 코드입니다. 주어진 이미지 크기 내에서, 랜덤하게 패치를 만들어내고 이에 대한 좌표를 반환하는 함수입니다.
패치의 폭과 넓이는, 주어진 이미지의 폭과 넓이에 np.sqrt(1-lam)을 곱하여 얻게 됩니다. 여기서 lam은 베타분포에서 랜덤하게 얻은 값입니다. 이렇게 얻어낸 패치 부분을, 랜덤하게 섞은 X(input)값들의 패치 부분으로 교체합니다.
이때 합성된 이미지의 레이블은, 합성된 전체 이미지에서 각각의 이미지가 차지하는 면적 비율만큼(lam)을 더한 값이 됩니다.

y = λ x invasive + (1-λ) x non-invasive

자세한 코드 설명은 아래에 있습니다.
이러한 cut mix 기법은 모델이 본 적이 없는 데이터에 대해서도 robust하게 예측할 수 있도록 도움을 줄 수 있습니다.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
def rand_bbox(size, lam): # size : [B, C, W, H]
    W = size[2] # 이미지의 width
    H = size[3] # 이미지의 height
    cut_rat = np.sqrt(1. - lam)  # 패치 크기의 비율 정하기
    cut_w = np.int(W * cut_rat)  # 패치의 너비
    cut_h = np.int(H * cut_rat)  # 패치의 높이

    # uniform
    # 기존 이미지의 크기에서 랜덤하게 값을 가져옵니다.(중간 좌표 추출)
    cx = np.random.randint(W)
    cy = np.random.randint(H)

    # 패치 부분에 대한 좌표값을 추출합니다.
    bbx1 = np.clip(cx - cut_w // 2, 0, W)
    bby1 = np.clip(cy - cut_h // 2, 0, H)
    bbx2 = np.clip(cx + cut_w // 2, 0, W)
    bby2 = np.clip(cy + cut_h // 2, 0, H)

    return bbx1, bby1, bbx2, bby2

Cut Mix 적용 과정

1. 배치 내의 데이터 셔플

데이터 로더에서 배치별로 데이터가 불러와지면, torch.randperm함수를 사용하여 인덱스를 랜덤하게 셔플합니다.
처음 람다값은 베타 분포에서 랜덤하게 가져옵니다.

1
2
3
4
5
6
7
8
9
10
X,y = next(iter(train_loader))
X = X.to(args["DEVICE"])
y = y.to(args["DEVICE"])

lam = np.random.beta(1.0, 1.0)  # 베타 분포에서 lam 값을 가져옵나다.
rand_index = torch.randperm(X.size()[0]).to(args["DEVICE"]) # batch_size 내의 인덱스가 랜덤하게 셔플됩니다.
shuffled_y = y[rand_index] # 타겟 레이블을 랜덤하게 셔플합니다.

print(lam)
print(rand_index)

1
2
3
4
0.2246911819269543
tensor([ 3, 26, 11, 24,  5, 28, 17, 31, 21, 27,  9, 13, 20, 12, 15,  1, 14, 16,
        30,  2, 25,  8, 19, 29, 18, 22,  6,  7, 10, 23,  4,  0],
       device='cuda:0')

2. 패치 부분 교체하기

기존 이미지의 패치 부분을, rand_index를 통해 셔플된 이미지들의 패치로 채워 넣습니다.

1
2
bbx1, bby1, bbx2, bby2 = rand_bbox(X.size(), lam)
X[:,:,bbx1:bbx2, bby1:bby2] = X[rand_index,:,bbx1:bbx2, bby1:bby2]

3. 람다 조정하기

픽셀 비율과 정확히 일치하도록 람다를 조정합니다.
lam은 이후 loss값을 계산하는데 사용됩니다.

1
2
lam = 1 - ((bbx2 - bbx1) * (bby2 - bby1) / (X.size()[-1] * X.size()[-2]))
print(lam)

1
0.49609375

4. 결과

섞인 이미지를 볼 수 있습니다.

1
plt.imshow(X[0].permute(1, 2, 0).cpu())

1
<matplotlib.image.AxesImage at 0x7fc575f0ad10>

png

Cut Mix 결과 이미지

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
def cutmix_plot(train_loader):
    fig , axes = plt.subplots(1,3)
    fig.set_size_inches(15,12)
    
    for i in range(3):
        for inputs, targets in train_loader:
            inputs = inputs
            targets = targets
            break

        lam = np.random.beta(1.0, 1.0) 
        rand_index = torch.randperm(inputs.size()[0])
        bbx1, bby1, bbx2, bby2 = rand_bbox(inputs.size(), lam)
        inputs[:, :, bbx1:bbx2, bby1:bby2] = inputs[rand_index, :, bbx1:bbx2, bby1:bby2]
        lam = 1 - ((bbx2 - bbx1) * (bby2 - bby1) / (inputs.size()[-1] * inputs.size()[-2]))
        axes[i].imshow(inputs[1].permute(1, 2, 0).cpu())
        axes[i].set_title(f'λ : {np.round(lam,3)}')
        axes[i].axis('off')
    return

cutmix_plot(train_loader)

png

모델 학습

Validation 함수

대회의 평가 지표가 area under the ROC curve이므로, validation 함수에서 AUC가 계산되도록 하였습니다.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
def validation(model, valid_loader, criterion):

    accuracy = 0
    valid_loss = 0
    y_score = []

    for i, (X, y) in enumerate(valid_loader):
        if torch.cuda.is_available():
            X = X.to('cuda')
            y = y.to('cuda')

        outputs = model(X)
        loss = criterion(outputs, y)
        valid_loss += loss.item()
        outputs_ = torch.argmax(outputs, dim=1)
        
        accuracy += (outputs_ == y).float().sum()
        
        # For roc_auc_score
        outputs = F.softmax(outputs, dim=-1)
        y_score.append(outputs.detach().cpu().numpy())
    
    y_score = np.array([i for sub in y_score for i in sub])

    return valid_loss, accuracy, y_score

Train 함수

cut mix를 적용한 모델 훈련 코드입니다.
스케줄러는 ReduceLROnPlateau을 사용하였습니다.
val_loss가 가장 낮은 모델만 저장되도록 하였습니다.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
def train_model(model, train_loader, valid_loader, criterion, optimizer, args, fold_num=1):
    steps = 0
    total_step = len(train_loader)
    train_losses, validation_losses = [], []
    best_val = np.inf
    
    scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=7, verbose=True)
    
    if torch.cuda.is_available():
        model = model.to(args["DEVICE"])

    for epoch in range(args['NUM_EPOCHS']):
        running_loss = 0
        for i, (X, y) in enumerate(train_loader):
            
            if torch.cuda.is_available():
                X = X.to(args["DEVICE"])
                y = y.to(args["DEVICE"])

                
            if args["BETA"] > 0 and np.random.random()>0.5: # cutmix 작동될 확률      
                lam = np.random.beta(args["BETA"], args["BETA"])
                rand_index = torch.randperm(X.size()[0]).to(args["DEVICE"])
                target_a = y
                target_b = y[rand_index]            
                bbx1, bby1, bbx2, bby2 = rand_bbox(X.size(), lam)
                X[:, :, bbx1:bbx2, bby1:bby2] = X[rand_index, :, bbx1:bbx2, bby1:bby2]
                lam = 1 - ((bbx2 - bbx1) * (bby2 - bby1) / (X.size()[-1] * X.size()[-2]))
                outputs = model(X)
                loss = criterion(outputs, target_a) * lam + criterion(outputs, target_b) * (1. - lam)
                
            else:
                outputs = model(X)
                loss = criterion(outputs, y)                  
                
            steps += 1
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            running_loss += loss.item()
            
            if steps % total_step == 0:
                model.eval()
                with torch.no_grad():
                    valid_loss, accuracy, y_score = validation(model, valid_loader, criterion)

                print("Epoch: {}/{}.. ".format(epoch + 1, args['NUM_EPOCHS']) +
                      "Training Loss: {:.5f}.. ".format(running_loss / total_step) +
                      "Valid Loss: {:.5f}.. ".format(valid_loss / len(valid_loader)) +
                      "Valid Accuracy: {:.5f}.. ".format(accuracy / len(valid_loader.dataset)) + # 전체 데이터 수에 대해 나눠준다
                      "Area Under Curve: {:.5f}..".format(roc_auc_score(valid_loader.dataset.df['invasive'].values, y_score[:, 1]))) 
                
                # Save Model
                if (valid_loss / len(valid_loader)) < best_val:
                    best_val = (valid_loss / len(valid_loader))
                    torch.save(model.state_dict(), f"{args['MODEL_PATH']}/{fold_num}_best_checkpoint_{str(epoch + 1).zfill(3)}epoch.tar")
                    try:
                        os.remove(f_pth)
                    except:
                        pass
                    f_pth = f"{args['MODEL_PATH']}/{fold_num}_best_checkpoint_{str(epoch + 1).zfill(3)}epoch.tar"
                
                train_losses.append(running_loss / len(train_loader))
                validation_losses.append(valid_loss / len(valid_loader))
                steps = 0
                running_loss = 0
                model.train()
                
        scheduler.step(valid_loss / len(valid_loader))

    return

Fold 없이 단일 모델 학습

150개 데이터에 대해서 코드에 이상이 없는지 빠르게 실험해봤습니다.

1
2
3
4
train, train_list, test_list = get_data(args)
train_loader, valid_loader = get_data_loader(train[:150])
model, criterion, optimizer = get_model(args)
train_model(model, train_loader, valid_loader, criterion, optimizer, args)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
Train 데이터 수 : 2295
Test 데이터 수 : 1531

Target 데이터 수 : 
1    1448
0     847
Name: invasive, dtype: int64

Epoch: 1/50.. Training Loss: 0.73771.. Valid Loss: 0.65644.. Valid Accuracy: 0.66667.. Area Under Curve: 0.62000
Epoch: 2/50.. Training Loss: 0.67826.. Valid Loss: 0.57433.. Valid Accuracy: 0.66667.. Area Under Curve: 0.76000
Epoch: 3/50.. Training Loss: 0.59255.. Valid Loss: 0.54534.. Valid Accuracy: 0.80000.. Area Under Curve: 0.88000
Epoch: 4/50.. Training Loss: 0.54118.. Valid Loss: 0.50904.. Valid Accuracy: 0.66667.. Area Under Curve: 0.86000
Epoch: 5/50.. Training Loss: 0.55861.. Valid Loss: 0.51992.. Valid Accuracy: 0.66667.. Area Under Curve: 0.90000
Epoch: 6/50.. Training Loss: 0.52161.. Valid Loss: 0.44656.. Valid Accuracy: 0.73333.. Area Under Curve: 0.94000
Epoch: 7/50.. Training Loss: 0.49751.. Valid Loss: 0.40690.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 8/50.. Training Loss: 0.45763.. Valid Loss: 0.37955.. Valid Accuracy: 0.80000.. Area Under Curve: 1.00000
Epoch: 9/50.. Training Loss: 0.45117.. Valid Loss: 0.39089.. Valid Accuracy: 0.73333.. Area Under Curve: 1.00000
Epoch: 10/50.. Training Loss: 0.46032.. Valid Loss: 0.33357.. Valid Accuracy: 0.86667.. Area Under Curve: 1.00000
Epoch: 11/50.. Training Loss: 0.53030.. Valid Loss: 0.31587.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 12/50.. Training Loss: 0.41064.. Valid Loss: 0.31270.. Valid Accuracy: 0.93333.. Area Under Curve: 1.00000
Epoch: 13/50.. Training Loss: 0.42074.. Valid Loss: 0.30051.. Valid Accuracy: 0.93333.. Area Under Curve: 1.00000
Epoch: 14/50.. Training Loss: 0.47569.. Valid Loss: 0.30176.. Valid Accuracy: 0.93333.. Area Under Curve: 1.00000
Epoch: 15/50.. Training Loss: 0.37441.. Valid Loss: 0.27918.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 16/50.. Training Loss: 0.42745.. Valid Loss: 0.27515.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 17/50.. Training Loss: 0.36221.. Valid Loss: 0.26101.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 18/50.. Training Loss: 0.40636.. Valid Loss: 0.27356.. Valid Accuracy: 0.93333.. Area Under Curve: 1.00000
Epoch: 19/50.. Training Loss: 0.37077.. Valid Loss: 0.24716.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 20/50.. Training Loss: 0.39795.. Valid Loss: 0.23801.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 21/50.. Training Loss: 0.33757.. Valid Loss: 0.23665.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 22/50.. Training Loss: 0.30681.. Valid Loss: 0.23004.. Valid Accuracy: 0.93333.. Area Under Curve: 1.00000
Epoch: 23/50.. Training Loss: 0.57297.. Valid Loss: 0.22571.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 24/50.. Training Loss: 0.43501.. Valid Loss: 0.23294.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 25/50.. Training Loss: 0.29009.. Valid Loss: 0.22755.. Valid Accuracy: 0.93333.. Area Under Curve: 1.00000
Epoch: 26/50.. Training Loss: 0.35224.. Valid Loss: 0.22339.. Valid Accuracy: 0.93333.. Area Under Curve: 1.00000
Epoch: 27/50.. Training Loss: 0.30319.. Valid Loss: 0.20292.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 28/50.. Training Loss: 0.38560.. Valid Loss: 0.21128.. Valid Accuracy: 0.93333.. Area Under Curve: 1.00000
Epoch: 29/50.. Training Loss: 0.29471.. Valid Loss: 0.20262.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 30/50.. Training Loss: 0.32983.. Valid Loss: 0.19320.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 31/50.. Training Loss: 0.31507.. Valid Loss: 0.18878.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 32/50.. Training Loss: 0.29588.. Valid Loss: 0.18774.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 33/50.. Training Loss: 0.38119.. Valid Loss: 0.17978.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 34/50.. Training Loss: 0.38593.. Valid Loss: 0.17570.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 35/50.. Training Loss: 0.40947.. Valid Loss: 0.18061.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 36/50.. Training Loss: 0.28186.. Valid Loss: 0.18099.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 37/50.. Training Loss: 0.45454.. Valid Loss: 0.17288.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 38/50.. Training Loss: 0.29359.. Valid Loss: 0.17260.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 39/50.. Training Loss: 0.28112.. Valid Loss: 0.17446.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 40/50.. Training Loss: 0.29126.. Valid Loss: 0.16450.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 41/50.. Training Loss: 0.47272.. Valid Loss: 0.16572.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 42/50.. Training Loss: 0.29763.. Valid Loss: 0.16253.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 43/50.. Training Loss: 0.23737.. Valid Loss: 0.15957.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 44/50.. Training Loss: 0.31854.. Valid Loss: 0.15708.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 45/50.. Training Loss: 0.28495.. Valid Loss: 0.15177.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 46/50.. Training Loss: 0.33081.. Valid Loss: 0.15521.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 47/50.. Training Loss: 0.36897.. Valid Loss: 0.14969.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 48/50.. Training Loss: 0.38459.. Valid Loss: 0.15459.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 49/50.. Training Loss: 0.24727.. Valid Loss: 0.15773.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000
Epoch: 50/50.. Training Loss: 0.24382.. Valid Loss: 0.15321.. Valid Accuracy: 1.00000.. Area Under Curve: 1.00000

5-Fold 학습

5번의 교차 검증 및 학습을 진행합니다. 각 교차 검증에서 가장 낮은 val_loss의 모델을 저장합니다. 이후 test 데이터 예측 시, 5개의 모델의 예측 결과를 voting을 통해 앙상블할 수 있습니다.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
def fold_train(args, train_df):
    folds = StratifiedKFold(n_splits=args["NUM_FOLDS"], shuffle=True, random_state=42)

    X = train_df

    for i, (train_index, valid_index) in enumerate(folds.split(X,X["invasive"])):
        fold_num = i+1
        X_train = X.iloc[train_index]
        X_val = X.iloc[valid_index]


        model, criterion, optimizer = get_model(args)
        train_loader, valid_loader = get_data_loader(X_train, X_val)

        print("=" * 100)
        print(f"{fold_num}/{args['NUM_FOLDS']} Cross Validation Training Starts ...\n")
        train_model(model, train_loader, valid_loader, criterion, optimizer, args, fold_num=fold_num)
        print(f"\n{fold_num}/{args['NUM_FOLDS']} Cross Validation Training Ends ...\n")

    return 

fold_train(args, train_df)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
====================================================================================================
1/5 Cross Validation Training Starts ...

Epoch: 1/50.. Training Loss: 0.54776.. Valid Loss: 0.40315.. Valid Accuracy: 0.81481.. Area Under Curve: 0.89644
Epoch: 2/50.. Training Loss: 0.41436.. Valid Loss: 0.35228.. Valid Accuracy: 0.84314.. Area Under Curve: 0.92516
Epoch: 3/50.. Training Loss: 0.40087.. Valid Loss: 0.32396.. Valid Accuracy: 0.84967.. Area Under Curve: 0.93788
Epoch: 4/50.. Training Loss: 0.37837.. Valid Loss: 0.31565.. Valid Accuracy: 0.87800.. Area Under Curve: 0.94291
Epoch: 5/50.. Training Loss: 0.36564.. Valid Loss: 0.29177.. Valid Accuracy: 0.87146.. Area Under Curve: 0.94726
Epoch: 6/50.. Training Loss: 0.32669.. Valid Loss: 0.29859.. Valid Accuracy: 0.86057.. Area Under Curve: 0.94791
Epoch: 7/50.. Training Loss: 0.35096.. Valid Loss: 0.27714.. Valid Accuracy: 0.89760.. Area Under Curve: 0.94918
Epoch: 8/50.. Training Loss: 0.36902.. Valid Loss: 0.28113.. Valid Accuracy: 0.89978.. Area Under Curve: 0.94979
Epoch: 9/50.. Training Loss: 0.35148.. Valid Loss: 0.27296.. Valid Accuracy: 0.90632.. Area Under Curve: 0.95223
Epoch: 10/50.. Training Loss: 0.35789.. Valid Loss: 0.26953.. Valid Accuracy: 0.89542.. Area Under Curve: 0.95172
Epoch: 11/50.. Training Loss: 0.36694.. Valid Loss: 0.28762.. Valid Accuracy: 0.86275.. Area Under Curve: 0.95213
Epoch: 12/50.. Training Loss: 0.39102.. Valid Loss: 0.26983.. Valid Accuracy: 0.89760.. Area Under Curve: 0.95384
Epoch: 13/50.. Training Loss: 0.38918.. Valid Loss: 0.27797.. Valid Accuracy: 0.88453.. Area Under Curve: 0.95359
Epoch: 14/50.. Training Loss: 0.38915.. Valid Loss: 0.26591.. Valid Accuracy: 0.89325.. Area Under Curve: 0.95589
Epoch: 15/50.. Training Loss: 0.37149.. Valid Loss: 0.26141.. Valid Accuracy: 0.89542.. Area Under Curve: 0.95728
Epoch: 16/50.. Training Loss: 0.37789.. Valid Loss: 0.26677.. Valid Accuracy: 0.89542.. Area Under Curve: 0.95542
Epoch: 17/50.. Training Loss: 0.35278.. Valid Loss: 0.26262.. Valid Accuracy: 0.89760.. Area Under Curve: 0.95646
Epoch: 18/50.. Training Loss: 0.38484.. Valid Loss: 0.26765.. Valid Accuracy: 0.88889.. Area Under Curve: 0.95656
Epoch: 19/50.. Training Loss: 0.38384.. Valid Loss: 0.26423.. Valid Accuracy: 0.89978.. Area Under Curve: 0.95709
Epoch: 20/50.. Training Loss: 0.37842.. Valid Loss: 0.26729.. Valid Accuracy: 0.89978.. Area Under Curve: 0.95528
Epoch: 21/50.. Training Loss: 0.35959.. Valid Loss: 0.26464.. Valid Accuracy: 0.89107.. Area Under Curve: 0.95663
Epoch: 22/50.. Training Loss: 0.35680.. Valid Loss: 0.29077.. Valid Accuracy: 0.87800.. Area Under Curve: 0.95760
Epoch: 23/50.. Training Loss: 0.35464.. Valid Loss: 0.25870.. Valid Accuracy: 0.90196.. Area Under Curve: 0.95734
Epoch: 24/50.. Training Loss: 0.36316.. Valid Loss: 0.30004.. Valid Accuracy: 0.88235.. Area Under Curve: 0.95589
Epoch: 25/50.. Training Loss: 0.36476.. Valid Loss: 0.25689.. Valid Accuracy: 0.90196.. Area Under Curve: 0.95923

.
.
.

참고 자료

Invasive Hydrangeas : http://jenmenke.com/who-knew-hydrangeas-were-invasive/
CutMix Code(clovaai) : https://github.com/clovaai/CutMix-PyTorch
CutMix 설명 - 이유한님 : https://github.com/kaggler-tv/codes/blob/master/cutmix/cutmix.ipynb

0%