Introduction to Kornia

Kornia is a differentiable computer vision library for PyTorch

It consists of a set of routines and differentiable modules to solve generic computer vision problems. At its core, the package uses PyTorch as its main backend both for efficiency and to take advantage of the reverse-mode auto-differentiation to define and compute the gradient of complex functions.

Inspired by OpenCV, this library is composed by a subset of packages containing operators that can be inserted within neural networks to train models to perform image transformations, epipolar geometry, depth estimation, and low-level image processing such as filtering and edge detection that operate directly on tensors.

Kornia

Why Kornia ?

With Kornia we fill the gap within the PyTorch ecosystem introducing a computer vision library that implements standard vision algorithms taking advantage of the different properties that modern frameworks for deep learning like PyTorch can provide:

  1. Differentiability for commodity avoiding to write derivative functions for complex loss functions.
  2. Transparency to perform parallel or serial computing eitherin CPU or GPU devices using batches in a common API.
  3. Distributed for computing large-scale applications.
  4. Production ready using the JIT compiler.

Hightlighted Features

At a granular level, Kornia is a library that consists of the following components:

Component

Description

kornia

a Differentiable Computer Vision library like OpenCV, with strong GPU support

kornia.color

a set of routines to perform color space conversions

kornia.contrib

a compilation of user contrib and experimental operators

kornia.feature

a module to perform feature detection

kornia.filters

a module to perform image filtering and edge detection

kornia.geometry

a geometric computer vision library to perform image transformations, 3D linear algebra and conversions using different camera models

kornia.losses

a stack of loss functions to solve different vision tasks

kornia.utils

image to tensor utilities and metrics for vision problems

Installation

pip install kornia

Denoise image using total variation


Loss in iteration 0 of 500: 3.085
Loss in iteration 25 of 500: 2.933
Loss in iteration 50 of 500: 2.727
Loss in iteration 75 of 500: 2.536
Loss in iteration 100 of 500: 2.362
Loss in iteration 125 of 500: 2.206
Loss in iteration 150 of 500: 2.066
Loss in iteration 175 of 500: 1.941
Loss in iteration 200 of 500: 1.829
Loss in iteration 225 of 500: 1.730
Loss in iteration 250 of 500: 1.642
Loss in iteration 275 of 500: 1.564
Loss in iteration 300 of 500: 1.495
Loss in iteration 325 of 500: 1.434
Loss in iteration 350 of 500: 1.381
Loss in iteration 375 of 500: 1.333
Loss in iteration 400 of 500: 1.292
Loss in iteration 425 of 500: 1.255
Loss in iteration 450 of 500: 1.223
Loss in iteration 475 of 500: 1.194

Implementation
import kornia
import cv2
import numpy as np

import matplotlib.pyplot as plt

# read the image with OpenCV
img: np.ndarray = cv2.imread('./data/doraemon.png')
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) / 255.0
img = img + np.random.normal(loc=0.0, scale=0.1, size=img.shape)
img = np.clip(img, 0.0, 1.0)

# convert to torch tensor
noisy_image: torch.tensor = kornia.image_to_tensor(img).squeeze()  # CxHxW


# define the total variation denoising network
class TVDenoise(torch.nn.Module):
    def __init__(self, noisy_image):
        super(TVDenoise, self).__init__()
        self.l2_term = torch.nn.MSELoss(reduction='mean')
        self.regularization_term = kornia.losses.TotalVariation()
        # create the variable which will be optimized to produce the noise free image
        self.clean_image = torch.nn.Parameter(data=noisy_image.clone(), requires_grad=True)
        self.noisy_image = noisy_image

    def forward(self):
        return self.l2_term(self.clean_image, self.noisy_image) + 0.0001 * self.regularization_term(self.clean_image)

    def get_clean_image(self):
        return self.clean_image

tv_denoiser = TVDenoise(noisy_image)

# define the optimizer to optimize the 1 parameter of tv_denoiser
optimizer = torch.optim.SGD(tv_denoiser.parameters(), lr=0.1, momentum=0.9)

# run the optimization loop
num_iters = 500
for i in range(num_iters):
    optimizer.zero_grad()
    loss = tv_denoiser()
    if i % 25 == 0:
        print("Loss in iteration {} of {}: {:.3f}".format(i, num_iters, loss.item()))
    loss.backward()
    optimizer.step()

# convert back to numpy
img_clean: np.ndarray = kornia.tensor_to_image(tv_denoiser.get_clean_image())

# Create the plot
fig, axs = plt.subplots(1, 2, figsize=(16, 10))
axs = axs.ravel()

axs[0].axis('off')
axs[0].set_title('Noisy image')
axs[0].imshow(img)

axs[1].axis('off')
axs[1].set_title('Cleaned image')
axs[1].imshow(img_clean)

plt.show()