Introduction to PyTorch

9. Introduction to PyTorch#

Marc BUFFAT, mechanical department, University Lyon 1 pytorch

Certain content was co-constructed with the support of a generative AI tool then reread and validated by the teacher.

from validation.libIA_GPU import Init_torchGPU

try: cuda_dev
except NameError: cuda_dev = Init_torchGPU(4, 0)
if cuda_dev is None: cuda_dev = "cpu"
print(cuda_dev)

Max threads : 4 / used threads 4
Attention: no GPU available! using CPU
cpu

/home/buffat/venvs/jupyter/lib/python3.10/site-packages/torch/cuda/__init__.py:107: UserWarning: CUDA initialization: The NVIDIA driver on your system is too old (found version 9010). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:109.)
  return torch._C._cuda_getDeviceCount() > 0

9.1. Introduction#

PyTorch is an Open Source Python machine learning software library https://pytorch.org that builds on Torch developed by Meta (Facebook).

Just like Numpy, PyTorch is a Python library that allows you to perform optimized numerical calculations with multidimensional arrays (tensor under Pytporch, ndarray under Numpy).

NumPy is the basis for numerical computing in Python, while PyTorch builds on similar concepts (tensors) but adds essential features for artificial intelligence, such as GPU acceleration and automatic gradient calculation.

The syntax of PyTorch is similar to that of Numpy.

WARNING by default pytorch uses single precision reals (float32) for memory reasons, while numpy uses double pressure reals (float64).

You must therefore be careful when converting numpy to pytorch

9.2. Verifying the installation#

The main python library is called torch (and for imaging applications torchvision). For use on GPU, it requires the installation of the cuda library (use of Nvidia GPU for HPC scientific computing).

import numpy as np
import matplotlib.pyplot as plt
import torch
import torch.nn as nn

print(torch.cuda.is_available())
device = torch.device(cuda_dev)
print("CUDA device:",cuda_dev)
print("Torch CUDA device:",device," threads:",torch.get_num_threads(),torch.get_num_interop_threads())

False
CUDA device: cpu
Torch CUDA device: cpu  threads: 4 4

9.3. Basic objects: tensors#

Tensors are the fundamental objects of PyTorch.

Creation of a tensor:

  • Vector

import torch
x = torch.tensor([1.,2.,3.])
print(x)

  • Matrix:

A = torch.tensor([[1.,2.],
                  [3.,4.]])

  • Random vector:

x = torch.randn(5)

  • Random matrix:

A = torch.randn(3,4)

import torch

x = torch.tensor([1.,2.,3.])
print(x)
A = torch.tensor([[1.,2.],
                 [3.,4.]])
print(A)

tensor([1., 2., 3.])
tensor([[1., 2.],
        [3., 4.]])

9.4. Matrix operations#

Using vector notations similar to Numpy

x = torch.tensor([1.,2.,3.])
y = 2*x + 1
print(y)

Tensor product (matrix) @:

  • pytorch.dot

  • pytorch.tensordot

A = torch.randn(3,3)
B = torch.randn(3,3)
C = A @ B

x = torch.tensor([1.,2.,3.])
y = 2*x + 1
print(y)
C = x@y
print(C)

tensor([3., 5., 7.])
tensor(34.)

9.5. Automatic calculation of derivatives#

This is the essential functionality for Neural Networks. It allows you to numerically calculate the value of the derivative using a graph analysis with backtracking.

We activate the calculation of gradients with respect to a variable when creating the associated tensor.

x = torch.tensor([2.0], requires_grad=True)
y = x**2
y.backward()
print(x.grad)

x = torch.tensor([2.0], requires_grad=True)
y = x**2
y.backward()
print(x.grad)

tensor([4.])

9.6. More complex example#

x = torch.tensor([3.0], requires_grad=True)
f = x**3 + 2*x
f.backward()
print(x.grad)

PyTorch calculates the numerical value of the gradient for \(x=3\)

\[ 3x^2+2=29. \]
x = torch.tensor([3.0], requires_grad=True)
f = x**3 + 2*x
f.backward()
print(x.grad)

tensor([29.])

9.7. Neural networks#

A neural network is a class inherited from nn.Module. We must define two methods:

  • __init__(self): to define the network structure, for example a linear multi-layer network with hyperbolic tangent activation functions (‘tanh`).

  • forward(self,x): to define how to calculate the value of the neural network for an input x. In the case of a sequential network, we call each layer successively, which is simplified if we use the Sequential function and the associated method in forward

A neural network depends on a certain number of parameters, which we will determine by minimizing a cost function using a gradient optimization method.

9.7.1. Creation#

network definition:

import torch.nn as nn

class NeuralNet(nn.Module):

    def __init__(self):
        super().__init__()
        self.layers = nn.Sequential(
            nn.Linear(1,20),
            nn.Tanh(),
            nn.Linear(20,20),
            nn.Tanh(),
            nn.Linear(20,1)
        )
    def forward(self,x):
        return self.layers(x)

which is equivalent to


class NeuralNet(nn.Module):

    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(1,20),
        self.ac1 = nn.Tanh(),
        self.fc2 = nn.Linear(20,20),
        self.ac2 = nn.Tanh(),
        self.fc3 = nn.Linear(20,1)
        

    def forward(self,x):
        x = self.fc1(x)
        x = self.ac1(x)
        x = self.fc2(x)
        x = self.ac2(x)
        x = self.fc3(x)
        return x

creation/use:

model = NeuralNet()

import torch.nn as nn

class NeuralNet(nn.Module):

    def __init__(self):
        super().__init__()
        self.layers = nn.Sequential(
            nn.Linear(1,20),
            nn.Tanh(),
            nn.Linear(20,20),
            nn.Tanh(),
            nn.Linear(20,1)
        )
    def forward(self,x):
        return self.layers(x)
# creation
model = NeuralNet()

9.7.2. Prediction#

To calculate the value of the neural network, we simply use the value of the class instance by providing the value of the input tensor

x = torch.tensor([[0.5]])
y = model(x)
print(y)

x = torch.tensor([[0.5]])
y = model(x)
print(y)

tensor([[0.3094]], grad_fn=<AddmmBackward0>)

9.7.3. Cost function (or loss function)#

It is the difference between the expected value and the value of the neural network: i.e. between the data and the prediction. We generally calculate a quadratic difference provided by the MSELoss() function.

prediction = model(x)
target = torch.tensor([[1.5]])
loss = ((prediction-target)**2).mean()

or

criterion = nn.MSELoss()
loss = criterion(prediction,target)

prediction = model(x)
target = torch.tensor([[1.5]])
criterion = nn.MSELoss()
loss = criterion(prediction,target)
print(loss)

tensor(1.4175, grad_fn=<MseLossBackward0>)

9.7.4. Optimization method#

Choice of optimization method. A widely used algorithm is the Adam algorithm. Adam is a first-order gradient-based optimization algorithm for optimizing stochastic objective functions. It is based on adaptive estimates of moments of order 1 and 2 from a stochastic evaluation of the gradient. It is the elementary iteration of a stochastic gradient descent (SGD) algorithm.

Creating the optimizer:

optimizer = torch.optim.Adam(
    model.parameters(),
    lr=1e-3
)

optimizer = torch.optim.Adam(
    model.parameters(),
    lr=1e-3
)

9.7.5. Training loop#

This is the iterative loop of the stochastic gradient descent algorithm. The iteration variable is commonly called epoch to indicate that the gradient calculation is done using the entire data set. For certain very large problems, we can include inside this loop, an internal loop of subiterations to calculate the gradients only on a subset of the data (mini-batch).

for epoch in range(1000):
    optimizer.zero_grad()
    prediction = model(x)
    loss = criterion(prediction,target)
    loss.backward()
    optimizer.step()

The three important steps are:

  1. calculation of the loss function (loss);

  2. calculation of gradients (backward());

  3. update of parameters or weights (step()).

for epoch in range(1000):
    optimizer.zero_grad()
    prediction = model(x)
    loss = criterion(prediction,target)
    loss.backward()
    optimizer.step()

9.8. Gradient calculation#

9.8.1. First derivative with respect to inputs#

For PINNs, we do not drift with respect to the weights but with respect to the inputs: i.e. time or space.

Calculating the gradient:

t = torch.linspace(0,1,100).reshape(-1,1)
t.requires_grad = True
y = model(t)
dy = torch.autograd.grad(
    y,
    t,
    grad_outputs=torch.ones_like(y),
    create_graph=True
)[0]

This instruction calculates the numerical value \(\frac{dy}{dt}.\) for the values of the inputs t.

9.8.2. Second derivative#

d2y = torch.autograd.grad(
    dy,
    t,
    grad_outputs=torch.ones_like(dy),
    create_graph=True
)[0]

We obtain \(\frac{d^2y}{dt^2}.\)

It is this ability to calculate high-order derivatives that makes PyTorch particularly suitable for PINNs.

9.9. Save/load a template#

Python allows you to save the RNN model in a .pt extension file such as zip-file.

torch.save(model.state_dict(),"model.pt")

Loading:

model = NeuralNet()
model.load_state_dict(torch.load("model.pt"))
model.eval()

9.10. GPU usage#

One of the strong points of PyTorch is the possibility of using an Nvidia GPU to perform tensor calculations in an optimized manner. To do this, we select the device to use to perform the calculations (‘cuda’ for an Nvidia GPU or ‘CPU’ for calculation without GPU).

Be careful if the tensors are created on the GPU, they must be explicitly transferred to CPU memory to use Numpy and the analysis and plotting libraries. Be careful, the transfer of data between GPU and CPU is not instantaneous and can become a bottleneck for machine learning algorithms if we do not minimize them.

device = torch.device(
    "cuda" if torch.cuda.is_available()
    else "cpu"
)

model = model.to(device)
x = x.to(device)

Otherwise, we can in the Pytorch function for creating a tensor add the device parameter to specify the device (cpu or gpu)

x = torch.zeros([5,10],device=torch.device("cuda:0")

To transfer data from GPU to CPU we use the cpu() method

xc = x.cpu()

9.11. General diagram of a PyTorch project for machine learning/PINN#

Création des données
        │
        ▼
Création du réseau
        │
        ▼
Choix de la fonction de coût
        │
        ▼
Choix de l'optimiseur
        │
        ▼
Boucle d'entraînement
    ├── calcul de la valeur: forward()
    ├── calcul de la fonction coût/perte: loss
    ├── calcul du gradient: backward()
    └── itération suivante
        │
        ▼
Évaluation de l'approximation RNN ou PINN
Contents