{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "8e837f8b-7b49-42af-a0f5-e0ff21cab09d",
      "metadata": {},
      "source": [
        "# Introduction to PyTorch\n",
        "**Marc BUFFAT**, mechanical department, University Lyon 1\n",
        "![pytorch](images/python-pytorch.png)\n",
        "\n",
        "*Certain content was co-constructed with the support of a generative AI tool then reread and validated by the teacher.*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "e339cd92-1d1c-4831-bd6e-d5e7452edfd8",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Max threads : 4 / used threads 4\n",
            "Attention: no GPU available! using CPU\n",
            "cpu\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/home/buffat/venvs/jupyter/lib/python3.10/site-packages/torch/cuda/__init__.py:107: UserWarning: CUDA initialization: The NVIDIA driver on your system is too old (found version 9010). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:109.)\n",
            "  return torch._C._cuda_getDeviceCount() > 0\n"
          ]
        }
      ],
      "source": [
        "from validation.libIA_GPU import Init_torchGPU\n",
        "\n",
        "try: cuda_dev\n",
        "except NameError: cuda_dev = Init_torchGPU(4, 0)\n",
        "if cuda_dev is None: cuda_dev = \"cpu\"\n",
        "print(cuda_dev)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1b14747d-29d9-40c9-bd28-4838d065051d",
      "metadata": {},
      "source": [
        "## Introduction\n",
        "PyTorch is an Open Source Python machine learning software library [https://pytorch.org](https://pytorch.org) that builds on Torch developed by Meta (Facebook).\n",
        "\n",
        "Just like Numpy, PyTorch is a Python library that allows you to perform optimized numerical calculations with multidimensional arrays (`tensor` under Pytporch, `ndarray` under Numpy).\n",
        "\n",
        "NumPy is the basis for numerical computing in Python, while PyTorch builds on similar concepts (tensors) but adds essential features for artificial intelligence, such as GPU acceleration and automatic gradient calculation.\n",
        "\n",
        "The syntax of PyTorch is similar to that of Numpy.\n",
        "\n",
        "**WARNING** by default pytorch uses single precision reals (`float32`) for memory reasons, while numpy uses double pressure reals (`float64`).\n",
        "\n",
        "You must therefore be careful when converting `numpy` to `pytorch`"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "70716f4c-a536-420c-8441-9d8b5a0cbf4a",
      "metadata": {},
      "source": [
        "## Verifying the installation\n",
        "\n",
        "The main python library is called `torch` (and for imaging applications `torchvision`). For use on GPU, it requires the installation of the cuda library (use of Nvidia GPU for HPC scientific computing).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "700b6762-77b6-4292-baec-3fa5f2aa0df9",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "False\n",
            "CUDA device:  cpu\n",
            "Torch CUDA device:  cpu  threads: 4 4\n"
          ]
        }
      ],
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "\n",
        "print(torch.cuda.is_available())\n",
        "device = torch.device(cuda_dev)\n",
        "print(\"CUDA device:\",cuda_dev)\n",
        "print(\"Torch CUDA device:\",device,\" threads:\",torch.get_num_threads(),torch.get_num_interop_threads())\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8ef8947d-5892-49d8-b4ec-6b787b416998",
      "metadata": {},
      "source": [
        "## Basic objects: tensors\n",
        "\n",
        "Tensors are the fundamental objects of PyTorch.\n",
        "\n",
        "Creation of a tensor:\n",
        "\n",
        "- Vector\n",
        "\n",
        "```python\n",
        "import torch\n",
        "x = torch.tensor([1.,2.,3.])\n",
        "print(x)\n",
        "```\n",
        "\n",
        "- Matrix:\n",
        "\n",
        "```python\n",
        "A = torch.tensor([[1.,2.],\n",
        "                  [3.,4.]])\n",
        "```\n",
        "\n",
        "- Random vector:\n",
        "\n",
        "```python\n",
        "x = torch.randn(5)\n",
        "```\n",
        "\n",
        "- Random matrix:\n",
        "\n",
        "```python\n",
        "A = torch.randn(3,4)\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "34526f6d-5bcf-4e56-a8d8-899567c2c742",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor([1., 2., 3.])\n",
            "tensor([[1., 2.],\n",
            "        [3., 4.]])\n"
          ]
        }
      ],
      "source": [
        "import torch\n",
        "\n",
        "x = torch.tensor([1.,2.,3.])\n",
        "print(x)\n",
        "A = torch.tensor([[1.,2.],\n",
        "                 [3.,4.]])\n",
        "print(A)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f31f6736-e7c6-4fd3-826e-56c79f4bb365",
      "metadata": {},
      "source": [
        "## Matrix operations\n",
        "\n",
        "Using vector notations similar to Numpy\n",
        "\n",
        "```python\n",
        "x = torch.tensor([1.,2.,3.])\n",
        "y = 2*x + 1\n",
        "print(y)\n",
        "```\n",
        "\n",
        "Tensor product (matrix) @:\n",
        "\n",
        " - pytorch.dot\n",
        " - pytorch.tensordot\n",
        "\n",
        "```python\n",
        "A = torch.randn(3,3)\n",
        "B = torch.randn(3,3)\n",
        "C = A @ B\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "id": "34b59c02-734c-4ace-901e-368912b300b5",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor([3., 5., 7.])\n",
            "tensor(34.)\n"
          ]
        }
      ],
      "source": [
        "x = torch.tensor([1.,2.,3.])\n",
        "y = 2*x + 1\n",
        "print(y)\n",
        "C = x@y\n",
        "print(C)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e5cfe5ba-b970-48b0-bef5-f8eb69c0a86b",
      "metadata": {},
      "source": [
        "## Automatic calculation of derivatives\n",
        "\n",
        "This is the essential functionality for Neural Networks. It allows you to numerically calculate the value of the derivative using a graph analysis with backtracking.\n",
        "\n",
        "We activate the calculation of gradients with respect to a variable when creating the associated tensor.\n",
        "\n",
        "```python\n",
        "x = torch.tensor([2.0], requires_grad=True)\n",
        "y = x**2\n",
        "y.backward()\n",
        "print(x.grad)\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "76fb8d4c-de69-410f-aad7-21003eb821ce",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor([4.])\n"
          ]
        }
      ],
      "source": [
        "x = torch.tensor([2.0], requires_grad=True)\n",
        "y = x**2\n",
        "y.backward()\n",
        "print(x.grad)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e31c2705-8042-442e-8353-e185362549a2",
      "metadata": {},
      "source": [
        "## More complex example\n",
        "\n",
        "```python\n",
        "x = torch.tensor([3.0], requires_grad=True)\n",
        "f = x**3 + 2*x\n",
        "f.backward()\n",
        "print(x.grad)\n",
        "```\n",
        "\n",
        "PyTorch calculates the numerical value of the gradient for $x=3$\n",
        "\n",
        "$$\n",
        "3x^2+2=29.\n",
        "$$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "7fb6d027-2d93-48a8-9c6b-622ad0307769",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor([29.])\n"
          ]
        }
      ],
      "source": [
        "x = torch.tensor([3.0], requires_grad=True)\n",
        "f = x**3 + 2*x\n",
        "f.backward()\n",
        "print(x.grad)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c02e2482-bc4b-4643-941b-315d2368b38e",
      "metadata": {},
      "source": [
        "## Neural networks\n",
        "\n",
        "A neural network is a class inherited from `nn.Module`. We must define two methods:\n",
        "\n",
        "- `__init__(self)`: to define the network structure, for example a linear multi-layer network with hyperbolic tangent activation functions ('tanh`).\n",
        "- `forward(self,x)`: to define how to calculate the value of the neural network for an input `x`. In the case of a sequential network, we call each layer successively, which is simplified if we use the `Sequential` function and the associated method in `forward`\n",
        "\n",
        "A neural network depends on a certain number of parameters, which we will determine by minimizing a cost function using a gradient optimization method.\n",
        "\n",
        "### Creation\n",
        "\n",
        "**network definition:**\n",
        "\n",
        "```python\n",
        "import torch.nn as nn\n",
        "\n",
        "class NeuralNet(nn.Module):\n",
        "\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.layers = nn.Sequential(\n",
        "            nn.Linear(1,20),\n",
        "            nn.Tanh(),\n",
        "            nn.Linear(20,20),\n",
        "            nn.Tanh(),\n",
        "            nn.Linear(20,1)\n",
        "        )\n",
        "    def forward(self,x):\n",
        "        return self.layers(x)\n",
        "```\n",
        "which is equivalent to\n",
        "\n",
        "```python\n",
        "\n",
        "class NeuralNet(nn.Module):\n",
        "\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.fc1 = nn.Linear(1,20),\n",
        "        self.ac1 = nn.Tanh(),\n",
        "        self.fc2 = nn.Linear(20,20),\n",
        "        self.ac2 = nn.Tanh(),\n",
        "        self.fc3 = nn.Linear(20,1)\n",
        "        \n",
        "\n",
        "    def forward(self,x):\n",
        "        x = self.fc1(x)\n",
        "        x = self.ac1(x)\n",
        "        x = self.fc2(x)\n",
        "        x = self.ac2(x)\n",
        "        x = self.fc3(x)\n",
        "        return x\n",
        "```\n",
        "\n",
        "\n",
        "**creation/use:**\n",
        "\n",
        "```python\n",
        "model = NeuralNet()\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "bf76dad9-c503-4e00-8d39-7b290719c826",
      "metadata": {},
      "outputs": [],
      "source": [
        "import torch.nn as nn\n",
        "\n",
        "class NeuralNet(nn.Module):\n",
        "\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.layers = nn.Sequential(\n",
        "            nn.Linear(1,20),\n",
        "            nn.Tanh(),\n",
        "            nn.Linear(20,20),\n",
        "            nn.Tanh(),\n",
        "            nn.Linear(20,1)\n",
        "        )\n",
        "    def forward(self,x):\n",
        "        return self.layers(x)\n",
        "# creation\n",
        "model = NeuralNet()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "40dd3dba-b4c9-419f-9312-4ff4c8861b75",
      "metadata": {},
      "source": [
        "### Prediction\n",
        "\n",
        "To calculate the value of the neural network, we simply use the value of the class instance by providing the value of the input tensor\n",
        "\n",
        "```python\n",
        "x = torch.tensor([[0.5]])\n",
        "y = model(x)\n",
        "print(y)\n",
        "```\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "cb3bea87-2051-42cd-ae54-17db57fc93b8",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor([[-0.0164]], grad_fn=<AddmmBackward0>)\n"
          ]
        }
      ],
      "source": [
        "x = torch.tensor([[0.5]])\n",
        "y = model(x)\n",
        "print(y)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "432d4206-37cb-4590-9854-4994d4ce2851",
      "metadata": {},
      "source": [
        "### Cost function (or loss function)\n",
        "\n",
        "It is the difference between the expected value and the value of the neural network: i.e. between the data and the prediction. We generally calculate a quadratic difference provided by the `MSELoss()` function.\n",
        "\n",
        "```python\n",
        "prediction = model(x)\n",
        "target = torch.tensor([[1.5]])\n",
        "loss = ((prediction-target)**2).mean()\n",
        "```\n",
        "\n",
        "or\n",
        "\n",
        "```python\n",
        "criterion = nn.MSELoss()\n",
        "loss = criterion(prediction,target)\n",
        "```\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "id": "786fa9c5-b0e6-4363-9fa0-135fc62e4d55",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor(2.2996, grad_fn=<MseLossBackward0>)\n"
          ]
        }
      ],
      "source": [
        "prediction = model(x)\n",
        "target = torch.tensor([[1.5]])\n",
        "criterion = nn.MSELoss()\n",
        "loss = criterion(prediction,target)\n",
        "print(loss)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2240c992-9637-41d8-8c3e-6a22ccdbd253",
      "metadata": {},
      "source": [
        "### Optimization method\n",
        "\n",
        "Choice of optimization method.  A widely used algorithm is the Adam algorithm. Adam is a first-order gradient-based optimization algorithm for optimizing stochastic objective functions. It is based on adaptive estimates of moments of order 1 and 2 from a stochastic evaluation of the gradient. It is the elementary iteration of a stochastic gradient descent (SGD) algorithm.\n",
        "\n",
        "**Creating the optimizer:**\n",
        "\n",
        "```python\n",
        "optimizer = torch.optim.Adam(\n",
        "    model.parameters(),\n",
        "    lr=1e-3\n",
        ")\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "id": "74bb0780-d0b1-4f68-a279-ff1c0fd2002f",
      "metadata": {},
      "outputs": [],
      "source": [
        "optimizer = torch.optim.Adam(\n",
        "    model.parameters(),\n",
        "    lr=1e-3\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c21db6c4-f7c6-4068-bd29-764fd13947e1",
      "metadata": {},
      "source": [
        "### Training loop\n",
        "This is the iterative loop of the stochastic gradient descent algorithm. The iteration variable is commonly called `epoch` to indicate that the gradient calculation is done using the entire data set. \n",
        "For certain very large problems, we can include inside this loop, an internal loop of subiterations to calculate the gradients only on a subset of the data (mini-batch).  \n",
        "\n",
        "```python\n",
        "for epoch in range(1000):\n",
        "    optimizer.zero_grad()\n",
        "    prediction = model(x)\n",
        "    loss = criterion(prediction,target)\n",
        "    loss.backward()\n",
        "    optimizer.step()\n",
        "```\n",
        "\n",
        "The three important steps are:\n",
        "\n",
        "1. calculation of the loss function (`loss`);\n",
        "2. calculation of gradients (`backward()`);\n",
        "3. update of parameters or weights (`step()`).\n",
        "\n",
        "---"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "id": "2a4dceb7-aa6e-4665-af0a-1f6af7114ad9",
      "metadata": {},
      "outputs": [],
      "source": [
        "for epoch in range(1000):\n",
        "    optimizer.zero_grad()\n",
        "    prediction = model(x)\n",
        "    loss = criterion(prediction,target)\n",
        "    loss.backward()\n",
        "    optimizer.step()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0e21c130-54d1-4460-832b-b2ee01d4f17d",
      "metadata": {},
      "source": [
        "## Gradient calculation\n",
        "\n",
        "### First derivative with respect to inputs\n",
        "\n",
        "For PINNs, we do not drift with respect to the weights but with respect to the inputs: i.e. time or space.\n",
        "\n",
        "Calculating the gradient:\n",
        "\n",
        "```python\n",
        "t = torch.linspace(0,1,100).reshape(-1,1)\n",
        "t.requires_grad = True\n",
        "y = model(t)\n",
        "dy = torch.autograd.grad(\n",
        "    y,\n",
        "    t,\n",
        "    grad_outputs=torch.ones_like(y),\n",
        "    create_graph=True\n",
        ")[0]\n",
        "```\n",
        "\n",
        "This instruction calculates the numerical value\n",
        "$\\frac{dy}{dt}.$ for the values of the inputs t.\n",
        "\n",
        "---\n",
        "\n",
        "### Second derivative\n",
        "\n",
        "```python\n",
        "d2y = torch.autograd.grad(\n",
        "    dy,\n",
        "    t,\n",
        "    grad_outputs=torch.ones_like(dy),\n",
        "    create_graph=True\n",
        ")[0]\n",
        "```\n",
        "\n",
        "We obtain\n",
        "$\\frac{d^2y}{dt^2}.$\n",
        "\n",
        "It is this ability to calculate high-order derivatives that makes PyTorch particularly suitable for PINNs.\n",
        "\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "133fecf1-dd9d-4cbd-b534-7f614593fe7f",
      "metadata": {},
      "source": [
        "## Save/load a template\n",
        "\n",
        "Python allows you to save the RNN model in a `.pt` extension file such as zip-file.\n",
        "\n",
        "```python\n",
        "torch.save(model.state_dict(),\"model.pt\")\n",
        "```\n",
        "\n",
        "Loading:\n",
        "\n",
        "```python\n",
        "model = NeuralNet()\n",
        "model.load_state_dict(torch.load(\"model.pt\"))\n",
        "model.eval()\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0ab662a4-a574-43ae-a806-bad9cb63bc3c",
      "metadata": {},
      "source": [
        "## GPU usage\n",
        "One of the strong points of PyTorch is the possibility of using an Nvidia GPU to perform tensor calculations in an optimized manner.  To do this, we select the device to use to perform the calculations ('cuda' for an Nvidia GPU or 'CPU' for calculation without GPU).\n",
        "\n",
        "Be careful if the tensors are created on the GPU, they must be explicitly transferred to CPU memory to use Numpy and the analysis and plotting libraries. Be careful, the transfer of data between GPU and CPU is not instantaneous and can become a bottleneck for machine learning algorithms if we do not minimize them. \n",
        "\n",
        "```python\n",
        "device = torch.device(\n",
        "    \"cuda\" if torch.cuda.is_available()\n",
        "    else \"cpu\"\n",
        ")\n",
        "\n",
        "model = model.to(device)\n",
        "x = x.to(device)\n",
        "```\n",
        "\n",
        "Otherwise, we can in the Pytorch function for creating a tensor add the `device` parameter to specify the device (cpu or gpu)\n",
        "\n",
        "\n",
        "```python\n",
        "x = torch.zeros([5,10],device=torch.device(\"cuda:0\")\n",
        "```\n",
        "\n",
        "To transfer data from GPU to CPU we use the `cpu()` method\n",
        "\n",
        "```python\n",
        "xc = x.cpu()\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8509c4cc-172b-4f97-93c3-fcdca8a927ec",
      "metadata": {},
      "source": [
        "## General diagram of a PyTorch project for machine learning/PINN\n",
        "\n",
        "```text\n",
        "Création des données\n",
        "        │\n",
        "        ▼\n",
        "Création du réseau\n",
        "        │\n",
        "        ▼\n",
        "Choix de la fonction de coût\n",
        "        │\n",
        "        ▼\n",
        "Choix de l'optimiseur\n",
        "        │\n",
        "        ▼\n",
        "Boucle d'entraînement\n",
        "    ├── calcul de la valeur: forward()\n",
        "    ├── calcul de la fonction coût/perte: loss\n",
        "    ├── calcul du gradient: backward()\n",
        "    └── itération suivante\n",
        "        │\n",
        "        ▼\n",
        "Évaluation de l'approximation RNN ou PINN\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "77c0fa86-45ee-46bd-b4d3-4e980e12bcfe",
      "metadata": {},
      "source": [
        "### Recommended approach\n",
        "\n",
        "To learn how to effectively use PyTorch in machine learning / PINN, you can follow this approach:\n",
        "\n",
        "1. Learn how to manipulate tensors and basic operations.\n",
        "2. Understand the automatic calculation of gradients (`autograd`).\n",
        "3. Build simple neural networks with `nn.Module`.\n",
        "4. Master the training loop (loss, backpropagation, optimizer).\n",
        "5. Calculate derivatives with respect to inputs with `torch.autograd.grad`.\n",
        "6. Implement a first PINN for a first order ODE, then extend to second order ODEs and finally to PDEs.\n",
        "\n",
        "Once these basics have been acquired, the transition to PINNs is natural, because they use the same building blocks of PyTorch: neural network, automatic differentiation and optimization. The main difference lies in the loss function, which integrates the residual of the differential equations and the initial or boundary conditions."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.12"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}