{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "6c22b917",
      "metadata": {
        "id": "6c22b917"
      },
      "source": [
        "# Лабораторная работа 5\n",
        "# Генеративные модели изображений: Stable Diffusion и Vision Transformer\n",
        "\n",
        "**Уровень:** бакалавриат\n",
        "\n",
        "## Общая идея работы\n",
        "\n",
        "В лабораторной работе рассматривается полный pipeline:\n",
        "\n",
        "```\n",
        "Текстовое описание\n",
        "        |\n",
        "        v\n",
        "Stable Diffusion\n",
        "        |\n",
        "        v\n",
        "Синтетический датасет изображений\n",
        "        |\n",
        "        v\n",
        "Vision Transformer\n",
        "        |\n",
        "        v\n",
        "Классификация изображений\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f7d88e89",
      "metadata": {
        "id": "f7d88e89"
      },
      "source": [
        "# Часть 1. Генерация изображений с помощью Stable Diffusion\n",
        "\n",
        "Генеративные модели позволяют создавать изображения по текстовому описанию.\n",
        "\n",
        "Основные понятия:\n",
        "\n",
        "- prompt — описание желаемого изображения;\n",
        "- negative prompt — описание нежелательных элементов;\n",
        "- inference steps — количество шагов генерации;\n",
        "- guidance scale — степень следования текстовому описанию.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e4f65b89",
      "metadata": {
        "id": "e4f65b89"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "from diffusers import StableDiffusionPipeline\n",
        "\n",
        "model_id = \"runwayml/stable-diffusion-v1-5\"\n",
        "\n",
        "pipe = StableDiffusionPipeline.from_pretrained(\n",
        "    model_id,\n",
        "    torch_dtype=torch.float16\n",
        ")\n",
        "\n",
        "pipe = pipe.to(\"cuda\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a439f801",
      "metadata": {
        "id": "a439f801"
      },
      "source": [
        "## Задание 1. Базовая генерация\n",
        "\n",
        "Создайте изображения по следующим запросам:\n",
        "\n",
        "1. Реалистичное животное.\n",
        "2. Фантастическое существо.\n",
        "3. Изображение в стиле известного художественного направления.\n",
        "\n",
        "Сравните:\n",
        "\n",
        "- насколько хорошо модель понимает описание;\n",
        "- какие детали теряются;\n",
        "- какие ошибки появляются.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3abd6810",
      "metadata": {
        "id": "3abd6810"
      },
      "outputs": [],
      "source": [
        "# 1. Реалистичное животное\n",
        "\n",
        "prompt = \"A fluffy cat with green eyes, realistic photography, high detail\"\n",
        "\n",
        "image_animal = pipe(\n",
        "    prompt,\n",
        "    num_inference_steps=50,\n",
        "    guidance_scale=7.5\n",
        ").images[0]\n",
        "\n",
        "image_animal\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "35uQiqIVdSRG"
      },
      "id": "35uQiqIVdSRG",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "14d0ccaf",
      "metadata": {
        "id": "14d0ccaf"
      },
      "source": [
        "# Задание 2. Prompt engineering для изображений\n",
        "\n",
        "Исследуйте влияние формулировки запроса.\n",
        "\n",
        "Используйте несколько вариантов:\n",
        "\n",
        "1. простой prompt;\n",
        "2. prompt с описанием стиля;\n",
        "3. prompt с детализацией объекта;\n",
        "4. prompt профессионального фотографа.\n",
        "\n",
        "Сравните результаты.\n",
        "\n",
        "Сделайте вывод о влиянии текста на качество генерации.\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "F3kCtuIHYaVX"
      },
      "id": "F3kCtuIHYaVX",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "6159c3af",
      "metadata": {
        "id": "6159c3af"
      },
      "source": [
        "# Задание 3. Negative prompt\n",
        "\n",
        "Используйте negative prompt для удаления нежелательных элементов.\n",
        "\n",
        "Исследуйте:\n",
        "\n",
        "- лишние объекты;\n",
        "- неправильные цвета;\n",
        "- артефакты;\n",
        "- искажения.\n",
        "\n",
        "Ответьте:\n",
        "\n",
        "Почему negative prompt улучшает результат?\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "P4l09--2Ya5-"
      },
      "id": "P4l09--2Ya5-",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "2d727859",
      "metadata": {
        "id": "2d727859"
      },
      "source": [
        "# Часть 2. Создание синтетического датасета\n",
        "\n",
        "Генеративные модели могут использоваться для создания данных.\n",
        "\n",
        "Задача:\n",
        "\n",
        "Создать набор изображений двух классов:\n",
        "\n",
        "- cats;\n",
        "- dogs.\n",
        "\n",
        "Полученный датасет будет использоваться для обучения классификатора.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "74aa006a",
      "metadata": {
        "id": "74aa006a"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "dataset_path = \"synthetic_dataset\"\n",
        "\n",
        "classes = [\"cats\", \"dogs\"]\n",
        "\n",
        "for cls in classes:\n",
        "    os.makedirs(\n",
        "        os.path.join(dataset_path, cls),\n",
        "        exist_ok=True\n",
        "    )\n",
        "\n",
        "print(\"Папки созданы\")"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "4wnVfztfYfgm"
      },
      "id": "4wnVfztfYfgm",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "97e1632c",
      "metadata": {
        "id": "97e1632c"
      },
      "source": [
        "## Задание 4\n",
        "\n",
        "Создайте минимум:\n",
        "\n",
        "- 100 изображений кошек;\n",
        "- 100 изображений собак.\n",
        "\n",
        "Обеспечьте разнообразие:\n",
        "\n",
        "- породы;\n",
        "- фон;\n",
        "- стиль изображения;\n",
        "- положение объекта.\n",
        "\n",
        "Проанализируйте:\n",
        "\n",
        "- есть ли признаки, которые модель генерации повторяет слишком часто?\n",
        "- насколько разнообразен датасет?\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "FZ_gG13eYgFX"
      },
      "id": "FZ_gG13eYgFX",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "de393369",
      "metadata": {
        "id": "de393369"
      },
      "source": [
        "# Часть 3. Обучение Vision Transformer\n",
        "\n",
        "Теперь используем созданный датасет для обучения модели классификации.\n",
        "\n",
        "Используется архитектура:\n",
        "\n",
        "Vision Transformer (ViT)\n",
        "\n",
        "Идея:\n",
        "\n",
        "```\n",
        "Изображение\n",
        "    |\n",
        "    v\n",
        "ViT encoder\n",
        "    |\n",
        "    v\n",
        "Класс изображения\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Задание 5. Обучение классификатора\n",
        "\n",
        "Настройте обучение:\n",
        "\n",
        "- batch_size;\n",
        "- learning_rate;\n",
        "- количество эпох.\n",
        "\n",
        "Используйте:\n",
        "\n",
        "- TrainingArguments;\n",
        "- Trainer;\n",
        "- accuracy как основную метрику.\n",
        "\n",
        "Исследуйте влияние количества данных на качество модели."
      ],
      "metadata": {
        "id": "ONu55ARjmj9U"
      },
      "id": "ONu55ARjmj9U"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "928ec81b",
      "metadata": {
        "id": "928ec81b"
      },
      "outputs": [],
      "source": [
        "!pip install transformers datasets accelerate evaluate\n",
        "from transformers import ViTForImageClassification, ViTImageProcessor\n",
        "\n",
        "labels = [\"cat\", \"dog\"]\n",
        "\n",
        "model_name = \"google/vit-base-patch16-224-in21k\"\n",
        "\n",
        "processor = ViTImageProcessor.from_pretrained(\n",
        "    model_name\n",
        ")\n",
        "\n",
        "model = ViTForImageClassification.from_pretrained(\n",
        "    model_name,\n",
        "    num_labels=2,\n",
        "    id2label={0:\"cat\",1:\"dog\"},\n",
        "    label2id={\"cat\":0,\"dog\":1}\n",
        ")\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from torchvision.datasets import ImageFolder\n",
        "from torchvision.transforms import Compose, Resize, ToTensor, Normalize"
      ],
      "metadata": {
        "id": "Zyp066ohfdBX"
      },
      "id": "Zyp066ohfdBX",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Преобразование изображений под ViT"
      ],
      "metadata": {
        "id": "NSpvL9h2fiaV"
      },
      "id": "NSpvL9h2fiaV"
    },
    {
      "cell_type": "code",
      "source": [
        "transform = Compose([\n",
        "    Resize((224,224)),\n",
        "    ToTensor(),\n",
        "    Normalize(\n",
        "        mean=processor.image_mean,\n",
        "        std=processor.image_std\n",
        "    )\n",
        "])"
      ],
      "metadata": {
        "id": "Ac5gkDpNfi4d"
      },
      "id": "Ac5gkDpNfi4d",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Создание синтетического датасета"
      ],
      "metadata": {
        "id": "KxWoS5VpfzIZ"
      },
      "id": "KxWoS5VpfzIZ"
    },
    {
      "cell_type": "code",
      "source": [
        "dataset = ImageFolder(\n",
        "    \"synthetic_dataset\",\n",
        "    transform=transform\n",
        ")\n",
        "\n",
        "\n",
        "print(dataset.classes)\n",
        "print(\"Количество изображений:\", len(dataset))"
      ],
      "metadata": {
        "id": "gTVeu7qsf2dP"
      },
      "id": "gTVeu7qsf2dP",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Разделение на train/test"
      ],
      "metadata": {
        "id": "g2bhC56ogACn"
      },
      "id": "g2bhC56ogACn"
    },
    {
      "cell_type": "code",
      "source": [
        "from torch.utils.data import random_split\n",
        "\n",
        "\n",
        "train_size = int(0.8 * len(dataset))\n",
        "test_size = len(dataset) - train_size\n",
        "\n",
        "\n",
        "train_dataset, test_dataset = random_split(\n",
        "    dataset,\n",
        "    [train_size, test_size]\n",
        ")\n",
        "\n",
        "\n",
        "print(len(train_dataset))\n",
        "print(len(test_dataset))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 218
        },
        "collapsed": true,
        "id": "2qq8SNtSgCnW",
        "outputId": "b59e514c-7e1b-4b77-f712-25d476695aaf"
      },
      "id": "2qq8SNtSgCnW",
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "name 'dataset' is not defined",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m/tmp/ipykernel_3695/2120648135.py\u001b[0m in \u001b[0;36m<cell line: 0>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m train_size = int(\n\u001b[0;32m----> 5\u001b[0;31m     \u001b[0;36m0.8\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m )\n\u001b[1;32m      7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mNameError\u001b[0m: name 'dataset' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Адаптиция датасета под Trainer"
      ],
      "metadata": {
        "id": "LkojicdXlXf1"
      },
      "id": "LkojicdXlXf1"
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "\n",
        "\n",
        "class ViTDataset(torch.utils.data.Dataset):\n",
        "\n",
        "    def __init__(self, dataset):\n",
        "        self.dataset = dataset\n",
        "\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.dataset)\n",
        "\n",
        "\n",
        "    def __getitem__(self, idx):\n",
        "\n",
        "        image, label = self.dataset[idx]\n",
        "\n",
        "        return {\n",
        "            \"pixel_values\": image,\n",
        "            \"labels\": torch.tensor(label)\n",
        "        }\n",
        "train_dataset = ViTDataset(train_dataset)\n",
        "\n",
        "test_dataset = ViTDataset(test_dataset)"
      ],
      "metadata": {
        "id": "pYvMpstrlbLx"
      },
      "id": "pYvMpstrlbLx",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "from evaluate import load\n",
        "\n",
        "\n",
        "accuracy = load(\"accuracy\")\n",
        "\n",
        "\n",
        "def compute_metrics(eval_pred):\n",
        "\n",
        "    logits, labels = eval_pred\n",
        "\n",
        "    predictions = np.argmax(\n",
        "        logits,\n",
        "        axis=-1\n",
        "    )\n",
        "\n",
        "    return accuracy.compute(\n",
        "        predictions=predictions,\n",
        "        references=labels\n",
        "    )"
      ],
      "metadata": {
        "id": "7preWz-ghYBV"
      },
      "id": "7preWz-ghYBV",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import TrainingArguments\n",
        "\n",
        "\n",
        "training_args = TrainingArguments(\n",
        "\n",
        "    output_dir=\"./vit_cats_dogs\",\n",
        "\n",
        "    num_train_epochs=5,\n",
        "\n",
        "    per_device_train_batch_size=8,\n",
        "    per_device_eval_batch_size=8,\n",
        "\n",
        "    learning_rate=5e-5,\n",
        "\n",
        "    eval_strategy=\"epoch\",\n",
        "\n",
        "    save_strategy=\"epoch\",\n",
        "\n",
        "    logging_strategy=\"epoch\",\n",
        "\n",
        "    load_best_model_at_end=True,\n",
        "\n",
        "    metric_for_best_model=\"accuracy\",\n",
        "\n",
        "    greater_is_better=True,\n",
        "\n",
        "    report_to=\"none\"\n",
        ")"
      ],
      "metadata": {
        "id": "Rg76lUxih-FK"
      },
      "id": "Rg76lUxih-FK",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Создание Trainer"
      ],
      "metadata": {
        "id": "FFIX5k9eh--Z"
      },
      "id": "FFIX5k9eh--Z"
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import Trainer\n",
        "\n",
        "\n",
        "trainer = Trainer(\n",
        "\n",
        "    model=model,\n",
        "\n",
        "    args=training_args,\n",
        "\n",
        "    train_dataset=train_dataset,\n",
        "\n",
        "    eval_dataset=test_dataset,\n",
        "\n",
        "    compute_metrics=compute_metrics\n",
        ")"
      ],
      "metadata": {
        "id": "XKUjwHHAiCME"
      },
      "id": "XKUjwHHAiCME",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Обучение модели"
      ],
      "metadata": {
        "id": "Zehyu_2mmFgw"
      },
      "id": "Zehyu_2mmFgw"
    },
    {
      "cell_type": "code",
      "source": [
        "trainer.train()"
      ],
      "metadata": {
        "id": "rJlbXkV1mHZH"
      },
      "id": "rJlbXkV1mHZH",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Проверка качества"
      ],
      "metadata": {
        "id": "U89-fGzXmLkv"
      },
      "id": "U89-fGzXmLkv"
    },
    {
      "cell_type": "code",
      "source": [
        "results = trainer.evaluate()\n",
        "\n",
        "print(results)"
      ],
      "metadata": {
        "id": "mF0GLa9KmM8Q"
      },
      "id": "mF0GLa9KmM8Q",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "8f390bce",
      "metadata": {
        "id": "8f390bce"
      },
      "source": [
        "# Задание 6. Анализ результатов\n",
        "\n",
        "Постройте графики:\n",
        "\n",
        "- accuracy;\n",
        "- training loss;\n",
        "- validation loss.\n",
        "\n",
        "Ответьте:\n",
        "\n",
        "1. Переобучилась ли модель?\n",
        "2. Достаточно ли разнообразен синтетический датасет?\n",
        "3. Какие изображения классификатор распознает хуже?\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "\n",
        "\n",
        "# Получаем историю обучения Trainer\n",
        "\n",
        "history = pd.DataFrame(\n",
        "    trainer.state.log_history\n",
        ")\n",
        "\n",
        "print(history)"
      ],
      "metadata": {
        "id": "pbCVANeeYmXu"
      },
      "id": "pbCVANeeYmXu",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "XV3NpPJQnFBK"
      },
      "id": "XV3NpPJQnFBK",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "S-o06p2qnFKc"
      },
      "id": "S-o06p2qnFKc",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "m-aJvPGSYx_B"
      },
      "id": "m-aJvPGSYx_B",
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}