By Zarxrax
Updated 9/14/2024
Overview
This guide provides an overview on how to use community-made AI upscaling models to improve the quality of anime videos. These models can generally provide the best overall quality when it comes to upscaling and cleaning up anime video.
A model can simply be thought of as a blob of computer code that has been trained to modify an image in a particular way. There are a lot of different community-made models to choose from. Some models don’t even do upscaling–they might do other tasks like remove certain types of artifacts from the image. Lots of different models can be found at OpenModelDB. I will recommend a few models to start with later in this guide.
There are a few important things to know about models. First, they have a fixed upscaling factor--typically 2x or 4x. So for example, if you have a video file that is 852x480, using a 2x model on it will result in a file that is 1704x960. In order to get it to a standard resolution like 1920x1080, you will need to follow up with a normal resize operation after upscaling.
Models have also been trained on a variety of different "architectures". When browsing models on OpenModelDB, it will tell you which architecture each model uses. All that you really need to know is that some architectures are faster, while others are slower but potentially higher quality. The following architectures are quite fast, and work well for video: Compact, SPAN, Real-CUGAN. Slightly slower are RealPLKSR or anything with "lite" or "light" in the model name. Anything else might be quite slow, but you can try them yourself to see.
The other thing to know is that models can come in different file formats. The most common format is .pth, however .onnx is also common. Models can be easily converted between these two formats, so don't worry if a particular model is not in the format that you need.
Another thing to know is that upscaling can potentially be quite slow, and will utilize your GPU. Nvidia GPUs perform best, but AMD Radeon GPUs are also alright. If your computer doesn't have a dedicated GPU, then upscaling will likely be too slow to be feasible.
Along with models, you will need a piece of software that will actually let you use the model. In this guide, we will use the following tools and websites:
Windows users should use VideoJaNai to upscale. chaiNNer can be used by Mac and Linux users (it's just slower than VideoJaNai).
chaiNNer will also be used to convert model files between .pth and .onnx formats.
You may also find chaiNNer useful for testing various models to see what kind of effect they have on your videos. You can load a single image into chaiNNer to upscale, which will allow you to test models much faster than processing an entire video.
Recommended Models
Model Name - Description
(2x) AniSD Suite - This is a collection of models for dealing with DVD sources. Because there are so many models, it can be slightly confusing, so I highly recommend reading over this wiki page which goes into detail about the different models.
(2x) AnimeJaNai V3 - This model is for upscaling anime from 1080p to 4k. The effect is quite subtle, so it is really only effective on relatively high quality sources.
(2x) LD-Anime_Compact - Upscales while fixing numerous video problems, including: noise/grain, compression artifacts, rainbows, dot crawl, halos and color bleed. Can over-smooth some textures though.
(1x) BleedOut Compact - Helps repair color bleed and heavy chroma noise that may be present on some older footage
(1x) Dotzilla Compact - Wipes out dot crawl and rainbows in animation.
Of course, if none of these seem to work well for your source, feel free to try out any other models in the model database to see if there are any that work better for you.
How to Set up and use VideoJaNai
VideoJaNai only runs on Windows, so if you use Linux or Mac, skip ahead to the section on using chaiNNer.
Otherwise, go to the releases page for VideoJaNai and look for the "Setup.exe" file from the latest release. Simply download and run the file to install it.
The interface to VideoJaNai should be fairly straightforward. Here is an overview of the main parts:
Video Input and Output settings
Input Video - This is where you select the video file that you want to upscale.
Output Folder/Filename - Tell it where to save the upscaled video file.
FFmpeg Output Video Settings - This text box allows you to provide custom FFmpeg commands. Most people should ignore this.
FFmpeg Presets - Some simple presets to select the output format that your video should be encoded to. I highly recommend choosing the "Lossless" option here, and then compressing the resulting file with AMVtool.
Upscaling Settings
ONNX Model Path - This is where you can select a model file. Note that VideoJaNai requires models to be in the .onnx format, so if you want to use a .pth model file, it must be converted first. See the section below about convering model files.
Add Model (button) - Clicking this button allows you to add multiple models. They will run in the same order that they are added.
Final Resize Height - This option can be used to perform a resize after upscaling. Note that if you plan to encode in AMVtool, you can simply resize there as well.
Upscaling Backend - You typically want to select "DirectML" here. The "TensorRT" option can provide faster performance on Nvidia GPUs, but it will require VideoJaNai to generate an engine file prior to upscaling. That process can take upwards of 15 minutes, so TensorRT may generally not be as useful for processing short videos.
Aside from that, you just want to make sure that "Enable Interpolation" setting is UN-checked, and then just press the green "Upscale" button to process your video.
How to Set up and use chaiNNer
Download and install the latest version of chaiNNer here https://chainner.app or from the Github releases page.
There is good documentation for getting started on chaiNNer’s Github page, so I recommend spending a few minutes reading over it.
After installing and launching the program for the first time, you will need to let it download the dependencies. This may take a while as it needs to download several gigabytes of data. If it doesn’t happen automatically, you can access the dependency manager in the upper right corner of the application window.
chaiNNer is a node-based image processing toolkit (and it can process videos as well). You basically create a flowchart by selecting building blocks that represent different operations, and then connecting them together.
I will not go into detail on how to upscale in chaiNNer, but you can find some tutorials on Youtube, such as these:
Note that chaiNNer is under heavy development, so older videos may look different than what you see in the application.
chaiNNer can work with both .pth and .onnx model files, just note that they need to be handled with different nodes! .pth (PyTorch) models need to use the orange Load Model and Upscale Image nodes, while .onnx models use the blue Load Model and Upscale Image nodes.
Converting Models in chaiNNer
To convert a model from .pth format to .onnx format, you simply need to use three nodes:
Orange (PyTorch) Load Model node - select a .pth model to load here.
Orange (PyTorch) Convert To ONNX node - The data type should usually be set to fp32, and opset should be set to 17.
Blue (ONNX) Save Model node - Select an output directory and filename to save the model as.
Press the Green button at the top of the window to convert the model.