AnimateDiff adds motion to image generation models. Rather than being a separate video model, it is a motion module that plugs into an existing Stable Diffusion setup, so the characters and the style you already get from your favourite image model can be made to move without retraining anything. That design is why it spread so quickly through the image generation community: people could keep their existing models and checkpoints and add animation on top. The results are short clips rather than long video, and consistency across frames is the thing you fight with. It is open source, it expects you to be comfortable with a local image generation stack, and it rewards patience with settings.
This repository contains the codes of "A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild", published at ACM Multimedia 2020. For HD commercial model, please try out Sync Labs
