Convert language models into video generators

Open

December 23, 2023

VideoPoet by Google

Videos

6,328

4.5(2)

Use tool Copy 🔗

126

Inputs:

Outputs:

Convert language models into video generators

Overview

Overview Releases Pricing Reviews Prompts Pros & Cons Q&A Alternatives Recommendations

Videos786 0

video generation language models Google Research MAGVIT V2 video tokenizer SoundStream audio tokenizer video synthesis

Most popular alternative: D-ID (1,465 saves)

View 7 alternatives Recommendations

By unverified author Claim this AI

Generated by ChatGPT

VideoPoet, by Google Research, represents a significant evolution in video generation, particularly in producing large, interesting, and high-fidelity motions.

This tool is used to convert autoregressive language models into a high-quality video generator. It includes components such as MAGVIT V2 video tokenizer and SoundStream audio tokenizer that transform images, video, and audio clips with variable lengths into a sequence of discrete codes in a unified vocabulary.

These codes are allied with text-based language models, allowing integration with other modalities such as text. An autoregressive language model, contends within this tool, learns across video, image, audio, and text modalities to autoregressively predict the next video or audio token in the sequence.

It further combines multimodal generative learning objectives into the training framework, such as text-to-video, text-to-image, image-to-video, video frame continuation, video inpainting and outpainting, video stylization, and video-to-audio.

VideoPoet can generate videos in square orientation or portrait to cater for short-form content. It also supports generating audio from a video input.

With capability of multitasking on a variety of video-centric inputs and outputs, VideoPoet illustrates how language models can synthesize and edit videos with desirable temporal consistency.

Releases

Get notified when a new version of VideoPoet by Google is released

Notify me

Initial release

December 22, 2023

Initial release of VideoPoet by Google.

+ Submit new release

Pricing

Pricing model

No Pricing

Use tool

Save

🔗 Copy link

🗳️ Vote Best AI Tool

Featured

Videos VideoPoet by Google

Videos

6,328

4.5(2)

Overview Releases Pricing Reviews Prompts Pros & Cons Q&A Alternatives Recommendations

Use tool

Save

Related tasks

Poetic image generation1 0

Video concept generation1 0

Reviews

4.5

Average from 2 ratings.

★ ★ ★ ★ ★ 1

★ ★ ★ ★ 1

★ ★ ★ 0

★ ★ 0

★ 0

How would you rate VideoPoet by Google?

Help other people by letting them know if this AI was useful.

Your rating

★ ★ ★ ★ ★

Post

Prompts & Results

Title:

Description:

Prompt type*:

Prompt*:

Output type*:

Output*:

Add your own prompts and outputs to help others understand how to use this AI.

VideoPoet by Google was manually vetted by our editorial team and was first featured on December 23rd 2023.

Promote AI Claim AI New release

CodeRabbit

Enhanced code review for improved workflow and quality.

Code reviews

Open

883,407
646
3.9

v1.6 released 27d ago
Free + from $12/mo
Share

775,212 coderabbit.ai

Sahil Mohan Bansal

🙏 235 karma

Nov 13, 2024

Reducing manual efforts in first-pass during code-review process helps speed up the "final check" before merging PRs

16021 Reply Share Edit Delete Report
Averi AI

The AI Marketing Workspace: Strategy, Content, Team in One

Marketing

Open

13,555
114
4.3

v2.1 released 2mo ago
Free + from $45/mo
Share

Zach Chmael

🛠️ 1 tool 🙏 59 karma

Mar 10, 2025

This is the first AI marketing tool I’ve used that actually helps me get things done, not just suggest generic ideas. The mix of AI-powered strategy and real experts executing the work makes it feel like having a high-performance marketing team without the overhead

509 Reply Share Edit Delete Report

Pros and Cons

Pros

High-fidelity motions

MAGVIT V2 video tokenizer

SoundStream audio tokenizer

Transforms variable length clips

Sequence of discrete codes

Integration with text modalities

Predicts next video/audio token

Combines multimodal generative learning

Generates square and portrait videos

Supports audio generation

Desirable temporal consistency

Text-to-Video capability

Image-to-Video capability

Video Inpainting

Video Outpainting

Video Stylization

Video-to-Audio capability

High-quality video generator

Multitasking on video-centric inputs/outputs

Maintains object identity preservation

Long video generation capabilities

Interactive video editing capabilities

Controllable camera motions

Zero-shot video generation

Controllable video motions

Audio matching for input video

Zero-shot controllable camera motions

Allows for stylization

Applies visual styles and effects

Capable of text-to-audio

View 25 more pros

Cons

Limited orientation

Unpredictable output

No real-time editing

Complex setup

Dependent on Google resources

Limited to Google's vocab

Requires large data

No user guides

Limited generations

No multilingual support

View 5 more cons

7 alternatives to VideoPoet by Google for Videos

freebeat AI

Turn Music & Ideas into Viral Videos In One Click

385,623
486
4.2

9.8.0 released 10d ago
Free + from $4.99/mo
Share
348,883 freebeat.ai

kanawati

🙏 1,072 karma

Mar 26, 2025

The concept is great.

555 Reply Share Edit Delete Report
D-ID

Create AI-generated videos with ease

177,307
1,465
4.5

Released 4y ago
Free + from $4.7/mo
Share

Franco Arteseros

🙏 68 karma

Oct 23, 2024

WE USE D-ID AT THE COLORADO VIRTUAL CREATIVE FACTORY...AND LOVE IT.

508 Reply Share Edit Delete Report
MagicLight

Transform text into captivating videos instantly.

129,310
105
4.2

Released 5mo ago
#11 in Trending
Share

Daniel Garaiacu

🛠️ 3 tools 🙏 1,005 karma

May 29, 2025

You get 300 credits upon signing up, which is enough to test out the app and see its potential. I had a bit of fun with it. It takes a few minutes to generate content, but the results are impressive. There are many styles, modifiers, and customization options available. I would definitely use this for content creation or storytelling.

12415 Reply Share Edit Delete Report
Kaiber

AI Video Generation

123,788
1,045
3.4

Released 2y ago
Free + from $5/mo
Share

Noeffen Way

🙏 17 karma

Jun 18, 2023

They're dreaming if they think I'd give them my credit card info just for a free trial. Most useless thing ever...

125 Reply Share Edit Delete Report
HeyGen

Create AI spokesperson videos from text

101,317
666
3.3

v3.2 released 6mo ago
Free + from $24/mo
Share
62,054 www.heygen.com

Pluto Radigund

🙏 100 karma

Jan 2, 2024

easy onboarding, quick to set up.

759 Reply Share Edit Delete Report
ByteCap

#1 AI Video Shorts Maker for Businesses & Creators

90,339
305
4.1

v2.0 released 1mo ago
#43 in Trending
Share
68,101 www.bytecap.io

bob miller

🙏 56 karma

May 22, 2024

Generating captions was super fast like seconds. Wish they had Anton font though

519 Reply Share Edit Delete Report
Manga TV

Generated comic videos.

75,735
445
2.3

Released 2y ago
Free + from $9.99
Share

Marcus Howard

🙏 69 karma

Sep 27, 2023

its garbage and not free

4611 Reply Share Edit Delete Report

View 174 more alternatives

Q&A

What is VideoPoet?

Andrei

🛠️ 40 tools 🙏 3,686 karma

wrote:

VideoPoet is a tool developed by Google Research, designed to represent a significant evolution in video generation. It essentially transforms autoregressive language models into a high-quality video generator. VideoPoet is proficient in producing large, interesting, and high-fidelity motions.

How does VideoPoet generate videos using language models?

VideoPoet generates videos by integrating and converting autoregressive language models into the video generation process. It uses components such as the MAGVIT V2 video tokenizer and SoundStream audio tokenizer to transform images, video, and audio clips into a sequence of discrete codes in a unified vocabulary. These codes are combined with text-based language models for integration with other modalities like text. In the process, this technology learns across modalities to predict the next video or audio token in a sequence.

What is the role of MAGVIT V2 video tokenizer in VideoPoet?

MAGVIT V2 video tokenizer plays a key role in VideoPoet by transforming images and video clips into a sequence of discrete codes in a unified vocabulary. The codes transformed by MAGVIT V2 are compatible with text-based language models, thereby facilitating modal integration. Essentially, it forms the video language that the autoregressive model learns and synthesizes.

How does SoundStream audio tokenizer contribute to VideoPoet functionality?

The SoundStream audio tokenizer in VideoPoet is responsible for transforming audio clips into discrete codes, similar to how the MAGVIT V2 video tokenizer works with video. These codes are used along with the codes from images and videos to be processed by the autoregressive language model. Moreover, it supports the generation of audio from a video input, marking a leap in multimodal learning.

Can VideoPoet generate both video and audio?

Yes, VideoPoet has the capability to generate both video and audio. The integrated process allows for the generation of audio from a video input, thus enabling a syncing of both audio and visual aspects of a clip.

What formats or orientations are supported by VideoPoet?

VideoPoet can generate videos in both square orientation and portrait. These formats particularly cater to the demands of short-form content, offering flexible options to cater to specific requirements.

+ Show 14 more

Can you edit videos with VideoPoet?

Yes, videos can be edited using VideoPoet. The integrated language model allows for the synthesis and editing of videos with a high degree of temporal consistency. It further provides an array of features like video inpainting and outpainting, and video stylization.

How does VideoPoet ensure temporal consistency in videos?

VideoPoet employs the use of an autoregressive language model that learns across different modalities such as video, image, audio, and text to ensure temporal consistency in videos. This allows for autoregressive prediction of the next video or audio token in a sequence, thus maintaining continuity and consistency throughout the video.

What are some examples of multimodal learning objectives in VideoPoet's training framework?

In the training framework of VideoPoet, there are multiple multimodal generative learning objectives incorporated that include tasks like text-to-video, text-to-image, image-to-video, video frame continuation, video inpainting and outpainting, video stylization, and even video-to-audio.

How is the autoregressive functionality used in VideoPoet?

The autoregressive functionality within VideoPoet is used to learn across various modalities such as video, image, audio, and text to autoregressively predict the next video or audio token in the sequence. This learning approach is instrumental in synthesizing and editing videos with a high degree of temporal consistency.

What is the process of converting text-to-video in VideoPoet?

The text-to-video conversion in VideoPoet is facilitated by an integration of the provided text with the learnt sequence of discrete codes from the images and audio clips. By predicting the next token in the sequence based on the text input, VideoPoet creates a video that aligns with the narrative framework provided by the text description.

Can you use VideoPoet to convert image-to-video?

Yes, VideoPoet does have the capability to convert image-to-video. It uses discrete codes from an input image to convert into a sequential video. The autoregressive language model then pieces these codes into a consistent video sequence.

Can you change the style of a video using VideoPoet?

Yes, VideoPoet can be used to change the style of a video. It combines multimodal learning objectives in the training framework, including stylization. This results in the alteration of existing video styles or application of new styles to create aesthetically desirable videos.

What is the aspect ratio of videos generated through VideoPoet?

VideoPoet supports generating videos in orientations that cater to short-form content. This includes both square and portrait orientations offering diversity in aspect ratios for end-users.

What is the video generation process in VideoPoet?

The video generation process in VideoPoet functions via learning from autoregressive language models. It employs the MAGVIT V2 video tokenizer and the SoundStream audio tokenizer to transform inputs from images, videos, and audio clips into sequence of discrete codes. Then, it merges these codes with text-based language models to predict the next video or audio token in a sequence that eventually forms a high-quality video.

What is the operational sequence of VideoPoet?

The operational sequence of VideoPoet involves the conversion of inputs (images, video, audio clips and text) into discrete codes, integration with text-based language models, and autoregressive prediction. Once the sequence of codes is generated and integrated, the autoregressive language model is set to work to predict the next video or audio token, thus synthesizing a complete video.

How does VideoPoet handle multiple video-centric inputs and outputs?

VideoPoet is designed to multitask on multiple video-centric inputs and outputs. This feature is achieved through the tool’s unique ability to learn across different modalities, including text, audio, video, and image, thus enabling the generation of multiple outputs selectively or simultaneously.

What kind of languages does VideoPoet support?

The language models that VideoPoet utilizes to generate videos are autoregressive. While specific languages are not defined on their website, this tool is typically designed to understand and process human languages, which suggests a potential capacity for multi-language support.

What is the output format of VideoPoet generated videos and audios?

The output format of videos and audios generated by VideoPoet isn't explicitly mentioned on their website. However, given that it is a Google Research product specializing in creating high-quality video content, it can be inferred that it supports popular video and audio formats commonly used in digital media.

How does text-to-audio feature work in VideoPoet?

Andrei

🛠️ 40 tools 🙏 3,686 karma

wrote:

VideoPoet's text-to-audio feature works by predicting the next audio token in the sequence based on the text input. It processes the text input through the autoregressive model, leveraging the learnt sequence of discrete codes from audio clips to form a consistent audio narrative that aligns with the text.

Ask a question

Submit

If you liked VideoPoet by Google

Verified tools

VMEG Clips to Videos

Transform clips into stunning marketing videos effortlessly.

Marketing videos

Open

17,172
115
3.1

Released 1y ago
Free + from $1/mo
Share
1,814 www.vmeg.pro

Prentice

🛠️ 2 tools 🙏 59 karma

Aug 5, 2024

Thank you for your feedback, and I apologize for any inconvenience you experienced. We are currently working to ease the resolution constraints for user uploads. Your insights are invaluable to us.

526 Reply Share Edit Delete Report
VideoTube

Generate AI videos from text, image, and video in minutes.

Videos

Open

12,010
64
3.5

Released 6mo ago
From $9.9/mo
Share
2,601 videotube.ai

sum sum

🙏 32 karma

Apr 21, 2025

One platform gives me access to various high-quality video models — it’s been used to generate 110k viral social media posts.

224 Reply Share Edit Delete Report