Supercharging Storyboarding with Generative AI

The worst words you can hear in a video production are unusable and reshoot. –Production manager for nationally-acclaimed TV show.

When it comes to making feature films, television shows, and advertisement videos, the cost of shooting on set gives many producers nightmares.

Production teams often have numerous people responsible for different parts of a project. However, at any given point during filming, only a few individuals are completely engaged.

Logistic coordination of different parties is difficult, especially in the service of a director's evolving creative vision. Mistakes in getting a shot are expensive and incur delays, wasted man-hours, and pricey reshoots.

To mitigate these production risks, video teams do their best to plan out time, people, and resource constraints in the preproduction stage of a project. One of the most important components in this stage is the storyboard.

In this post, we will discuss what the storyboarding process looks like today and how it stands to be completely revolutionized by generative AI technology.

What is a Storyboard

A storyboard is the centerpiece of the preproduction phase of a video project, including feature films, TV commercials, music videos, etc. It is a previsualization document used to depict what each shot in the script will look like.

The frames in a storyboard are typically low fidelity sketches mapped to sections of a script that are intended as a rough guide of what the setting for a scene will be, how characters will move during a shot, and where cameras need to be placed to capture a shot.

Storyboards are collaborative documents that coordinate the work of all parties in a project: the screenwriter, director, producer, and members of the technical crew such as a cinematographer.

A well-constructed storyboard is an organizational blueprint that can make shooting a video during the production stage seamless. Therefore, getting storyboards right is crucial to making production efforts as effective as possible.

Storyboarding Today is Cumbersome

Though storyboards are an essential step to a successful video project, the existing status quo for making them is incredibly inefficient.

There are two primary ways that storyboards are created today: 1) aggregating shots from other movies, shows, and projects or 2) hiring a storyboard artist.

The first technique typically involves taking a scene from your script and finding relevant shots from an existing shot collection such as film-grab.com, shotdeck.com, or Google Images.

The problems with this technique are:

1. The exact right shot you need might not exist in another movie or TV show.
2. Even if an appropriate shot exists, it takes a long time to find it.
3. Once you put together all these borrowed shots, the storyboard ends up being a hodge-podge of images in drastically different styles, with inconsistent characters. This makes it hard to conceptualize and de-risk a project if that's the mental model you have coming out of previsualization.

The second technique used is hiring a professional storyboard artist to illustrate your script frame-by-frame. What you end up getting is often a rough, pencil-sketched illustration of your script.

There are several issues with this approach:

1. It takes very long to get a storyboard illustrated, anywhere from weeks to months in the case of a full-length feature film. This unnecessarily prolongs the time to actual production of a project.
2. Hiring a storyboard artist is expensive, costing upwards of tens of thousands of dollars to make the full board. This makes getting an artist out of the reach of many independent filmmakers. Additionally in the early stages of a project before funding is secured, storyboarding is one of the largest upfront costs that may never be repaid.
3. Because storyboards are often rough sketches, there is a huge aesthetic mismatch between the board and the final produced shot. This makes it difficult to accurately explore a project vision during the brainstorming stages, hindering the ability to play out different production choices.

It's clear that the existing storyboarding process is due for an overhaul.

Generative AI Meets Storyboarding

Today generative AI has taken the technology world by storm with the introduction of incredible systems such as ChatGPT, DALL·E 2, and Stable Diffusion.

While some of the attention is clearly just a hype wave, generative AI models end up being a great solution to the storyboarding use-case for a number of reasons:

1. Generative techniques enable infinite experimentation at the cost of a model call. If you can get shots in seconds instead of days, video creatives are able to explore their visions more freely.
2. These models are visually expressive and rich in ways we've never seen before from artificial systems, enabling artists to follow their creative whimsy in new ways.
3. Because these models can learn from data, they continuously improve and adapt as they are used more. This allows for learned personalization of an artists' style and brand, something that existing storyboarding solutions cannot do.
4. Storyboarding is a fault-tolerant problem for generative models to be applied to. In other words, while it's clear that generative systems aren't perfect today, the storyboarding use-case tolerates occasional hiccups such as distorted faces or additional limbs. This means generative AI can provide benefits to the storyboarding workflow today.

The beauty of the generative AI-powered storyboarding solution is that it enables completely new possibilities for how storyboards feed into the creative process.

As an example, because generative systems are low-latency, they can now become powerful creative assistants during the screenwriting process allowing an artist to interactively experiment with their visual aesthetic even before a script is done. This is something that is impossible with the storyboarding status quo.

Challenges in Generative AI Storyboarding

While generative AI seems to be a great solution, there are still a number of technical challenges that must be addressed to make it work completely for storyboarding.

As others have noted, you can't expect the existing generative vision models such as Midjourney to work out-of-the-box.

While generative systems can produce visually expressive depictions, they tend to lack character and environment consistency across frames. Generative techniques aren't nearly as bad as using separate shots from film-grab.com, but for a sequential episodic medium like storyboarding there is still room for consistency improvements.

Another issue is that prompt engineering for rich textual information like scripts is still very challenging. A good prompt will make or break the quality of a generated image, but it is difficult to automatically parse and generate appropriate prompts from a dense script scene description. This requires advanced natural language processing and understanding techniques.

Finally, because generative AI applied to storyboarding is still such a new task, the existing evaluation metrics for assessing the quality of a generative system are ill-defined. Metrics such as FID or CLIP scores have really only been applied to single-image captioning systems, so an accurate end-to-end evaluation for multiple related, generated images still does not exist.

While there are still numerous problems to solve in generative AI storyboarding, it's encouraging to see active research interest in addressing them.

Concluding Thoughts

Though storyboarding is a crucial component of a successful video project, the existing techniques for creating storyboards suffer from numerous issues including time, cost, and aesthetic inconsistency.

Generative AI promises to drastically augment and improve storyboarding workflows in ways never before possible, enabling incredible new possibilities for creative expression.

Expect some phenomenal technical and artistic leaps in the future.

At Storia, we're building the future of AI-driven video solutions with some exciting new products coming out soon. If you get excited about AI and video, we want to hear from you!