The EMC tech behind 'Avatar'

"We will solve our last problem on the day we finish shooting," Avatar's director James Cameron told Jon Landau, Avatar's producer, as they embarked on the journey to tell the story of the digitally produced blue-coloured Na'vi tribe which faces extinction on Pandora.

Landau knew they had a challenge on their hands, as they had plans to shoot using technology that had not existed yet.

The movie is not animated. Actors act and not simply lend their voices to the script, which is what makes this movie so incredibly rich.

At EMC World in Las Vegas, Landau explained that choosing the right technology partner, such as EMC, was the key to allow the movie to come to life. Even though the team had no idea how they would overcome software hurdles to shoot the digital motion capture, EMC offered the backbone that could deliver a robust system - it could be scaled up to accommodate new processes whenever they would be required and in whatever shape they would happen to be.

"The entire movie creation is based on end-to-end software. Once the script is themed for the story to be told, the design is digital, the production is virtual and the distribution is digital, which underscored the need for the Avatar team to choose tech partners correctly. Spontaneity can only happen once, but if we capture it digitally, then we can create it in a close-up or in a wide shot, where the actor could never recreate it," said Landau. He quickly added: "But remember that technology doesn't make a bad movie good."

Everyone can be a director as anyone can create content, but where does one store such large data and for how long remain the big data nightmares. The first Avatar used 180 terabytes of storage. The new Avatar has up to 2 petabytes, and the digital effects company will have 10 petabytes.

"How do we manage that?" Landau asked, as distribution channels such as Netflix and mobile cannot handle 2 petabytes of data to distribute Avatar. Landau explained: "The studio looked at the codex, and some were not approved as they could not distribute the content of good enough quality. So, if Netflix is only presenting its content on the TV screen then it only has to be good enough for that screen, the same as if someone is doing it for a mobile device - so the size is manageable."

Historically, there are two types of cloud systems. The private cloud, where everything is controlled and owned by the business; or the public cloud, where a service provider provides the hardware and software, and users simply use their services to share data, e-mail, store information, etc. The concept EMC introduced is known as a hybrid cloud, where some of the cloud is owned by the business and some is owned by a service provider.

In the studio, there is no mistaking which cloud services they use - private cloud. "For now and the foreseeable future, we need to protect our IP. The Avatar team needs collaborate with its New Zealand-based company, Weta Digital. In order to do that, they use EMC's Vplex to set up a private cloud to share data.

"We need to keep the wizard behind the curtain. If people see the first pass of Avatar and criticise that, we can't adapt. Once it is out, we can go to the public cloud." Landau cites two cases for when the studio can use the hybrid cloud, such as the 'Pendopedia',which is an encyclopaedia/Wikipedia about the world of Pandora. This can be opened up to let people contribute to it. "Or, the perfect opportunity for a public cloud could be in a game play, where someone can influence the game for someone else."

It's too risky to have Avatar's IP accessible on someone else's system. In fact, Landau warns: "In terms of security of content, we do it internally. We create networks within networks to protect access to content. We no longer have to worry about big brother; we have to worry about little brother."

Tech has a price, and just like every other company, even the Avatar team has to fight for the studio to spend money. "We don't have an open cheque book - rather, the opposite. The first Avatar movie did not get the green light as the studio was worried about the financial risk. It's hard to justify to someone in the studio IT spend. They can understand if you need more money for a star or visual effects or a bigger set, but when you say you want three different EMC Isilon servers, they don't understand why you need any of that."

Avatar has gone on to be the highest-grossing film of all time, surpassing 'Titanic', using revolutionary techniques. However, the goal at the outset was not to disrupt the industry. "Our result in success is a disruption. Look historically. The businesses that succeed in creating disruptions are successful."

Landau summarises: "Regardless of industry, people never get fired for not trying something new. People get fired for trying something new and failing, which therefore means that people are hesitant and afraid to innovate."

In the past, the rules were 'plan, build, run' which was a cycle that took years; it was far too slow. With virtualisation and the ability to deploy hundreds of servers at the touch of a button, trying something new and failing is no longer a disaster, and so the envelope is being pushed. "There should always be a market gap - that is what spurs us to fill it."