Digital Frontier

Header

Main

CG MAKING

Back to the list

GANTZ:O

October 2016 [CG]

The Use of Innovative Head Mounted Performance Capture Systems

Set in a real city with a progression of one fantastical struggle-to-the-death battle scene after another, this is the thrilling fascination of "GANTZ: O". The city of Osaka does really exist which makes it even more surreal. Here is the interview with Director Mr. Yasushi Kawamura, Mr. Ryota Minamoto (Sets Art), Mr. Naoyuki Fujii (Character), Mr. Kensuke Morita (Rigging & Simulation), Mr. Koki Koshita (Motion Capture), Mr. Takeshi Kamekawa (Layout & Animation) and Mr. Kensaku Ishiyama (Facial). They are the ones who mainly worked on the first half of the film. We discussed about the backdrop and the characters that lived in this realistic world.

Reflections on location hunting in Osaka

MINA-
MOTO

We went to Osaka for location hunting in April 2014 since this was based on the Osaka arc from the original book. First of all, the setting was described very realistically in the original work, so I felt like it could have happened here, even the unrealistic events. Because of this, when designing the CG images, I realized we had to create life like things similar to real things.

KAWA-
MURA

It may sound easy if we had just used the existing place, but it had to be even more realistic since we were familiar with it in the real world already. From an honest point of view, it would have been much easier to accept this if it was a fantasy film with impossible architecture and design.

MINA-
MOTO

The screenplay was already finished when we went location hunting and some places were already selected. So we went there and decided how we would have the characters move with motion lines, how we would display the background, and so on.

KAWA-
MURA

I searched for the backgrounds in the original work by Google Map and found them. Our team members all knew the original work well so they knew where we would go without my detailed explanation.

MINA-
MOTO

I had my hands full with taking pictures. There were so many people and they looked at us like “What’s going on?”

KAWA-
MURA

We took pictures of things that ordinary tourist would never shoot, advertising displays, surfaces on the ground. I realized that I had started to take pictures next to a LOVE HOTEL, then “Oh, I don’t need these picture” (laugh). We shot 360 degree using special cameras like for Google Street. I released the shutter every five seconds with the special camera, causing a lady on the street to keep asking me “What are you doing?” numerous times.

  • Image of 360 degree camera

-Was it different than Tokyo?

KAWA-
MURA

Most of the lights were fluorescent in Shinjyuku so the lighting was greenish. There were a lot of restaurants around Dotonbori in the “Kuidaore” area, so it was orange or some warm color. I found the differences that way.

MINA-
MOTO

I saw many unique advertising displays like this dragon.

  • Shinjuku
  • Osaka
  • Osaka

KAWA-
MURA

The original work was published about five years ago, so there were no Starbucks there and the display sign of “Dotonbori” was then a regular electric board not a digital signage like today. We updated those things in the film.

MINA-
MOTO

I took huge amount of pictures, requiring spare batteries. I took some backdrops that were not in the original work and shot during both day and night. During the day I tried to capture the atmosphere of Osaka city and also took some pictures for the semblance that would be pasted into the background. During the night I tried to capture its real atmosphere since the settings in the film were during the night. I also took some measurements called HDR which would be used for lighting for the characters. For shooting the semblance, we had to make sure that there were no shadows or shades by the lights in the pictures for editing purpose. Overcast weather was the best, but it was fair weather at that time.

KAWA-
MURA

We needed large amounts of information to create a real peopled feel of the real world. It was a challenge to see how far we could take it.

MINA-
MOTO

Yeah, I added garbage and a puddle of blood on the ground to increase the data of the image. I wanted to maintain the quality. Although some portions of the background situation were already deleted during the making of the screenplay, just one background still had a lot of information. So I had to think about how we could complete it efficiently with zero waste. Making one shot by one shot cost a lot, so we gathered common building units which we later could converted into a single resource, then put them together all at once to make a whole city, and then we placed the camera where the Animation team roughly indicated in the data, and then touched up the visible areas. I always kept things like this in mind so we could save time.

-Which background of the scene was the most difficult to make?

MINA-
MOTO

I would say it was Ebisu Bridge. It was quite open when I saw it from 360 degree because so many buildings were still lined up even till when they were very far off so that it would become a large amount of data and a very heavy scene. If that happened, the team members involved in the next step would be in trouble because compiling this much data would be a challenge. And I also needed to make sure that there were no strange advertisement displays popping up in the scenery during the emotional scenes.

KAWA-
MURA

No “Takoyaki” (laugh)

MINA-
MOTO

We replaced the advertisements displayed on purpose. Sometimes we asked the Director to design some first or we would design them ourselves.

KAWA-
MURA

Even the timing for the neon color changing from blue to red was under my direction. We could go that far because it was a CG film which was fun.

The balance of the Character’s uniqueness with reality

FUJII

I actually joined the Character team as the lead in the middle of the project, not from the start. The screenplay was not revised yet at the time, so we had also created other characters which wouldn’t appear in the film. During development, we created characters that were realistic. The hardest part was finding the point of juxtaposition between the character’s individuality and the semblance to reality. Mr. Kawamura retouched the characters and sent them to us which helped us a lot and made our work easier.

  • Early stage model which had been inspected (1) with direction notes
  • Early stage model which had been inspected (2)
  • Retouched
  • Final image

KAWA-
MURA

Although it was a more realistic style at first, I found that it would be better to add a little bit of the anime style to it so that it would be closer to its original look, also if we had made them similar to real Japanese people like making their eyes smaller, then their countenance would be even tinier. It might be better if they were realistic like at a live action film level, though if we did go for that, it might end up failing both styles. Also I heard that many team members had the opinion that it would be better to emphasize the character’s uniqueness like the original material. If we designed it like in the original manga, it would not work with the facial capture system. So I had it retouched it later in order to balance it fairly with a CG’s facial expression. CG character, especially realistic characters, has an issue called “Uncanny Valley”. I still felt anxious about how much the general audience could accept.

-If there were many wrinkles on a face, it would become more difficult to make?

FUJII

We used the software ZBrush which allowed us to prepare the character’s mould for modeling like using clay. We could add details through intuition but this was not easy, we needed to stick with a basic face and add on elements and so on. It was much easier to add details like wrinkles that would disguise the basic face whereas a female face which is a smooth face was actually more difficult.
The hardest part was that there were many different scenarios for each character, more so than any previous jobs. This demonstrated our obsession. For instance, we made different faces for each body of “Nurarihyon” and his subsequent alter egos beginning with the first character of the old man. There was immediate oversight on details like how to stain the blood and so on that helped us and the work to go smoothly, however the quantity of work was beyond comparison than any previous jobs. Even hair styles had scenarios such as being wet and so on. I really cared about the face, therefore I kept the cost low for the body by using a new tool that makes a basic body with adjustable parameters which allowed us to make manual adjustments for each character’s uniqueness.

KAWA-
MURA

CG never gets dirty, it stays smooth forever, but this is a survival action story so it cannot stay clean from the beginning till the end.

FUJII

Kato had nine dirty scenarios and I had to ask for this number to decrease (laugh).

KAWA-
MURA

For example, there was a scene where he jumps into Dotonbori, he gets dirty nicely but then he gets wet so his blood stains become slightly gone.

FUJII

Basically I arranged to do this by switching the semblance but I needed to adjust the hair for things like styling, being wet, and so on.

-Which character did you put in your strongest efforts?

FUJII

I worked on the Nyotai alien for the longest time. The checking environment and the shooting environment were different so I had to keep adjusting the semblance until the last minute. Also I had to combine it with other elements such as search lights so the rendering was heavy. If we moved too much the monsters didn’t look gigantic, so it was difficult. Also we placed dead bodies in the background at some pivotal points, which was the job for our character team.

KAWA-
MURA

“Aren’t dead bodies background team’s job?”. What about “weapons on the ground?”. We were sometimes confused about which team should be in charge. How about Robots?

FUJII

Robots were for the background team.
Gantz bike as well.

KAWA-
MURA

If it is hard stuff, it will be for the background team.

MINA-
MOTO

In our production, the background team is better at modeling hard surface.

-Did you decide which specialists were working?

MINA-
MOTO

These days, most of the staff would indicate “I would like to work in this department” when they join the company. In past days, they use to be hired as a generalist to get many experiences, then they would find out which field they would like to work in.

KAWA-
MURA

These days most people have already decided what to do when they were a student. It might not fit even though they like to do it and it can be the wrong decision, I suspect.

Exploring the elements in rigging & simulation

MORITA

Putting bones into the characters to make them move is called Rigging. Main characters wore GANTZ suits most of the time so they were close to being naked, so I set up their profiles carefully, especially when they raised their arms and pulled their shoulder down. There were many characters though I made the main character Kato first. I kept working on the development of how to plant Kato’s rig to form the basis for the other characters. There was a way that I could set up each character to emphasize their characteristics, but it took a lot of man-hours, so I kept thinking about the best work flow that could save man-hours and produce good quality as I was working. Aside from this, I really focused on clothes and hair simulation this time.

-What is clothes and hair simulation?

MORITA

The clothes and hair means outfit and hair. If we didn’t do simulation then it would be in a condition called Bind where the clothes and hair would become controlled by the joints. There would be no deformation like from swinging or with wrinkles appearing on the outfits and so on. This would be a huge cost if we were to set up simulations for each body, so we made a system where we created a basic model for a variety of outfit scenarios from which to be based upon.
Mob characters and supporting characters are usually chosen for non-application of clothes and hair simulations to reduce man-hours. However, for this film we determined that the Mob should not be like the usual Mob meaning we were to raise the Mob’s quality. That is to say that we didn’t ease up for the Mob, we burned both ends of the candle for the mob just as much as for the main characters.

KAWA-
MURA

People often say CG is “stiff”, but if we spend the time we can make it soft. It can be soft and look natural if we make the clothes and hair undulate.

-Does that means it took longer for female character?

MORITA

That’s right. Reika has long hair so it was difficult to control. We outsourced it this time but they were not familiar with DF’s specification, so I created the curve for the initial simulation so that anyone could easily control for it.

KAWA-
MURA

Clothes and hair require rigging to move just like the body. If a body part has no rigging, it would look like a doll. A female character’s boobs are a good example.

MORITA

In one scene, a Tengu was bounded by a Y Gun with the Y Gun’s strings, we used simulation to show the clothes being tied and so on.

KAWA-
MURA

As it became more and more realistic, the amount of things to portray increased. Fortunately GANTZ Suits are tightly fit to the body. If it was a skirt or a coat, we were over (laugh).

MORITA

Yamazaki's hair drifts so it was more difficult than Reika’s long hair. I made a simulation once and asked Mr. Kawamura to check it, he said “This is not like Yamazaki’s”. After that hair simulation, her hair style had changed since and went straight down. If I had tried to retain the volume of her hair, the hair would have looked hard.

  • Yamazaki
  • Yamazaki's hair simulation

KAWA-
MURA

Making it anatomically realistic isn’t good enough physically.

MORITA

In regard to one of the Nyotai alien scene, there were so many boobs that were swinging, though not all of them were, it was quite tough to control. When they were swinging, they looked like water balloons so I took some time to portray the realistic anatomical weight. If I hadn’t done the simulations, nothing would have moved. Mr. Kawamura was next to me and said, “They should be swinging”

KAWA-
MURA

How could you say such thing calmly (laugh).

MORITA

Well, he was right, they should be swinging.

Using Performance Capture in a fully integrated network

KOSHITA

I joined the Motion Capture team in 2013 when we shot the pilot episode and were requested to shoot the facial expressions through Head Mounted Cameras. We were used to using the method of placing multiple three millimeters size marks on a face and then filming it at a special studio for face shooting. This time, we used the method of Performance Capture which is to film facial expressions and body movement at the same time through a Head Mounted Camera.

  • Shooting with Motion Capture at the studio

KAWA-
MURA

This technique of filming movements and facial expression at the same time gained attention from the film “Avatar”. This technique has further improved by now.

KOSHITA

When we shot with Head Mounted Cameras, we could also record eyes movement data. Acting including the eye movements is very important for the expression of feelings, so it was much more realistic than the method of using markers for filming facial expressions and then adding eye movements through keyframe animation.

KAWA-
MURA

Thank you to improvements in pixel counts in digital cameras and computer’s face authentication program. Our work improved from the technological progress on the set.

KOSHITA

We only had three Head Mounted Cameras when this film was green lighted. We wished we could use them for all eight main characters but it was impossible. So we decided to use five cameras and had to choose between either buying commercially available cameras or to make them ourselves. I had hoped that we could buy them so I made a plan with three different levels called pine, bamboo, and plum and sent them to my boss. But the pine which was the best plan cost as much as a car (laugh). Eventually we decided to make them ourselves for many reasons.

KAWA-
MURA

Koshita’s team was highly skilled in technological engineering so they could make anything.

KOSHITA

We purchased headgears from commercially wearable technology and set up the network ourselves. We rewrote the firmware that controlled the cameras by removing and re-installing a duplicate chip-on-board in order to allow us to control multiple units at the same time from a PC. There was a risk that the cameras would break if this failed (laugh). We also created the tool which allowed the Head Mounted Cameras to automatically synchronize starting and stopping with the motion-capture system. The tool itself had been ready to use much earlier, but the user interface which operated the tool was only completed two days before we started shooting. After that, we polished the tool during shooting from the operator’s feedback.

KAWA-
MURA

We had to search foreign websites and translate them and then make the changes with materials that could be found in Japan.

KOSHITA

That’s right. Using Performance Capture system that synchronized five cameras in a fully integrated network that was handmade could be a first time in Japan.

  • Head Mounted Camera

-How long did it take for you to shoot?

KOSHITA

With the regular Motion Capture, we could take fifteen minutes of scenes per day and with the Performance Capture, we took about six minutes of scenes per day. So it took three weeks. This method was tougher so we had to do many rehearsals with the equipment before the actual shooting so that we could prevent delay caused by complex multiple devices that had to be simultaneously operated.

KAWA-
MURA

It took more time since I had to inspect it twice at each shooting; wide images with entire body movements and facial expressions, and both had to be OK.

-During shooting, the actors’ face were under the spotlight all the time.

KOSHITA

They had patience though later that day they would say “it was tough.” Mobs were shot through the traditional marker method for two days at once, separately. Mr. Ishiyama of the Facial team gave me in impossible order (laugh), so we had to put three hundred markers for shooting. We recruited Mob characters’ faces from within the company and the MC team conducted 3D scans of eighteen people.

FUJII

After scanning, we pasted polygon mesh to the data for editing.

KOSHITA

Our company staff appeared here and there as extras in the film.

Animation that explored the coolest action

KAME-
KAWA

The first job for the Animation team was to integrate the data shot by Motion Capture with the characters. We made a framework for each scene and edited it with the camera work. Camera work were based on the storyboard, however we had a lot of action scenes so it was difficult to cut straight from the storyboard. So we worked with Mr. Kawamura in order to keep the camera work in mind.
Basically we referenced some action scenes with good camera work from live action films. We included some action scenes by keyframe animation for Demons that could not be shot in Motion Capture. This film is action based so my goal was to emphasize more brutal camera work and to also give a hand held camera kind of feeling, and I was conscious not to overly emphasize the CG, anyways I tried to make the coolest images.

KAWA-
MURA

When we shoot live action films there are some physical compensation made due to the weight of the camera overcoming inertia, so we did this on purpose.

KAME-
KAWA

I guess live action film crews tries to prevent this, but we didn’t in order so that we could produce a more realistic feeling. But because there were so many action scenes, we had a hard time editing Motion Capture like “Let’s accommodate the Motion Capture’s data for the camera to emphasize this” “Let’s delete here” and so on. We also made changes to the Demon’s movement to fit each of their uniqueness.

KAWA-
MURA

We did not have a huge set comparable to Ebisu Bridge, so we divided it into about ten blocks and then filmed the Motion Capture.

KAME-
KAWA

For the giant character like the GANTZ Robot and Gyuki, we added many elements through keyframe animation based on the Motion Capture’s movement and then conveyed the heaviness and size with camera work. We referenced “Pacific Rim”.

KAWA-
MURA

It is important to note that the original manga was published before “Pacific Rim”. It is interesting to see that Japan and USA influenced each other as if we tossed a ball back and forth.

KAME-
KAWA

If the giant size characters had moved slowly right at shooting, it would seem deliberate, so we asked them to move at a normal speed that they felt comfortable with and then we made it slow down to convey size and heaviness. With regards to the GANTZ Robot, several team members worked on it so I had to accommodate for this when I was inspecting since each member had a different sense of speed. Personally, I wanted to work more on the Robot VS Gyuki scene but there was a limit because the data was so heavy. I loved the scene where the Gyuki appeared from the river and faced the GANTZ Robot. I also liked the action scenes in Shibuya at the beginning and the scene when Kurono and Ochan appear while riding a bike.

  • Robot VS Gyuki scene
  • Kurono and Ochan riding a bike scene

KAWA-
MURA

There were ten action scenes which included some small ones so it was hard to think for each of them, therefore we first proceeded with the rough plans I had made in terms of their formations and images which were then passed on to the Action Director Mr. Kensuke Sonomura. Mr. Kamekawa then checked if Mr. Sonomura’s ideas could technically mesh with the CG.

KAME-
KAWA

We left it up to Mr. Sonomura’s decision for the action scenes since he could make it cooler. It was a full length film with many action scenes that I have been anticipating, plus it was good challenging work for an animation team. I was quite satisfied with the result since everybody gave their full effort to it so that we would have no regrets. I usually work with drama scenes and conversation scenes. I am fine with that but working on GANTZ with such great materials, it was an amazing challenge for me as an animator.

KAWA-
MURA

The layout improved so much, I think. I didn’t have to ask you to redo it.

KAME-
KAWA

Our team members’ skills had improved and all our members saw and referenced the films which Mr. Kawamura mentioned at the beginning so that everyone had the same ideas for the images.

KAWA-
MURA

What did I say to watch?

KAME-
KAWA

“Edge of Tomorrow” “Hellboy” “Amazing Spider-Man” “Captain America: The Winter Soldier” and so on. We were supposed to complete Mr. Kawamura’s requirements and we thought that we could even add some suggestions to it. He was an in-company Director so it was convenient for communication.

KAWA-
MURA

I was used to working as an animator, so I am good at explaining technical terms and I knew how hard that was, so I tried to make it fun for everyone… however, the last half of the schedule was tight. I regretted not being able to say some sincere words of concern.

KAME-
KAWA

It was good to have some tensely charged situations but only a little bit sometimes (laugh).

The face is life for full CG film

ISHI-
YAMA

I took care of the entirety of the characters’ face from facial rigging to animation. My goal for “GANTZ” was to see how far we could recreate the characters’ expression of feeling and convey it to the audience. That was the main matter for me. Mr. Kawamura also kept mentioning this. We shot pilot episodes twice and we tried to work out how much more expression of feeling we could convey in the second one.
Japanese people don’t over exaggerate their face like western people so we focused on subtle changes in facial expression. Japanese people speak like they are muttering. For fragile, delicate, oral movements, if the rig was not good enough then it would look as stiff as a robot, like it just opens and shuts. So we tried to make it look tenderer.

-Did using Performance Capture create many changes to your work?

ISHI-
YAMA

The body and face were perfectly synchronized so it should have better conveyed feelings since it operated simultaneously from body, especially the eye’s life. If the eyes were fine then it looked alive. We often hear “they have dead eyes,” in the CG industry.
Additionally, I personally cared about the blinking. In live action films, actors try not to blink as per instructions. That is fine for live action film but if it was CG they would look like a doll, so I let them blink on purpose. Before, I used to insert a blink when characters changed their eye’s direction. This time for more realism and honesty, I didn’t delete that much from what we shot so that it looked more real.

KAWA-
MURA

Facial expression is the most important thing in a full CG film, so the CG producer and I placed him under a lot of pressure (laugh). He is such a passionate guy so he responded to us very well. Normally we don’t do that much testing midway through but we did because we all cared about reaching certain levels of quality.

ISHI-
YAMA

We made the first pilot episode, the second pilot episode, and the film itself, from respectively different methods. We destroyed and re-created it again and again, including the structural matters and everything.

KAWA-
MURA

We did not just make a film. We also had to make new technique developments, which was a drag.

ISHI-
YAMA

I actually conducted a test where I put about 350 markers on my face, it took about three hours.

  • Facial test

KOSHITA

We made some major changes in the studio for that shot (laugh).

ISHI-
YAMA

We secretly worked on it with Motion team’s cooperation without Mr. Kawamura. In short, we shot about eighty different kinds of movement such as eyebrow up, open a mouth, and so on as motion data, and then embedded them all into CG characters. Three hundred points moved for just opening a mouth, requiring a lot of information to make them move smoothly. Eventually it was all about how realistic they were. I found out that a nose also moved a lot surprisingly.

KAWA-
MURA

He just simplified the secret of how human beings are able to read feelings to three hundred points of information.

ISHI-
YAMA

The toughest part was how the image we saw in the facial stage of the work and the final image rendered were different so that I had to reconcile them within my brain as I worked. In addition, we often discovered side effects from lighting and so on after rendering which required fixing. So we had to double-check.

-Each character had his/her own Facial team?

ISHI-
YAMA

Most of the team members are fans of GANTZ, so they divided the team to each character to improve their work. They tried to compete against each other like Team Kato vs Team Reika since they were emotionally attached to their character.

KAWA-
MURA

They must be all guys for Team Yamazaki then.

ISHI-
YAMA

No, it was two guys and two girls. Though the guys knew what kind of face would be cute for guys, it shouldn’t be obsequious, so adding a female point of view was an asset. However, Team Reika was all guys since they would like to work on her fervently (laugh).
The hardest part to make was the placement of the nasolabial folds, especially for female characters and Nishi. If we didn’t put them, there was no realism. If we put too much, it looked scary and grotesque. So we discussed this with Mr. Kawamura many times.

KAWA-
MURA

Foreign characters like those from Disney, they openly show nasolabial folds, so I have been interested in this cultural difference. Maybe the difference is in how we talk.

ISHI-
YAMA

For live action film as well, if there was something a bit disgusting on a face in some scenes, it added richness to the character overall I think. Also, if pretty actors showed strange faces, that can be cute. So I put some efforts towards making a good balance to this. In addition, I heard that the Lighting team also worked very hard to cover the nasolabial folds.

-After I listened to your stories, I find that the office world is the same as the GANTZ world. You can’t go home until the mission is completed.

KAWA-
MURA

“Do it. We have no other way” we would say at the work site (laugh).

MINA-
MOTO

We couldn’t see the end.

KAWA-
MURA

Who the heck is the enemy, maybe I AM Nurarihyon (laugh).

Most readers may have wanted to hear from the leading actors, however they always give an obligatory talk for the promotion of a new film in order to keep up their reputation and will stick to a given line. For movie lovers, it is very interesting to listen to the behind-the-scene stories from the production staff. It might be hard to read since there were some technical terms, however many interesting stories were pulled out by this interview and it will be good for people who wants to know about film making and for those who would like to be a creator in the future.

(From the left) Morita, Minamoto, Kamekawa, Ishiyama, Fujii, Koshita

© Hiroya Oku / Shueisha "GANTZ: O" Production Committee