UrbanGuitar.com

Main Stage :: Guitarmed and Dangerous

Guitarmed and Dangerous

Experience Modal Kombat, “The First Ever Guitar-Controlled Video Game Battle”

September 03, 2006

Think PlayStation can make you a Guitar Hero? Think again.

New Yorkers David Hindman and Evan Drummond have created a new form of gaming, and the guitar- a real one- is the controller. Conceived and developed while Hindman was a student at New York University, “Modal Kombat” is an extension of the classic game Mortal Kombat. Instead of battling with buttons, however, characters fight by responding to notes, chords, and melodies played live by participants on guitar. The resulting performance is “a modern-day ‘dueling banjos,’” says Hindman. “It has raised the bar for performers, composers, interface designers, and competitive gamers. Modal Kombat delivers an engaging combination of music, choreography and good old-fashioned fantasy street fighting.” I asked David where the idea for Modal Kombat came from.

David Hindman: Right after graduating from the Yale School of Music, I enrolled in an interdisciplinary graduate program which concentrated on the convergence of art and technology. The Interactive Telecommunications Program at NYU (ITP for short) admits artists, scientists, programmers and those with a wide range of other professional backgrounds. Those with no computer programming experience are required to take a number of foundation courses that teach the fundamentals of computational media. My final project for the foundation class "Introduction to Computational Media", was a video game entitled "Sonic Pong". It was basically a version of the classic game PONG, but in this iteration, the volume of incoming sound controls the paddle position, allowing the player to engage in PONG by simply yelling at the screen.

From this point onward, I knew that sound would be a valid and exciting gaming interface. As a trained guitarist and budding programmer, the future scenario for the sonic gaming interface for me became crystal clear: Two battling guitarists in a classic street-fighting video game. I thought, what better game to play that Mortal Kombat, and what better adversary than friend, colleague and monster guitarist Evan Drummond? We debuted the performance piece "Modal Kombat" at the 2005 Interactive Media Culture Expo at the Chelsea Art Museum.

Urban Guitar: On your website, you talk about music as a competitive sport. With a scenario such as this, it's tempting to imagine that the person who plays better or faster or more musical defeats the enemy, like the way one would be judged in a conventional competition. But that's probably not the case.... what determines one player as the victor?

David Hindman: Unfortunately, a computer can no less judge "musicality" or the even more nebulous concept of "better" than even the most qualified of human musical judges. These subjective terms do not serve as a basis for interface algorithms in Modal Kombat.

Modal Kombat is a three movement work, where we only fight to the death in the third movement. Movements I and II are explorations of choreography and sonically-driven theatrical lighting. Early on, we discovered that it was equally interesting to make characters dance as it was to make them fight. In the third movement we have various musical gestures assigned to trigger a character's moves. For example, in movement III, when I play any three-note, ascending arpeggio across the bottom three strings, Scorpion (the character who I usually choose) will shoot his "Get Over Here" snake-thing out of his hand. In this case the large musical gesture maps to the large aggressive virtual gesture. Each of our top strings are mapped to punches, so playing a tremolo on the first string will result in a fast sequence of punches. On the 2nd and 3rd strings, if either of us play an ascending passage, the player will advance forward whereas descending passages result in a player moving backward. The mapping possibilities are virtually limitless, so our current mapping system takes into consideration the idea of synesthesia, which is essentially experiencing physical phenomena through alternative senses; examples are seeing sound or hearing color. For example, a big chord "looks" like a lightning toss, and a fast tremolo "looks" like a fast succession of punches.

While both Evan and I compete through the video game, we actually work together to create listenable music in the overall context. While my large chords may trigger a very powerful special move, this can quickly become unmusical, and therefore the overall concert suffers. The victor in the performance is simply he who uses his mappings well enough to destroy the other musician's avatar. However, we both lose if any one of us neglects musical concerns for the sake of winning the game. Performing Modal Kombat requires a delicate balance of thoughtful musicality and brute aggression.

Software can judge "faster" easy enough, but one of the goals Evan and I decided on at the beginning was to try and create listenable music while simultaneously taking into consideration the asynchronous rhythmic patterns inherent to video game controller input. In other words, while we are competing against each other vis-a-vis the video game, we are working together to create a tolerable listening experience for the audience. This question hit hard at some of the fundamental pillars of human-computer interaction.

Urban Guitar: Do specific notes or chords trigger certain moves? What about speed- Does the velocity of a note or the repetition of notes make a difference?

David Hindman: All of the above scenarios occur in Modal Kombat where appropriate. The idea is to take sonic gestures and map them to similar virtual gestures. For example, a loud chord might trigger the character's special move, which may include but is not limited to throwing lighting, fireballs, etc. Fast tremolos on the first string we found to map well to punches; quick, exact musical articulations manifest themselves well as rapid virtual physical gestures in this case.

There is a certain volume threshold that we must exceed otherwise the character may not respond, but other than that, the triggering of moves is strictly binary; on or off. For instance, when playing a video game with a standard controller, sometimes we may think that pushing the buttons harder makes them more responsive, while this simply not the case. As far as speed is concerned, there are particular sections of Modal Kombat where first-string articulations trigger punches. In a one-to-one mapping such as this, the speed at which we execute our tremolos has direct effect on how fast the character will punch.

So the short answer to this question is “all of the above.”

Urban Guitar: Have you applied this concept to other video games?

David Hindman: “Guitariokart” is our latest video game piece. Same idea as Modal Kombat but we control the classic Nintendo Game MarioKart with our guitars instead.

Urban Guitar: What about the instruments- Can this be used with something other than guitar?

David Hindman: The very genesis of this project came out of a class at ITP called Introduction to Physical Computing, a class that taught the fundamentals of microcontrollers and electronics. At this time, I was working on the technology that would allow the birth of Modal Kombat with a fellow classmate, Spencer Kiser, sound designer and ASCAP employee. We called the original project The Sonictroller, and experimented with many instruments including keyboards, trumpet, guitar, voice, and midi-drums. We succeeded in creating small demonstrations controlling the video game "Rampage" with keyboard and voice, trumpet and guitar, and created a demonstration of "Mortal Kombat" that was controlled with keyboard and voice that was showcased at the NIME 2005 conference in Vancouver. While this initial research laid the technological groundwork for Modal Kombat, guitars have been the only instruments used in performance so far.

Urban Guitar: Do you have a "dream instrument" you'd like to apply this to?

David Hindman: The dream instrument for me is the guitar, which is why it was first to be showcased in performance.

Urban Guitar: How have audiences reacted to this project?

David Hindman: I think audiences really like it, in spite of the fact that they are watching other people play video games. Just watch the videos and you can hear and see how people respond to it.

Urban Guitar: Do you plan to eventually make this available for purchase, or is this something that you intend to keep only for your own performances?

David Hindman: There is quite a lot of technology involved to make these performances work. I wouldn't be against marketing a device that would allow others to engage in Modal Kombat, or to control any video game with their guitar, but every instrument is so different and every game is so different that it would be difficult (but not impossible) to create a system that is general enough to accommodate a wide range of instruments, games, and game consoles. For the time being the effort will be on developing this performance genre to its fullest. That said, I would like to host guitar competitions that would utilize the technology to allow guitarists or other instrumentalists to fight each other through video games in large, public spectacles. This may be the next stage in Modal Kombat's evolution, and may be the first likely scenario in which others may be able to apply this technology to their own respective instruments.

Urban Guitar: If it was made available for purchase, what equipment would people need to buy?

David Hindman: Let's see... you'll need a fast laptop (preferably a mac), a gaming console, the Mystery Box (the custom-built musical instrument-video game console interface... I own the only 3 of these that I'm aware of), and some pixie dust.

Urban Guitar: Can this be used with standard instruments, or are there special requirements?

David Hindman: Every instrument is very different.... each has its own range, some transpose, and each has a different set of harmonic overtones that accompany fundamental pitches. Some can produce simultaneous pitches, and some are single-lined only. For these reasons, very different systems might have to be created to enable different instruments to control video games.

Urban Guitar: In what ways would you expand the project?

David Hindman: The next piece Evan and I are planning will involve an larger musical ensemble and will control a more modern and therefore more complex game, from an interface standpoint. For example, a four-person shooter like Bond or Doom could make for some interesting musically driven visual events. For example, we could use two sustaining instruments like trumpet or violin to control direction, while quick guitar attacks could trigger machine guns. We could delegate tasks of two game characters across two different musicians, or employ any combination of musician and game sprite for the sake of competition or collaboration.