I’ve been working on my Vive-tracking robotic cameras for just over a year now, and I’m falling over with excitement to announce that they’re now autonomous enough to have their own name:
Cinema Swarm
For the first public outing of the multi-camera system, we filmed 24 bands in 2 days.
To be in Brisbane’s best music venue, surrounded by hugely talented people, and have this crazy thing working, moving, tracking, and capturing footage exactly how I’d hoped it would when I first designed it inside my brain. That was a pretty special time.
Most of my recent work in VR and Mixed Reality has been rough, proof-of-concept stuff. Nawlz is an attempt to create something with a little more structure. Returning to Australia from a secret overseas Tilt Brush project, SUTU stayed over for a couple of nights and we documented the entire creation of a Tilt Brush artwork.
The piece is a 3D re-imagining of a panel from his interactive comic Nawlz (Season 1, Episode 5, Page 3 for those following along at home).
The video combines a number of mixed-reality techniques I’ve been developing for months: VR timelapse, VR Automatic Camera Tracker in both timelapse and realtime modes, in-app greenscreen painting and some other secret spices.
Australia is lucky to have one of the greatest youth-oriented national broadcasters on the planet – Triple J. Their weekly current affairs show Hack covers a huge variety of topics, and an excellent example of this is my interview about working on VR and music performance being introduced by 10 minutes of Tinder Date Murder Trial coverage. [Skip to 11:25 to avoid murders]
Listen: music video and VR on Triple J Hack
Broadcast 18 October 2016. Reporter Stephen Stockwell visited my place to experience Ball Park Music’s Whipping Boy VR video, then we talked about Tilt Brush, Reggie Watts playing AltspaceVR, and what virtual gigs might mean for Australian bands.
In brief: Sam Cromack painted the clip in Tilt Brush; I directed and built the technical rig; the conversation moves from turning a room-scale work into a 2D music video to performing live inside VR.
“If you can put on a show that people from all around the world can come to—without having to physically move around a lot—that’s a pretty magical thing.”
Jaymis Loveday, Triple J Hack, 18 October 2016
Archive update, 2026: This 3:47 Jay-focused excerpt uses 11:25–14:40, 16:22–16:48 and 17:03–17:10 of the preserved 29:45 broadcast. The intervening interview with Lincoln Savage and brief headset-removal chatter are omitted; the full source recording remains preserved offline. Audio: Triple J Hack / Australian Broadcasting Corporation.
Transcript
Speaker-labelled transcript generated from the preserved broadcast with Whisper (medium), checked against the source timings, and lightly edited for names, punctuation and readability. The audio remains the authority.
TOM TILLEY: Now, when you think about virtual reality, you probably think about it in porn or the gaming context. And in just a moment, we’ll take you inside a virtual reality game so real that some players even vomit. But what about music? Some virtual reality developers say we could be going to virtual reality gigs within months. There’s already some serious movement in the virtual reality video clip space. Stephen Stockwell went inside the first video clip ever created in virtual reality to find out what’s in store.
STEPHEN STOCKWELL: I’ve never been in virtual reality before, so I’m trying to figure out how I actually describe this place. I’ve never had to sit in a galaxy looking at a volcano erupting from a man’s chest through a spiral rainbow. That’s all part of a film clip. This is a piece of art painted inside virtual reality—which is the music video for Ball Park Music’s Whipping Boy. The world I’m standing in was drawn by Sam from Ball Park Music. And Jaymis, whose house we’re sitting in right now, was the director and provided the technical gear to make it all happen.
JAYMIS LOVEDAY: The best way to experience this is to own a VR rig and walk around inside it with the song playing. But most people can’t do that, so we sort of tried to figure out how we could show it to people on a 2D screen and have it still be exciting.
STOCKWELL: It’s really hard to describe, but the light painting that Sam has created in virtual reality is really immersive. And you put on the headset and you put on the headphones and it really feels like you’re in another world. Now, there are still limitations with the cost of virtual reality. The headset I’ve been wearing is about $1,500. And then you need a pretty pricey computer to run all the software—probably a couple of grand there as well. I don’t want to tell you that it’s worth it, but I did have an incredible time in virtual reality.
JAYMIS: VR itself—it is actual virtual reality. That’s the weird thing. People will not walk off a cliff in VR because that’s scary. And being inside these psychedelic scrawlings of Sam’s, you get in the mindset of someone who’s sitting inside this crazy piece of artwork floating in a galaxy. While you’re in there, it’s entirely convincing.
STOCKWELL: While the Whipping Boy video was the first film clip made in virtual reality, it’s really just the first step.
JAYMIS: From here, we’re going to start seeing a lot more of this kind of stuff—interactive music experiences, and even virtual reality gigs. Altspace put on a show with the comedian and musician Reggie Watts a couple of months ago. They basically put him in a full motion-capture suit, built him a custom avatar that looks like Reggie—with the amazing afro and beautiful kind of suspenders that he always tends to wear—and he put on a live comedy show in a virtual comedy club. But that’s comedy, not music. There are some electronic gigs that are happening in virtual reality already, but there’s a bit more technical work that needs to be done before you can get really sweaty to Violent Soho in your living room. Electronic music is much easier because it’s mostly about hand movements, which are already tracked. To get live rock bands with people playing guitars and things like that is going to take a little bit more time because generally there are more people. So there are a little bit more technical hurdles to get—for instance, a Ball Park Music set live in VR—but it’s not out of reach at the moment.
STOCKWELL: Jaymis isn’t the only person doing this kind of thing. He’s not even the only person in Brisbane working in this space.
[The broadcast’s separate interview with Lincoln Savage is omitted from this Jay-focused excerpt.]
STOCKWELL: Being able to go to a variety of live performances in VR is something that’s going to be happening inside of a year. While the idea of strapping on a VR headset to check out a band might sound pretty futuristic, Jaymis says it’s not far off at all.
JAYMIS: Touring for a band in Australia is really hard because it’s such a big country with such a small, relatively, number of people. If you can put on a show that people from all around the world can come to—without having to physically move around a lot—that’s a pretty magical thing.
TOM TILLEY: Welcome back, Stocky. That was Jaymis he was speaking to there, who directed the first film clip created in virtual reality.
It’s been a little over a week since I launched my Automatic Camera Tracker project. It’s still a work in progress, but as SUTU is monopolising my Vive and nerd-basement at the moment, it felt like a good time to check in on its progress.
The current version is working alongside other VR apps, and can track a controller or VR headset at realtime or timelapse speeds as the user moves around the VR and IRL space. A little like this:
I now have a robot camera sidekick who follows my movements around the room while I'm in VR! Can't believe I got this working so quickly. pic.twitter.com/dNAhFdOkDa
… which is exciting, because it’s a project I’ve had in my mind since I first used the Vive over a year ago, at PAX Prime in Seattle. As soon as I waved the controller in front of my face and noticed the tracking speed and accuracy I started mentally designing camera tracking systems. I wanted a VR system in my life because I love games, but I needed a Vive for filmmaking science experiments.
I’ve spent today with SUTU filming with my most recent version of the system, using it for both timelapse and realtime, handheld shooting while he worked on a Tilt Brush version of a piece from Nawlz.
I’ve been thinking for a while about taking the Vive outside to test how it handles Nature. The object-tracing capabilities of Tilt Brush have already been pretty well established. SUTU and I had planned to take the VR rig outside last week, but it ended up raining, so I had to do it by myself after he left :|
To view the art in Tilt Brush yourself, and spend a little time in my virtual backyard download the .TILT file here.
The process was great fun! Outside Vive is a super strange feeling, but it worked well. Even climbing a ladder while blind was quite fun. (Note: Don’t take your Vive outside at daytime, tracking won’t work and sunlight through the lenses can burn the screens.)
I took a couple of precautions, most important was applying some tape to the controller so it didn’t get scratched up by the brutal nature in my suburban backyard. I figured out last week that the most accurate way to “trace” objects in Tilt Brush is to use the back of the controller, as the “stylus point” is at the top of the handle, not the tip of the tracking ring.
I’m still experimenting with mixed-reality motion-control shots. This version I stabilised the Tilt Brush feed in After Effects as it had a constant slow oscillation which I think may be to due to vibrations from my robot’s stepper motors.
While investigating tracking precision for motion control timelapse filming in VR I had a thought, and then sidetracked myself for half a day designing various Tilt Brush/Vive pendulum rigs. The first one is very simple, but worked beautifully:
After completing the Nihilist Party Album video, the Ball Park guys spent a couple of hours trying my Vive VR rig, and they were quickly on board with the idea that their next video should be produced in VR.
In the lead up to the release of Emma Louise’ new album Supercry, Mushroom Records asked if I’d spend a day with Emma and record some intimate, solo live tracks in one of her favourite creative spaces, Brisbane’s Old Museum.
Teensy 2.0 uses 5v logic, so is easier to use with a rotary encoder that requires 5-20v. (Other Teensy models work, other than the LC which doesn’t support 5v logic.)
Official Store – PJRC – US$19 Australian Stockist – Little Bird – AU$29
Remove bike casing and the pedal on the opposite side to the drive belt
Screw printed pedal gear on to crank, and then put pedal back on
Connect encoder to Teensy, either solder pins directly or using a breadboard:
Red Wire (Pulse): Pin D0 (pin 5 on Teensy 2.0 and in Arduino software)
Green Wire (Pulse): Pin D1 (pin 6 on Teensy 2.0 and in Arduino software)
White Wire (+5V): Pin VCC
Black Wire (Ground): Pin GND
(These wire colours may change depending on your encoder source – so check your part documentation, or do some testing before soldering to the board)
Use Encoder Bracket and cable ties to attach the encoder to bike frame, so the gears mesh together. Cable ties allowed me to adjust the encoder position to ensure the gears weren’t pressed together too hard
Use some more cable ties/velcro to keep cables away from moving parts
Plug Teensy into your computer with long USB cable
Set the Arduino software to use Teensy as a Joystick under menu: Tools > USB Type > Serial + Keyboard + Mouse + Joystick
Upload the sketch to Teensy (Ensure that “Teensy” is selected under menu: Tools > Port), open Serial Monitor and test that the encoder values are changing when the crank moves
Test your maximum pedalling speed outputs a value of +512, increase or decrease the “speedMultiplier” variable (line 17) if you’d like to pedal faster or slower to output full speed on your controller
Once you’re happy with your pedalling speed, change the “readDelay” variable (line 20) to 5, to make the controller latency faster
PC – x360ce
Connect Teensy and Xbox Controller (or other controller). Copy x360ce into game location (for Rocket League this is: /Program Files (x86)/Steam/steamapps/common/rocketleague/Binaries/Win32/) When playing without exercise controller, you need to temporarily remove the xinput1_3.dll file, or the game won’t start correctly.
Configuration:
Controller 1: Xbox
Triggers Off (for throttle. remove trigger axis)
Advanced:
Pass through: Off
Combine this controller: Combine into: One
Controller 2: Teensy
IHAxis 1, HAxis 1 – added to triggers or other controller.
Advanced:
Combine this controller: Combine into: One
Options: Allow Combining
Mac – ControllerMate
ControllerMate script is available on Github. The “Bike Controller Axes” page should contain “Wireless 360 Controller” and “Teensy Keyboard/Mouse/Joystick.” If you’re using a different controller, you may need to add the axes yourself.
The “Bike Controller Combined” page should create a virtual controller.
Start your game, and enjoy the exercise!
Video Transcript
Hey there Internet. I’m Jaymis, and I have a project I’ve been working on, that I’d like to share with you.
It’s a game controller that attaches to an exercise bike. It takes your calories, burns them alive, and outputs joystick movements or button presses.
I wanted to create a system that could be used with many different games, would produce forward and backward, analogue-style controls, and which could utilise whatever rotation-based exercise equipment players already own.
My solution uses a rotary encoder connected to a Teensy USB development board, with a 3D printed gear and mounting system that can be customised to different equipment geometries. The circuit requires only basic soldering and electronics skills, and it uses very simple, customisable code. Everything can be put together for around $50 if you have access to a 3D printer, or a bit more if you need to order printed parts.
To integrate with an existing game controller requires 3rd party software which works on PC or Mac. This allows you to combine input from a standard game controller with the bike controller input.
I’ve been super addicted to Rocket League the past year, so I’ve mostly used it to control flying cars, however it works well for games like DayZ, Minecraft, and just about anything that involves plenty of walking or running.
You do lose a bit of in-game dexterity though, as real humans can’t change between forward and backward directions as quickly as in-game characters and vehicles, so if you’re playing online and are concerned about your rankings, it will probably make your scores a little sad.
If you feel that you’d like to convert some of your gaming time into exercise time though, it really is a great solution. I now regularly ride for hour-long sessions without getting bored of exercising, which seems like a Good Thing.
So I’m releasing all of the designs and instructions, to help other nerds to cultivate their own Good Habits.
Good luck. Have fun! (GLHF!), and if you’re interested in future updates or other projects involving gaming, performance, video, art, electronics, music, cooking, and other renaissance nerd topics, please do subscribe, or come find me on the internets.
This video started the same way all Ball Park Music videos start – drinking coffee and talking about internets. The difference with Nihilist is that we grabbed those internets and decided they should be the whole video.
