Back

XR | Music

Interaction work for music creation in XR


My interaction work for PolysoundXR. A finalist prototype built for the XR Design Challenge 2024.


This is from the team build of Polysound XR an immersive music creation prototype built for the XR Design Challenge ‘24 in which we were a finalist. My job within the team was to work on the main build with Tomas focusing on visual & interaction design and this is the work I produced.

Object Interaction

Rings Interaction

In our initial discussions we knew we wanted the experience to be 360º within the space so I built an interactive ring section to test the idea. This worked but the rings proved troublesome as a full 'sequencer' due the the array creating too many collisions and too messy for instrument changes.

Grouped structure cubes

Tom built a second example which worked a lot better. By utilising a cylindrical structure built from squares the interaction was a lot cleaner. When I updated with the instrument signifiers & toms logic we ended up with a result that worked. Toms sequencer logic after this really tied it all together.

Gesture Interaction

Instrument selection

To feel 'free' and for speed it makes sense not to have a traditional UI to pull up and interact with so adding a swipeable interaction that easy to access through palm up made sense.


Instrument selection

There are four instruments in total and they are represented as shapes. This is visually consistent and provides instant recognition with a uniform visual language and offers scalability.

Spatial positioning

Sound is generated realtime and manipulated depending on spatial positioning. A unique sounds is generated from each hand and tones vary in volume from left to right (X) & pitch from top to bottom (Y).

Play Gesture

A thumbs up on the left hand triggers the play sequence. This was a main function and having it on the same hand as the instrument selection meant the gesture was clean with no clashes.


Stop Gesture

It felt right to use the thumbs up gesture on the opposing hand to remain understandable and recognisable to the user for a smooth experience.

Reset Gesture

We needed a way to reset the sequence so to balance out the interactions it felt like the right move to invert the stop gesture to keep controllability restricted to hands and with something that takes a touch more effort to recognise.

Conclusion

Working on the Interaction for PolysoundXR was cool! Super fun to try and work things out. Yes, there was a lot of testing and a hell of a lot of failures (Including sickness and headaches) but with persistence we were able to get it to where we wanted. Overall, great to work with people that had the same passion and enthusiasm for XR.

©2026

►