A mind-blowing project from AMPD immerses human participants in virtual reality (VR) environments where they interact with computers to, collectively, build the experience.
The School of the Arts, Media, Performance & Design (AMPD) at York University produces cutting edge research-creation projects that bring together seemingly disparate disciplines – art, mathematics, philosophy, music and virtual reality (VR) – in astonishing ways. With a lab that focuses on “computational worldmaking,” how could it fail to capture imaginations everywhere?
One compelling endeavour is the Curious Creatures project by Sarah Vollmer, a PhD student under the supervision of Professor Graham Wakefield, a core member of VISTA (Vision: Science to Application) and the Canada Research Chair in Interactive Information Visualization. Vollmer was the first digital arts and computer science trainee in the VISTA program.
From left: Sarah Vollmer and Graham Wakefield
Curious Creatures is an interactive VR project, funded by VISTA and the Susan Crocker and John Hunkin Scholarship in the Fine Arts. In this setting, human participants engage in VR environments where they interact with computers in order to, collectively, build the experience.
This project was profiled in the MOCO ’19 Proceedings of the 6th International Conference on Movement and Computing (October 2019).
Alice Lab
Curious Creatures began in the Alice Lab, directed by Wakefield. This unique space is designed to foster computationally literate art practice in the construction of responsive artificial worlds. It integrates computer graphics, augmented reality, computer vision, complex systems and compiler technology to create novel art and software.
Inside the Alice Lab
Curious Spaces: A creative environment where software can improve itself
Curious Spaces is a project of research in the Alice Lab that explores ways in which software can suggest changes and enhancements to make to itself and accept or reject these changes according to reward functions.
Wakefield underscores that these reward functions do not serve a prior externally specified goal, such as better classifying pictures of animals or genres of music, but rather that these rewards are intrinsic to the artificial agent’s own development. “It’s inspired by human creative and playful processes that have intrinsic rewards, such as discovery, curiosity and self-actualization,” he says.
The objective of Curious Spaces is to pursue new depths of mixed reality human-machine interaction and responsive environments toward a larger goal of intensifying aesthetic experience through meaningful collaborative human-machine interaction over extended durations.
Curious Spaces, in essence, facilitates intricate, fluid or open-ended interchanges between humans and technology. In doing so, it sets up a situation in which artificial realities can display high levels of artificial intelligence.
Virtual environments that make themselves, with human help
Vollmer’s Curious Creatures project, borne in Curious Spaces, is a project of research-creation, which the Social Sciences and Humanities Research Council defines as an approach to research that combines creative and academic research practices, and supports the development of knowledge and innovation through artistic expression, scholarly investigation and experimentation.
A sample structure created by Curious Creatures, illustrating a design used for testing the construction of a scenario
In this project, sensorial engagement and interactions between a human participant and their VR environment – as both agents of design and agents of collaboration during the creation process – reflect intellectual and emotional decisions encountered throughout the ongoing construction process.
“Whether the experiential moment is augmented, controlled or induced by techniques, such as visual cues, the main motivation is a curiosity-driven fusion of art, science and the technology. Identifying moments that we share and experience daily through collective agency are those that I seek to imitate, create and research-create,” Vollmer explains.
Compelling process fully immersive
The process and design of this project are compelling. During the coding experimentation, Vollmer worked with CodePen.io, an online hub of coders who build test cases, troubleshoot and find inspiration. “You can erase, rebuild, rearranging the contextual emotions made tangible as you watch it built (rendered) right in front of you,” she states.
To facilitate the building and creation process, a room-scale VR system was used and connected to an enabled browser, such as Google Chrome.
The process is fully immersive: custom built haptic (related to touch) devices can be worn on the participant’s body. This ties the virtual and visual with the physical and tangible – essentially fusing both worlds. Also connected are a set of ManusVR gloves that the participant can use to "touch" within the VR space, enticing curious creatures found within that world to either approach or interact with the participant.
Custom built haptic devices can be worn at different points of a participant’s body
“Graham and I are most excited about this artificial life/nature (a world of curious creatures) arising from the participant’s (a singular curious creature) artistic gestures, and how this living virtual creature-inhabited world may cause us to react and act in ways ‘it’ finds interesting,” she says.
Vollmer sees this project in its infancy, as VR offers limitless dimensions, environments and scenarios for interacting. “The consideration of a more complex personality-matrix and possibly the inclusion of a type of artificial life are beginning to drive my curiosity,” Vollmer says. “By including gesture tracking, unique artistic integration is possible – fluidic paint-like trails can leave the participant’s hands with a life of their own, such as twisting, floating, diffusing and combining into new forms of artificial, yet curious, life.”
Vollmer says that the project has grown into a living space of countless forked paths of questions and attempted answers induced by the exploration of the technical infrastructure available for VR creation. She sees parallels to art, engineering and technology that inspire this curiosity-driven exploration.
Vollmer's article on the project is available for download. More about Wakefield is available on his Faculty profile page. Additional information on the Alice Lab for Computational Worldmaking can be found on the lab's website. More about Curious Spaces can also be found on the Alice Lab website.
To learn more about Research & Innovation at York, follow us at @YUResearch; watch our new animated video, which profiles current research strengths and areas of opportunity, such as Artificial Intelligence and Indigenous futurities; and see the snapshot infographic, a glimpse of the year’s successes.
By Megan Mueller, senior manager, Research Communications, Office of the Vice-President Research & Innovation, York University, muellerm@yorku.ca