All in Research

Following the Master’s Hands: Capturing Piano Performances for Mixed Reality Piano Learning Applications

Following the Master’s Hands: Capturing Piano Performances for Mixed Reality Piano Learning Applications

Cagri Zaman
Augmented Reality, Conference Talk, Research

ACM Human Factors in Computing System (CHI) 2023.

Piano learning applications in Mixed Reality (MR) are a promising substitute for physical instruction when a piano teacher is absent. Existing piano learning applications that use visual indicators to highlight the notes to be played on the keyboard or employ video projections of a pianist provide minimal guidance on how the learner should execute hand movements to develop their technique in performance and prevent injuries. To address this gap, we developed an immersive first-person piano learning experience that uses a library of targeted visualizations of the teacher’s hands and 3D traces of hand movements in MR.

The Latent Archive

The Latent Archive

Cagri Zaman
Artificial Intelligence, Virtual Reality, Research

National Endowment for the Humanities Digital Humanities Advancement Grant. 2022

Previously, extracting information from moving images has been challenging and time consuming, requiring historians and film scholars to access footage by manually reviewing sequences over and over to parse the setting, the rituals, camera angle, narratives, and the material cultures involved. Now, developments in computer vision and spatial analysis technologies have opened up exciting possibilities for these scholarly processes, with direct implications for improved public access and future translational tools for disabled communities. The “latent archive” that has always been embedded in moving images can now be captured via machine-enabled analysis: locating the urban or architectural setting, producing 3D spatial reconstructions, and allowing fine-grained examination of point-of-view and shot sequence.

Towards Adversarial Architecture

Towards Adversarial Architecture

Cagri Zaman
Research, Publication, Artificial Intelligence

ACADIA Conference, 2022. Vanguard Paper Award Runner-Up

A key technological weakness of artificial intelligence (AI) is adversarial images, a constructed form of image-noise added to an image that can manipulate machine learning algorithms but is imperceptible to humans. Over the past years, we developed Adversarial Architecture: A scalable systems approach to design adversarial surfaces, for physical objects, to manipulate machine learning algorithms.

NavigAid: AI-Driven Orientation and Mobility System for the Blind

NavigAid: AI-Driven Orientation and Mobility System for the Blind

Cagri Zaman
Artificial Intelligence, Augmented Reality, Research

National Science Foundation SBIR Phase II Project. 2020

NavigAid is an AI-driven mobile Orientation and Mobility (O&M) system, which provides contextually relevant, task-driven solutions to problems such as finding objects, identifying paths of ingress and egress, and understanding the layout of an environment. NavigAid is enabled by our core technical innovation, Ally Networks, which represents a novel neural network architecture that is capable of extracting semantically and functionally relevant spatial features from images, which help to create a human-like understanding of physical environments.

Reasonable Perception: Connecting Vision and Language Systems

Reasonable Perception: Connecting Vision and Language Systems

Cagri Zaman
Publication, Artificial Intelligence, Virtual Reality, Research

Human Robot Interaction 2018. ACM.

Understanding explanations of machine perception is an important step towards developing accountable, trustworthy machines. Furthermore, speech and vision are the primary modalities by which humans collect information about the world, but the linking of visual and natural language domains is a relatively new pursuit in computer vision, and it is difficult to test performance in a safe environment. To couple human visual understanding and machine perception, we present an explanatory system for creating a library of possible context-specific actions associated with 3D objects in immersive virtual worlds.

Hallucination Machine: A Body Centric Model of Space Perception

Hallucination Machine: A Body Centric Model of Space Perception

Nil Tuzcu
Research, Thesis

SM Thesis. Massachusetts Institute of Technology, 2014

In this thesis I present a novel approach to space perception. I provide a body-centric computational model, The Hallucination Machine, that integrates bodily knowledge with senses in a common modality which I call "the sphere of embodiment". Understanding the human experience of space is an important inquiry not only in the context of design and architecture, but in a broad range of scholarly disciplines where humans are the subject of study, whether as biological, social, or cognitive entities.

Six Degrees: A Social Platform to Connect People and Services

Six Degrees: A Social Platform to Connect People and Services

Cagri Zaman
Research

MIT Design Lab, 2014

Hotel lobbies, even when busy, can be places where people—especially those traveling alone—can feel disconnected. Marriott Hotels asked us to think about how digital technologies could create a more social lobby. We began by thinking about how the physical architecture of a space can promote and support social interactions and meaningful experiences using artificial intelligence, the Internet of Things, and so on. Ultimately, we developed Six Degrees, a social network prototype through which guests in the physical space of the lobby can discover how they might be connected to other guests and that supports socializing in events planned by Marriott

LUME Interactive Media Display

LUME Interactive Media Display

Cagri Zaman
Research, Tangible Interfaces, Digital Fabrication

MIT Design Lab, 2013

Interactive LUME Display project explores tangible interactions with digital content through spatially embedded computation in public space at multiple scales from low to high resolution of interactivity. Composed of four interactive touch points, from a mobile app to a large display installation, LUME aims to build the identity for the department of Comparative Media Studies/Writing at Massachusetts Institute of Technology, by increasing awareness and visibility of the different research labs within the department, displaying variety of content, acting as a place finder, and engaging all potential users.

Mediate VR

Mediate VR

Cagri Zaman
Virtual Reality, Research

MIT DesignX Startup Accelerator Challenge Winner. 2017.

Mediate VR is a platform for speech-driven user research in virtual reality. It captures the emotions, challenges, and pleasures of spatial experience through voice recordings.

Users explore a virtual environment, respond verbally to prompts, and engage in tasks. People don’t remember the details of experiences very well after-the-fact; eyewitness accounts are notoriously unreliable. But ask someone what they think or see during an event, and they can give specific, accurate feedback. Mediate contextualizes user voice recordings through data captured from the virtual environment, synthesizing insights and feedback in real time for our clients through an admin dashboard