Following the Master’s Hands: Capturing Piano Performances for Mixed Reality Piano Learning Applications

Following the Master’s Hands: Capturing Piano Performances for Mixed Reality Piano Learning Applications

Cagri Zaman
Augmented Reality, Conference Talk, Research

ACM Human Factors in Computing System (CHI) 2023.

Piano learning applications in Mixed Reality (MR) are a promising substitute for physical instruction when a piano teacher is absent. Existing piano learning applications that use visual indicators to highlight the notes to be played on the keyboard or employ video projections of a pianist provide minimal guidance on how the learner should execute hand movements to develop their technique in performance and prevent injuries. To address this gap, we developed an immersive first-person piano learning experience that uses a library of targeted visualizations of the teacher’s hands and 3D traces of hand movements in MR.

The Latent Archive

The Latent Archive

Cagri Zaman
Artificial Intelligence, Virtual Reality, Research

National Endowment for the Humanities Digital Humanities Advancement Grant. 2022

Previously, extracting information from moving images has been challenging and time consuming, requiring historians and film scholars to access footage by manually reviewing sequences over and over to parse the setting, the rituals, camera angle, narratives, and the material cultures involved. Now, developments in computer vision and spatial analysis technologies have opened up exciting possibilities for these scholarly processes, with direct implications for improved public access and future translational tools for disabled communities. The “latent archive” that has always been embedded in moving images can now be captured via machine-enabled analysis: locating the urban or architectural setting, producing 3D spatial reconstructions, and allowing fine-grained examination of point-of-view and shot sequence.

Towards Adversarial Architecture

Towards Adversarial Architecture

Cagri Zaman
Research, Publication, Artificial Intelligence

ACADIA Conference, 2022. Vanguard Paper Award Runner-Up

A key technological weakness of artificial intelligence (AI) is adversarial images, a constructed form of image-noise added to an image that can manipulate machine learning algorithms but is imperceptible to humans. Over the past years, we developed Adversarial Architecture: A scalable systems approach to design adversarial surfaces, for physical objects, to manipulate machine learning algorithms.

September 1955 - A Virtual Documentary of the Istanbul Pogrom

September 1955 - A Virtual Documentary of the Istanbul Pogrom

Cagri Zaman
Exhibition, Virtual Reality

Virtual Experience Design Lab, 2017

September 1955 is a 8-minute virtual-reality documentary of the Istanbul Pogrom, a government-initiated organized attack on the minorities of Istanbul on September6-7, 1955. This interactive installation places the viewer in a reconstructed photography studio in the midst of the pogrom, allowing one to witness the events from the perspective of a local shop-owner.

Supersense AI

Supersense AI

Cagri Zaman
Artificial Intelligence, accessibility

Supersense is a mobile assistive technology application for visually impaired people & blind (VI&B) Mobile artificial intelligence systems present a significant opportunity for improving the lives of more than 300 million VI&B by reducing the challenges in their everyday life such as navigation, orientation and mobility (O&M), reading, and more. Accessibility-enabled applications often yield poor and overwhelming experiences for VI&B who has different needs and drastically different ways of interacting with mobile technology. In making Supersense, we adopted a user-centered design process and introduced several guidelines for designing accessibility-first applications. The projectis funded by the MIT DesignX program, the National Science Foundation, and the Veteran Affairs. It has reached more than one hundred thousand blind users globally.

Spatial Experience in Humans and Machines

Spatial Experience in Humans and Machines

Cagri Zaman

PhD Dissertation. Massachusetts Institute of Technology. 2020

Spatial experience is the process by which we locate ourselves within our environment, and understand and interact with it. Understanding spatial experience has been a major endeavor within the social sciences, the arts, and architecture throughout history, giving rise to recent theories of embodied and enacted cognition. Understanding spatial experience has also been a pursuit of computer science. However, despite substantial advances in artificial intelligence and computer vision, there has yet to be a computational model of human spatial experience. What are the computations involved in human spatial experience? Can we develop machines that can describe and represent spatial experience. In this dissertation, I take a step towards developing a computational account of human spatial experience and outline the steps for developing machine spatial experience.

NavigAid: AI-Driven Orientation and Mobility System for the Blind

NavigAid: AI-Driven Orientation and Mobility System for the Blind

Cagri Zaman
Artificial Intelligence, Augmented Reality, Research

National Science Foundation SBIR Phase II Project. 2020

NavigAid is an AI-driven mobile Orientation and Mobility (O&M) system, which provides contextually relevant, task-driven solutions to problems such as finding objects, identifying paths of ingress and egress, and understanding the layout of an environment. NavigAid is enabled by our core technical innovation, Ally Networks, which represents a novel neural network architecture that is capable of extracting semantically and functionally relevant spatial features from images, which help to create a human-like understanding of physical environments.

Virtuality & Presence

Virtuality & Presence

Cagri Zaman
Teaching, Augmented Reality, Virtual Reality

This course aims to help students invent and analyze new forms of extended reality (XR) experiences, computer-based art, gaming, social media, interactive narrative, and related technologies. This semester’s focus is on the theory, design, and implementation of experiences and technologies of virtuality. Toward this end, we shall look at topics including definitions of virtual, virtual reality, augmented reality, alternate reality, hybrid reality, virtual worlds, virtual selves, and more.

Immersive Media

Immersive Media

Cagri Zaman
Teaching, Virtual Reality, Design

In this course, students develop independent projects in immersive media as it relates to architectural design and spatial storytelling. We examine technologies of immersion including VR/AR, web-based experiences, sound installations, and virtual production workflows. We explore the field of immersive media through readings, precedent artwork, and hands-on experience with production tools. We specifically focus on the production pipelines that use real-time simulations and game engines. Students were encouraged to work on the aspects of their studio projects and thesis. The workshop provides a studio environment for the students to collaborate and provide feedback to each other.

Reasonable Perception: Connecting Vision and Language Systems

Reasonable Perception: Connecting Vision and Language Systems

Cagri Zaman
Publication, Artificial Intelligence, Virtual Reality, Research

Human Robot Interaction 2018. ACM.

Understanding explanations of machine perception is an important step towards developing accountable, trustworthy machines. Furthermore, speech and vision are the primary modalities by which humans collect information about the world, but the linking of visual and natural language domains is a relatively new pursuit in computer vision, and it is difficult to test performance in a safe environment. To couple human visual understanding and machine perception, we present an explanatory system for creating a library of possible context-specific actions associated with 3D objects in immersive virtual worlds.

Venice Biennale of Architecture Turkish Pavilion Digital Roundtable

Venice Biennale of Architecture Turkish Pavilion Digital Roundtable

Cagri Zaman
Invited Talk

Digital Roundtable Presentation. June 2018.

Vardiya / the Shift commits a program full of public events to turn Pavilion of Turkey into an open space for temporary accommodation, production and encountering. It is envisioned as a hotspot for workshops, roundtable discussions and informal meetings, welcoming over a hundred international students of architecture, tens of tutors, guest professionals, keynote speakers and visitors while inviting all to a continuous learning and production process throughout the twenty-five weeks of the biennial.

Beyond Light - Archeologia Invisibile Exhibition

Beyond Light - Archeologia Invisibile Exhibition

Cagri Zaman
Exhibition, Augmented Reality

Exhibition at the Museo Egizio, Turin, Italy. 2019-2021

The purpose of the “Invisible Archeology” exhibition was to illustrate the principles, tools, examples and results of the meticulous work of recomposition of information, data and knowledge made possible today by the application of science to other disciplines and, in particular, to the study of the artifacts. What can an object tell of itself? Our senses give us basic information about it, such as its appearance, size, shape, colour, even the traces that human, nature or time have impressed on it. Yet, all this is obviously not enough to reveal the whole story and the life cycle.

MIT.nano Opening Event

MIT.nano Opening Event

Cagri Zaman
Virtual Reality, Exhibition

VR/AR Exhibition. 2018.

MIT Virtual Experience Design Lab was commissioned by MIT nano leadership for an immersive experience to be exhibited during the opening ceremony of MIT nano, the largest and most sophisticated nano science laboratory in US. At the core of the activities in the lab is the use of advanced instruments of measurement and vision in the nanoscale. At this scale, the air should be so clean (class 100 cleanroom means no more than 100 particles in a cubic meter of air is allowed) that without special protocols no one is allowed inside the spaces.

A Hand-Motion Based Tool for Conceptual Model Making in Architecture

A Hand-Motion Based Tool for Conceptual Model Making in Architecture

Cagri Zaman
Thesis, Augmented Reality, Computational Making, Design

SM Thesis. Istanbul Technical University. 2011

Three-dimensional perception facilitated by physical model making supports the creative process and enhances the designer’s ability to evaluate the formal qualities of their design. In our this thesis, we aim to sustain this support in different design environments and study the transfer of the hand motions used in the physical model making processes and use in the digital design environments. We show the results from the observations taken from the conceptual model making processes and analyze the actions involved in these processes as the first step of the research. Depending on this observations we classify the actions according to the main characteristics of the hand motions and propose a recognition schema to be processed in the digital platform. Following the analysis and the classification of the hand motions used in the model making process we aim on translating these hand motions into the digital platform. We discuss the technologies and methods used for hand motion capturing and recognition, and develop a design environment utilizing the hand motions used in the model making processes. We finalize our approach by presenting a set of algorithms for the object deformations conducted with the hand motions. Finally, we test the design environment and discuss the results.

Towards Hands-On Computing in Design

Towards Hands-On Computing in Design

Cagri Zaman
Publication

METU Journal of the Faculty of Architecture. 2011

Early stages of a design process are where the designer experiments with the initial concepts and the perceptual qualities of the envisioned design idea. In our contemporary culture of technology, the acts of design performed in these stages are being transferred into the digital domain more and more as design and production processes merge and gain speed. The visual and spatial integrity of design performances in earlier stages has value for the entire design process and should be sustained in the digital domain as well. The value of hands-on design thinking, or in other words the spatial and visual thinking with materials at hand, comes forth in pedagogical discussions that emphasize design reasoning in the field of arts and design as early as the beginning of the 19th century.

An inquiry into active spatial language using virtual reality

An inquiry into active spatial language using virtual reality

Cagri Zaman
Publication, Conference Talk

International Conference on Spatial Cognition 2018.

In this research we aim to understand the interactions between visual space perception and language in various environmental conditions and tasks. In particular, we investigate spatial concepts in language that are used during active explorations and characterize them in relation to corresponding visuospatial context. We introduce an immersive virtual reality system that allows conducting spatial exploration tasks and recording perceptual and language data. Using this system, we conduct an experiment with 16 subjects, and generate a novel dataset, which includes audio recordings and time coded transcriptions, visual modalities such as RGB and depth, and camera extrinsic for a 10-minute exploration per subject.

Wild Worlds and Tamed Minds

Wild Worlds and Tamed Minds

Cagri Zaman
Lecture, Lecture Series

MIT Architecture Lecture Series. 2016

. The series brings together designers, scientists, media theorists, anthropologists and philosophers to cultivate an interdisciplinary discussion on theories of mind, body and world; their influence on understanding and performing creative activities; and the role of computation as a communicative tool between actively experienced worlds and mental representations.

Hallucination Machine: A Body Centric Model of Space Perception

Hallucination Machine: A Body Centric Model of Space Perception

Nil Tuzcu
Research, Thesis

SM Thesis. Massachusetts Institute of Technology, 2014

In this thesis I present a novel approach to space perception. I provide a body-centric computational model, The Hallucination Machine, that integrates bodily knowledge with senses in a common modality which I call "the sphere of embodiment". Understanding the human experience of space is an important inquiry not only in the context of design and architecture, but in a broad range of scholarly disciplines where humans are the subject of study, whether as biological, social, or cognitive entities.

Six Degrees: A Social Platform to Connect People and Services

Six Degrees: A Social Platform to Connect People and Services

Cagri Zaman
Research

MIT Design Lab, 2014

Hotel lobbies, even when busy, can be places where people—especially those traveling alone—can feel disconnected. Marriott Hotels asked us to think about how digital technologies could create a more social lobby. We began by thinking about how the physical architecture of a space can promote and support social interactions and meaningful experiences using artificial intelligence, the Internet of Things, and so on. Ultimately, we developed Six Degrees, a social network prototype through which guests in the physical space of the lobby can discover how they might be connected to other guests and that supports socializing in events planned by Marriott

LUME Interactive Media Display

LUME Interactive Media Display

Cagri Zaman
Research, Tangible Interfaces, Digital Fabrication

MIT Design Lab, 2013

Interactive LUME Display project explores tangible interactions with digital content through spatially embedded computation in public space at multiple scales from low to high resolution of interactivity. Composed of four interactive touch points, from a mobile app to a large display installation, LUME aims to build the identity for the department of Comparative Media Studies/Writing at Massachusetts Institute of Technology, by increasing awareness and visibility of the different research labs within the department, displaying variety of content, acting as a place finder, and engaging all potential users.

Load More