All in Augmented Reality

Following the Master’s Hands: Capturing Piano Performances for Mixed Reality Piano Learning Applications

ACM Human Factors in Computing System (CHI) 2023.

Piano learning applications in Mixed Reality (MR) are a promising substitute for physical instruction when a piano teacher is absent. Existing piano learning applications that use visual indicators to highlight the notes to be played on the keyboard or employ video projections of a pianist provide minimal guidance on how the learner should execute hand movements to develop their technique in performance and prevent injuries. To address this gap, we developed an immersive first-person piano learning experience that uses a library of targeted visualizations of the teacher’s hands and 3D traces of hand movements in MR.

NavigAid: AI-Driven Orientation and Mobility System for the Blind

National Science Foundation SBIR Phase II Project. 2020

NavigAid is an AI-driven mobile Orientation and Mobility (O&M) system, which provides contextually relevant, task-driven solutions to problems such as finding objects, identifying paths of ingress and egress, and understanding the layout of an environment. NavigAid is enabled by our core technical innovation, Ally Networks, which represents a novel neural network architecture that is capable of extracting semantically and functionally relevant spatial features from images, which help to create a human-like understanding of physical environments.

Virtuality & Presence

This course aims to help students invent and analyze new forms of extended reality (XR) experiences, computer-based art, gaming, social media, interactive narrative, and related technologies. This semester’s focus is on the theory, design, and implementation of experiences and technologies of virtuality. Toward this end, we shall look at topics including definitions of virtual, virtual reality, augmented reality, alternate reality, hybrid reality, virtual worlds, virtual selves, and more.

Beyond Light - Archeologia Invisibile Exhibition

Exhibition at the Museo Egizio, Turin, Italy. 2019-2021

The purpose of the “Invisible Archeology” exhibition was to illustrate the principles, tools, examples and results of the meticulous work of recomposition of information, data and knowledge made possible today by the application of science to other disciplines and, in particular, to the study of the artifacts. What can an object tell of itself? Our senses give us basic information about it, such as its appearance, size, shape, colour, even the traces that human, nature or time have impressed on it. Yet, all this is obviously not enough to reveal the whole story and the life cycle.

A Hand-Motion Based Tool for Conceptual Model Making in Architecture

SM Thesis. Istanbul Technical University. 2011

Three-dimensional perception facilitated by physical model making supports the creative process and enhances the designer’s ability to evaluate the formal qualities of their design. In our this thesis, we aim to sustain this support in different design environments and study the transfer of the hand motions used in the physical model making processes and use in the digital design environments. We show the results from the observations taken from the conceptual model making processes and analyze the actions involved in these processes as the first step of the research. Depending on this observations we classify the actions according to the main characteristics of the hand motions and propose a recognition schema to be processed in the digital platform. Following the analysis and the classification of the hand motions used in the model making process we aim on translating these hand motions into the digital platform. We discuss the technologies and methods used for hand motion capturing and recognition, and develop a design environment utilizing the hand motions used in the model making processes. We finalize our approach by presenting a set of algorithms for the object deformations conducted with the hand motions. Finally, we test the design environment and discuss the results.

Design Tools of The Future Hackathon

Hackathon Organization as part of ACADIA 2017. MIT Media Lab.

How does a VR tool work in the future to help an architect presenting a design of a new public school to the local community? What is an AR tool that real estate brokers can use to help promote their properties to home hunters? What is a future MR tool on the construction site that helps building faster, safer and more economically?

Focusing on challenges such as these in the real design world context, including but not limited to architectural, landscape and product design as well as urbanism, real estate and construction, this hackathon fostered an interdisciplinary environment for proposing strong innovative solutions to the future