Supported By: National Endowment for the Humanities (NEH)

Co-PIs: Dr. Cagri Hakan Zaman & Prof. Caroline Jones

Research Associate: Deniz Tortum

Research Assistants: Rohit Sanatani, Adriana Giorgis, Lauren Gideonse

A significant part of history is stored in newsreels, educational, documentary, fiction films, and even home movies. Moving images constitute one of the richest sources of information for a wide range of disciplines grounded in visual literacy including art and architectural history, archeology and museology, film and media studies, as well as history in general. This vivid archive is poised to reveal significant historical and creative information through tools that can be developed and combined to enhance scholarly research, teaching, and public programming in the humanities. Previously, extracting information from moving images has been challenging and time consuming, requiring historians and film scholars to access footage by manually reviewing sequences over and over to parse the setting, the rituals, camera angle, narratives, and the material cultures involved. Now, developments in computer vision and spatial analysis technologies have opened up exciting possibilities for these scholarly processes, with direct implications for improved public access and future translational tools for disabled communities.

The “latent archive” that has always been embedded in moving images can now be captured via machine-enabled analysis: locating the urban or architectural setting, producing 3D spatial reconstructions, and allowing fine-grained examination of point-of-view and shot sequence. We aim to combine and develop technologies to produce a single prototype platform for translating moving image archives into spatial formats that can be examined in immersive media (i.e. Virtual & Augmented Reality). By the end of phase 1 of the grant period, we intend to post publicly available software, along with moving image training sets on a publicly available website, that will help humanists such as media theorists, historians, and urban planners to utilize moving images’ spatialized 3D historical information. Latent Archive will open exciting new opportunities for studying material history and producing new spatially informed narratives about the past.

Translating moving images into 3D Spaces

In this Phase I Digital Humanities Advancement project, we introduce a tool for extracting spatial information from moving image archives and examining them in an immersive virtual reality environment. Our main goal is to evaluate various techniques for spatializing the moving images and develop a scalable workflow for future uses. There are three main challenges addressed in this phase:

Image Processing: What are the automated image processing techniques that can improve the quality of spatial reconstruction? The techniques we shall consider are image stabilization, image super-resolution, colorization, and lens correction.

Spatial Reconstruction: Can we reconstruct environments and objects found in moving image archives using state-of-the-art spatialization methodologies? The methodologies we shall consider are photogrammetry, structure from motion (SfM), monocular depth estimation using deep neural networks (DNNs), 2D & 3D semantic scene segmentation using those neural nets.

Immersive Media Presentations: What are the benefits of examining spatialized content in VR/AR? We will study interaction mechanisms and narrative tools in VR development platforms, and begin to design the early features of the platform. The features we currently consider useful are (1) navigating within reconstructed environments, (2) alignment of image frames to 3D models, (3) adding textual meta-data to reconstructed scenes and objects, and (4) exporting walk-through animations. We will revise our feature list based on user tests and meetings with the advisory board.