Segmentation applications in scientific imaging domains such as seismic interpretation require large quantities of labeled data annotated by expert interpreters, which is a costly and time-consuming process. Where existing works to minimize dependence on labeled data assume the data annotation process to already be completed, active learning—a field of machine learning—works by selecting the most important training samples for the interpreter to annotate in real time simultaneously with the training of the interpretation model itself, thereby reducing cost and effort to produce annotated training samples while minimizing the negative impact on performance. We develop a unique and first-of-a-kind active learning framework for seismic facies interpretation using the manifold learning properties of deep autoencoders.

Accurate interpretation of visual data for relevant information forms an important component of many real-world applications such as medical disease diagnosis, geological hazard assessment, hydrocarbon exploration, etc. Producing fine-grained annotations on computational images is an expensive, laborious, and time-consuming process. The human brain is wired to selectively focus its attention on certain aspects of the visual scene. This perception mechanism is driven both by low-level signal cues, such as changes in color, contrast, intensity, shapes etc., as well as high-level cognitive factors such as one’s prior knowledge, goals, expectations, and constraints with respect to the task at hand. By leveraging and modeling knowledge of this attention process into the training of deep neural networks, we demonstrate the network to perform significantly better even with sparse annotations.

Large Multimodal Foundation Models for Visual Question Answering (VQA) have made large strides in understanding natural image scenes and videos. In particular, they have come a long way when it comes to perceiving low level details about a given scene, such as the type and number of objects, their colors and sizes etc. However, deploying these models in critical applications such as robotics-assisted surgery would require that they pass a much higher threshold of reasoning. In particular, they would need to correctly respond to user questions under degraded image conditions, be able to correctly guess the previous and next states of surgical tools, states etc., decline to answer false-premise/ambiguous questions etc. We develop a first of its kind Surgical VQA benchmark containing image-question pairs from a variety of surgical procedures explicitly testing VLMs for their robustness, temporal, counterfactual, and fine-grained spatial reasoning capabilities.   

Foundation vision models for interactive segmentation like the Segment Anything Model (SAM) have demonstrated impressive zero-shot generalization across a range of open-world scenes. However, their efficacy frequently degrades when deployed in highly specialized scientific domains characterized by unique visual structures and noise profiles. By incorporating geophysical tools like seismic attributes as inputs to the model, we benchmark its segmentation accuracy over a range of geological features such as channels, facies, and salt domes. We additionally investigate the tuning of input prompts by the user by augmenting the initially sparse point prompts with the model's internal activations to produce a more refined output. Our investigations reveal that domain-guided input transformations along with efficient prompting strategies can significantly improve the model's zero-shot generalization for seismic segmentation applications. 

The process of inferring subsurface material properties from indirect seismic measurements, called seismic inversion, is an important geophysical application. The problem is by nature ill-posed and computationally demanding. While deep neural networks have traditionally been used to learn the inverse operator from a limited set of labeled training examples, they have suffered from poor generalization. We systematically design novel architectures and training paradigms to improve generalization performance in limited label scenarios.