Chengyuan (CY) Xu

I'm a Research Engineer at Adobe Firefly. I received my Master's in Computer Science and my PhD in Media Arts and Technology at UC Santa Barbara, where I was advised by Prof. Tobias Höllerer at the Four Eyes Lab.

I'm interested in human-AI collaboration, computer vision, and their application to real-world data and problems.

Email  /  Github  /  CV

• A multimodal representation for physical objects
• "In-situ" interactive learning to train AI on physical objects
• Two real-world demo applications on Magic Leap 2

A fantastic journey thanks to the great company I’ve had along the way!

We found evidence that the use of virtual or augmented realities can influence conversation content. Users chatting with avatars in virtual reality made significantly more references to the location or the space, suggesting they tended to perceive conversations as occurring in the agent’s space, whereas the physical AR environment was perhaps more perceived as the user’s space. Conversations with the human avatar improve user recall of the conversation, even though there is no evidence of increased information extracted during the conversation.

My passion project as a new parent! Rather than buying more gear, why not repurpose old tech into a baby monitor that fits my needs? It's more than just a simple monitor - it detects the baby's movements, sends alerts, tracks the baby's sleep, and performs exceptionally in low-light conditions...

Careful exploitation of the tradeoffs in AI precision and recall can harness the complementary strengths in the human-AI collaboration to significantly improve team performance. Naively pairing humans with an AI system designed for autonomous settings could potentially have a negative training effect on the users.

An open-source software toolkit for identifying, inspecting, and editing tiny objects in multi-megapixel HDR images. These tools offer streamlined workflows for analyzing scientific images across many disciplines, such as astronomy, remote sensing, and biomedicine.

Cosmic-CoNN is a generic deep-learning cosmic ray (CR) detector deployed at Las Cumbres Observatory's 24 telescopes around the world. We built a large and diverse ground-based CR dataset and proposed a novel loss function and a neural network optimized for telescope imaging data to train generic CR detection models. Our model achieves a high precision on Las Cumbres imaging data and maintains a consistent performance on new ground-based instruments never used for training.

A worldwide team led by scientists at Las Cumbres Observatory has discovered the first convincing evidence for a new type of stellar explosion -- an electron-capture supernova. While they have been theorized for 40 years, real-world examples have been elusive. I had the pleasure to provide supplementary evidence to help rule out the presence of cosmic-ray hits at or around the progenitor site to contribute to closing the 40-year-old theoretical loop.

We produced a new dataset and a detection model to identify 15 types of ocean-bound river wastes like plastic bottles or bags, foam fragments, and other inorganic wastes in complex trash wheel images. The project aims to support more efficient and more accurate data collection for a greater understanding of the types and sources of river waste and to ultimately turn off the tap of plastic and other solid waste pollution into the ocean.

We propose a novel generative adversarial network (GAN) architecture to achieve spatially and temporally coherent video style transfers. Started in 2018, this work was one of the first deep learning-based methods for video style transfers.

motionLight is a playful interactive visual audio installation inspired by Jim Campbell’s low resolution artworks. It reads camera and microphone signals for lighting and motion changes to generate six modes of temporal and spatial interpolations.

Before I dived into convolutional neural networks (CNNs), I was a cameraman covering China for CNN and BBC.

Website source code from Jon Barron.