DeepFly3D: The Deep-Learning Way to Design Fly-Like Robots
October 11, 2019 | EPFLEstimated reading time: 2 minutes
EPFL scientists have developed a deep-learning based motion-capture software that uses multiple camera views to model the movements of a fly in three dimensions. The ultimate aim is to use this knowledge to design fly-like robots.
“Just think about what a fly can do,” says Professor Pavan Ramdya, whose lab at EPFL’s Brain Mind Institute, with the lab of Professor Pascal Fua at EPFL’s Institute for Computer Science, led the study. “A fly can climb across terrain that a wheeled robot would not be able to.”
Flies aren’t exactly endearing to humans. We rightly associate them with less-than-appetizing experiences in our daily lives. But there is an unexpected path to redemption: Robots. It turns out that flies have some features and abilities that can inform a new design for robotic systems.
“Unlike most vertebrates, flies can climb nearly any terrain,” says Ramdya. “They can stick to walls and ceilings because they have adhesive pads and claws on the tips of their legs. This allows them to basically go anywhere. That's interesting also because if you can rest on any surface, you can manage your energy expenditure by waiting for the right moment to act.”
It was this vision of extracting the principles that govern fly behavior to inform the design of robots that drove the development of DeepFly3D, a motion-capture system for the fly Drosophila melanogaster, a model organism that is nearly ubiquitously used across biology.
In Ramdya’s experimental setup, a fly walks on top of a tiny floating ball—like a miniature treadmill—while seven cameras record its every movement. The fly’s top side is glued onto an unmovable stage so that it always stays in place while walking on the ball. Nevertheless, the fly “believes” that it is moving freely.
The collected camera images are then processed by DeepFly3D, a deep-learning software developed by Semih Günel, a PhD student working with both Ramdya’s and Fua’s labs. “This is a fine example of where an interdisciplinary collaboration was necessary and transformative,” says Ramdya. “By leveraging computer science and neuroscience, we’ve tackled a long-standing challenge.”
Different poses of the fruit fly Drosophila melanogaster are captured by multiple cameras and processed with the DeepFly3D software. Credit: P. Ramdya, EPFL.
What’s special about DeepFly3D is that is can infer the 3D pose of the fly—or even other animals—meaning that it can automatically predict and make behavioral measurements at unprecedented resolution for a variety of biological applications. The software doesn’t need to be calibrated manually and it uses camera images to automatically detect and correct any errors it makes in its calculations of the fly’s pose. Finally, it also uses active learning to improve its own performance.
DeepFly3D opens up a way to efficiently and accurately model the movements, poses, and joint angles of a fruit fly in three dimensions. This may inspire a standard way to automatically model 3D pose in other organisms as well.
“The fly, as a model organism, balances tractability and complexity very well,” says Ramdya. “If we learn how it does what it does, we can have important impact on robotics and medicine and, perhaps most importantly, we can gain these insights in a relatively short period of time.”
Suggested Items
Cadence, TSMC Collaborate on Wide-Ranging Innovations to Transform System and Semiconductor Design
04/25/2024 | Cadence Design SystemsCadence Design Systems, Inc. and TSMC have extended their longstanding collaboration by announcing a broad range of innovative technology advancements to accelerate design, including developments ranging from 3D-IC and advanced process nodes to design IP and photonics.
Ansys, TSMC Enable a Multiphysics Platform for Optics and Photonics, Addressing Needs of AI, HPC Silicon Systems
04/25/2024 | PRNewswireAnsys announced a collaboration with TSMC on multiphysics software for TSMC's Compact Universal Photonic Engines (COUPE). COUPE is a cutting-edge Silicon Photonics (SiPh) integration system and Co-Packaged Optics platform that mitigates coupling loss while significantly accelerating chip-to-chip and machine-to-machine communication.
Siemens’ Breakthrough Veloce CS Transforms Emulation and Prototyping with Three Novel Products
04/24/2024 | Siemens Digital Industries SoftwareSiemens Digital Industries Software launched the Veloce™ CS hardware-assisted verification and validation system. In a first for the EDA (Electronic Design Automation) industry, Veloce CS incorporates hardware emulation, enterprise prototyping and software prototyping and is built on two highly advanced integrated circuits (ICs) – Siemens’ new, purpose-built Crystal accelerator chip for emulation and the AMD Versal™ Premium VP1902 FPGA adaptive SoC (System-on-a-chip) for enterprise and software prototyping.
Listen Up! The Intricacies of PCB Drilling Detailed in New Podcast Episode
04/25/2024 | I-Connect007In episode 5 of the podcast series, On the Line With: Designing for Reality, Nolan Johnson and Matt Stevenson continue down the manufacturing process, this time focusing on the post-lamination drilling process for PCBs. Matt and Nolan delve into the intricacies of the PCB drilling process, highlighting the importance of hole quality, drill parameters, and design optimization to ensure smooth manufacturing. The conversation covers topics such as drill bit sizes, aspect ratios, vias, challenges in drilling, and ways to enhance efficiency in the drilling department.
Elevating PCB Design Engineering With IPC Programs
04/24/2024 | Cory Blaylock, IPCIn a monumental stride for the electronics manufacturing industry, IPC has successfully championed the recognition of the PCB Design Engineer as an official occupation by the U.S. Department of Labor (DOL). This pivotal achievement not only underscores the critical role of PCB design engineers within the technology landscape, but also marks the beginning of a transformative journey toward nurturing a robust, skilled workforce ready to propel our industry into the future.