Aerial Embodied Intelligence and Vision-Language Models (VLMs) on a Drone-Manipulator Platform
L1 adaptive control for an aerial manipulator (3–4 cm average manipulation error), PPO sim-to-real correction, 400 Hz control from a Jetson Orin, and Qwen2.5-VL-driven autonomous grasping.
Period: 2025/10 – present · University of Toronto Institute for Aerospace Studies
Ongoing project. We have made significant improvements in manipulation. Vision-based object localization and LLM control have both been tested, and we are now running experiments to close the remaining gap to fully autonomous grasping in flight.
- Led a four-person team (3 Ph.D. students, 1 master’s student) to design interactive grasping and trajectory-tracking algorithms for a drone-manipulator platform. I was personally responsible for the L1 adaptive control design, the debugging and calibration of the manipulator arm, and the overall system architecture, including the communication links and the software system design. By analyzing the multibody dynamics equations of the manipulator platform, proposed an aerial embodied intelligence framework based on L1 adaptive control, filling the gap in theoretical stability proofs for aerial intelligent manipulator platforms. Achieved an average dynamic object-manipulation error of 3–4 cm in flight experiments — the highest accuracy for object manipulation with aerial serial manipulators reported in the existing literature. Implemented VLM capabilities for the aerial embodied platform based on Qwen2.5-VL-7B (Tongyi Qianwen), enabling the platform to autonomously recognize target objects and invoke the low-level visual localization algorithm for pre-grasp positioning; implemented natural-language control of the drone based on Qwen2.5-7B.
- To address parameter uncertainties in the flight system — such as the effects of aerodynamic drag and propeller downwash (the downward airflow generated by the rotors) on flight — used Proximal Policy Optimization (PPO) reinforcement learning combined with domain randomization to narrow the Sim2Real (simulation-to-reality transfer) gap. Trained the PPO control-correction algorithm in NVIDIA Isaac Sim 6.0 and Isaac Lab 3.0.
- Deployed the embodied-intelligence control framework on an NVIDIA Jetson Orin onboard computer, with high-speed, low-latency UDP communication over Ethernet between the onboard computer and the flight control unit. Achieved, for the first time, direct control of the drone’s propellers and the manipulator by the NVIDIA Jetson Orin at 400 Hz, greatly improving manipulation and tracking accuracy.
- Using XM430-W350 joint actuators, designed FOC (field-oriented control) current-loop control, improving torque actuation accuracy by about 20% in torque-control mode and laying the foundation for improving overall object-manipulation accuracy.
- Using a RealSense depth camera and running YOLOv8 with Ultralytics, combined RGB object detection with depth information to achieve autonomous target recognition and localization, with centimeter-level localization accuracy for static targets.
- Deployed the Qwen2.5-VL-7B large model locally on a ground-station computer equipped with an RTX 4070. The drone connects to the ground control station over a P2P network, which obtains the motion states of the drone and the manipulator in real time. By recognizing objects in camera images and invoking the precise localization results of the RealSense depth-camera module, the large model enables a fully autonomous environment-interaction system for the aerial embodied platform. The large model invokes the embodied control framework proposed in this research to command the drone to grasp and place objects, realizing a closed-loop physical-AI expert system centered on a large model.
Sim-to-real with Isaac Sim
- Measured data. Motor speed and thrust are measured on a test stand and fitted against throttle. Together with the CAD model, they set the parameters of the simulation.
- High-fidelity simulation. The CAD design is exported from Onshape to USD and imported into NVIDIA Isaac Sim with the PhysX engine. A ROS 2 interface reads the ground truth and sets the joint efforts, so the same control code runs in simulation and on the vehicle.
- Parallel RL training. A PPO control-correction policy is trained in Isaac Lab across many parallel environments, with domain randomization over effects that are hard to model, such as aerodynamic drag and propeller downwash, to narrow the sim-to-real gap.
- Experimental validation. The controller and the policy fly on the real platform, and the flight data feeds the next iteration.
Flight experiments
The flight at the top of this page shows the platform in the indoor flight arena. Key results from the experiments:
- 3–4 cm average error in dynamic object manipulation, using the L1 adaptive control framework.
- 400 Hz control from the NVIDIA Jetson Orin, which drives the propellers and the arm directly through low-latency UDP over Ethernet to the flight control unit.
- About 20% better torque accuracy in the XM430-W350 joints with field-oriented (FOC) current-loop control.
- Centimeter-level localization of static targets from the RealSense depth camera and YOLOv8.
LLM and VLM in the loop
natural language
language to task
finds the target in the image
3D target position
400 Hz on Jetson Orin
- Language. Qwen2.5-7B turns a natural-language instruction into a task for the drone.
- Vision. Qwen2.5-VL-7B recognizes the target object in the camera image.
- Localization. The RealSense depth camera and YOLOv8 give the target’s 3D position for pre-grasp positioning.
- Control. The embodied control framework flies the drone and the arm to grasp the object and place it.
Qwen2.5-VL-7B runs locally on a ground-station computer with an RTX 4070. The drone connects to the ground station over a peer-to-peer network that streams the drone and arm states in real time, which closes the loop around the large model.
Status. Vision-based object localization and LLM control have both been tested; experiments to connect them into fully autonomous grasping in flight are under way.
The team