A classical robotics pipeline built from scratch on a real 7-DoF arm. HSV perception, Jacobian IK, and RRTConnect motion planning working together to pick and place objects autonomously.
The goal was to build a pick and place system on a real 7-DoF Franka Panda arm using nothing but classical robotics tools. No end-to-end learning, no pretrained policies. Just perception, planning, and control working together cleanly.
The pipeline has three stages that are deliberately decoupled. Perception finds the object. Planning computes a collision-free path to it. The IK controller executes that path on the real joints. Each stage is replaceable without touching the others.
Separating planning from execution was the key design decision. When something breaks you know exactly which stage to look at. That matters on real hardware more than anything else.
I chose HSV segmentation deliberately. It is simple, transparent, and every failure mode is visible — you can literally see the mask and know what went wrong. For this task it is the right tool.
The centroid computation uses image moments, which gives sub-pixel accuracy. The depth reading at that pixel plus the camera intrinsics gives a 3D point in camera frame. ROS 2 TF handles the rest.
The Franka Panda has 7 joints, so the configuration space is 7-dimensional. RRTConnect builds two trees simultaneously, one from the current joint configuration and one from the goal. They grow toward each other with random samples until a bridge is found.
The bidirectional approach converges significantly faster than a single tree on high-DoF arms, which matters when the robot is waiting to execute. The resulting path is time-parameterised for smooth velocity profiles before it goes to the IK controller.
Forward kinematics gives you the end-effector pose from joint angles. IK inverts that. For a 7-DoF arm the problem is redundant — there are infinitely many joint configurations that reach the same pose — so I use the damped pseudoinverse Jacobian to find the minimum-norm solution.
The damping factor λ keeps the solution stable near singularities where the plain pseudoinverse would blow up. Pinocchio provides the analytical Jacobian at each configuration so the loop runs fast enough for real-time control.
After getting the classical pipeline working I also trained a PPO-based policy in MuJoCo to see how a learned approach compared on the same task. The classical system is more interpretable and easier to debug on hardware. The RL policy generalises better to object position variation once trained. Both have their place — I just found the planning work more interesting to build.