Until now, object detection and tracking system are widely utilized in many fields. Quite impressive attention has been paid for real-time object detection and tracking. However, it‘s always a challenge to perform the task with fast inference while maintain the accuracy. Although Graphics Processing Units(GPUs) are more efficient and stable, they require large power, energy consumption, and have large computational load problems. Recently, Xilinx Field Programmable Gate Array(FPGA) and PYNQ DPU module [1-3] provide us an efficient way to reduce such cost by hard-ware acceleration of the model and overcome this problem. In this work, we propose a Unmanned Aerial Vehicle(UAV) tracking system on ultra96 v2 PYNQ board based on the combination of object detection deep learning model and mathematical algorithm such as Kalman filter. The Kalman filter provides us a fast but preliminary estimation of UAV position while the object detection models further localize the UAV. Our implementation meets 28fps on the Ultra96 v2 board and nearly achieve the real-time detection.