I was experiencing this as well, and recently discovered that it was a USB bandwidth issue. I found this warning reported in the Leap software log:
"Low Data Rate. Connect directly to a computer (not through a USB hub or the Oculus Rift DK2 USB)."
I had been using the HMD, Rift camera, and Leap on the same USB hub which was causing the Leap to lag. When I plugged the Leap into its own dedicated USB 3 port my framerate issues disappeared.