Watch the relationship between your hand actions and the combo of sphereRadius and sphereCenter. When a fist is made, the sphereRadius is relatively small compared to open hand and closed or open fingers. For the circumstances when the sphereRadius is small with open hand, the sphereCenter.z is significantly more forward.
There are more correlations that are less significant such as the sphereCenter.y response but these first two correlations may be enough and maybe with a low-pass filter applied. Also, I imagine different hand sizes and conditions may affect the exact values of correlation so calibrating may be required to have high consistency.
For example, the test pseudocode right now that is performing quite well for my hand/environment:
First transform the sphere center to the hand's frame of reference (see documentation).
if (leapHand.fingers.count == 0 && leapHand.sphereRadius < 75 && transformedSphereCenter.z > -20)
print("Closed Fist);
else
print("Open Hand);
Please, post any findings on variations to values or further correlations that help!