Essentially, it is useful for comparing the heading of 2 vectors. What it does, it returns scalar(regular number) in range of 1 to -1, where 1 means that vectors are headed same way and -1 means the opposite. That means you can get this dot product between saved finger direction and current finger direction and check how similar the direction is.
Here is an example:
Vector savedDir = customGesture.savedDir; // This is how I'd imagine it could look
Vector currentDir = bone.direction();
float heading = savedDir.dot(currentDir);
if (heading > 0.8) // so, if heading is more than 80% accurate
// Mark this bone as matching
You can check this page for information on finger bones: link.
I suppose you should make custom class for the custom gestures and have other custom class for it's bones as well, so you can somehow mark matching bones. Well I hope you have a better idea now, of how it could be done.