The notion of a swipe works best in a physical tablet based interaction. There, we have a clear binary of touching the glass that happens before we even begin detecting motion for a swipe.
With hands in VR, our hands are usually in motion already. So knowing how, when and where to begin detecting the swipe are questions we need to define for a successful interaction. This could be done by asking the user to swipe through a visible detection volume or other type of visual affordance.
For the simple interaction you're describing, a reliable way to do this switching would be with a 3D button like we provide in our Interaction Engine examples in \Assets\LeapMotion\Modules\InteractionEngine\Examples\2. Basic UI