Old thread, but to complete @Acienty's answer, I believe the tag in question is the RSN Information tag which I don't see present for WEP. Furthermore, in the Auth Key management (AKM) type: XXX You can see if it is WPA or WPA2 (PSK).
In other words:
-WPE ONLY : wlan.fc.type_subtype == 0x0008 && wlan.fixed.capabilities.privacy == 1 && !wlan.tag.number == 48 && !wlan.wfa.ie.type == 0x01
wlan.fc.type_subtype == 0x0008
: Looks only for beacons packets
wlan.fixed.capabilities.privacy == 1
: Looks for encrypted connections (WEP or WPA but no open wifis)
!wlan.tag.number == 48
: Removes packets with RSN information tags (I believe that those with WEP never have this tag - might be wrong)
!wlan.wfa.ie.type == 0x01
: Removes packets Vendor specific WAP (I don't know if this is universal, but again, those with WEP don't have this)
-WPA ONLY : wlan.fc.type_subtype == 0x0008 && wlan.fixed.capabilities.privacy == 1 && ((wlan.tag.number == 48 ) || (wlan.wfa.ie.type == 0x01))
- same as above, but includes RSN information tag (WPA & WPA2) or vendor WAP on the already filtered subtypes (mind the parentheses!)
-OPEN WIFI : wlan.fc.type_subtype == 0x0008 && wlan.fixed.capabilities.privacy == 0
I believe this is exhaustive, but I might be overlooking specific cases. I'll come back to this post if I have any updates for more precise filter expressions.