Unreal Gestures
C++ Unreal Engine 5.6 System for easily recording and detecting hand gestures, when using hand tracking in VR.
I implemented a system in Unreal Engine 5.6, that allows you to record gestures, and gesture sequences. These can then be detected, by simply subscribing to the respective event. Registration can be done automatically by using Unreal Engine's reflection API. You can create sequences/chains and chords, for example, only activate if both of these gestures are active, or if this gesture was activated N amount of seconds after another gesture. You can also choose, which hand can activate the gesture, for example only the right hand. If the gesture was recorded with the left hand, but you want the right hand to activate it, it will automatically translate the values to the correct space.
Implementation
There is a recorder actor, which you should place in a scene during development. Then once you load into this scene, you can hold your hand in the pose you want, and press a button. This will record the current gesture into a data asset, with a randomized name, using a guuid and the current date, to avoid name collisions, and allow you to recognize it more easily. Once you have this data asset, you can simply select it inside of the different provided methods, and they will compare the recorded data to the current data. The data asset is immutable, and can be safely used across many actors/components, without any problems. During the recording it measures the angles of each joint in quaternions, and it takes note of things such as which hand recorded it, so it can later translate it to the right handedness if needed. Quaternions were choses, since they should have less hand-to-hand variance than for example measuring the distance between joints, that would vary a lot between people. In the data asset you can also set blacklists, these joints will not be considered in the evaluation. You can also select a rotation error margin, this is how much the current value may deviate from the recorded, in order to still count as active.
There is an evaluator component. Here you can provide a list of gestures it should evaluate. You can manually call the evaluate method, or have it automatically evaluate it every frame. Callbacks can be registered explicitly, or auto-registered. If you have the correct name and signatured on an event, and have auto registration enabled, during startup the evaluator component, will use reflection to automatically register the applicable events to their respective gesture. There are also standalone methods you can use, if you would like to avoid using the provided component. Automatic registration requires the event to have "GestureEvent" and the name of the Gesture in the event name. When there are no bound events to a gesture, the gesture will not be evaluated. You can swap out the solver specifically, by selecting a default solver class, the default instance of this will be taken, and is what determines if a gesture is held or not. There's also the option to enable evaluation of relative rotations, what this entails is that the gesture won't be reliant on the rotation of the hand, so for example a thumbs up, would still be evaluated as true if it is a thumbs down. Evaluation of a Gesture is O(N), with N being the amount of joints. Future plans could include making the evaluation centralized, so each gesture is only evaluated once, but currently it is done per component.
You can have callbacks called only when the gesture has been held for a certain amount of time through the AsyncEvaluateGesture.
Here is an example of the gesture sequences. The max interval is the maximum amount of time that can be beteween the steps. For example if the second step takes longer than 1 second to be held, the chain will exit. There are two events available, one when the full sequence is completed, and one that gets activated each time the sequence advances.
