VR App for Dice simulation
A VR app for Dice that takes accelerometer data and simulates shaking action
I played this classic Indian board game Ludo with my nephew and niece and complained about losing the die too often. This inspired me to create a virtual dice that the user can physically manipulate and it won't be based on some pseudorandom number generation but as close to true randomness that one can achieve. As research, I looked up some apps on Google Play Store and found none of them do what I wanted. Some implementations just applied an impulse force to the die/dice in the scene and they jumped and gave a number. Some others employed animation and pseudorandom number generation to show the particular face.
These apps were not good enough and as I pondered more about the idea, I thought, why hasn't anyone done this yet? An Android phone has an accelerometer sensor with fusion algorithms that combines data from multiple sources to filter out the data. We can figure out the forces being applied to the mobile and use relativistic frames of reference to manipulate the die. We also have physics and collision detection, which can contain the die within an enclosed space. It's just a matter of putting everything together.
Now that I had the basic layout ready, I had to find the right ingredients to make it work. I had to find a lightweight game engine that runs on Android and possibly supports other platforms. Depending on my skillset, I looked for a C++-based game engine and came across Cocos2dx. It is open-source, comes with 2D/3D physics integration, multimedia support, and a ton of other game-ready features like networking. It can deploy on Android, iOS, Linux, Mac OSX and Windows platforms, which is just what I needed. There is Cocos Creator as well, which is more like Unity and has a graphical interface and script-based development system. But I don't particularly appreciate using those and I'll illustrate why later.
Let's imagine that you are holding a transparent cuboid, a bounding box that contains a die. If we shake the cuboid and introduce acceleration, the die will not experience any force until it hits the cuboid walls and then proceeds to bounce off in a random direction. Another way to see this is to attach a camera to one of the cuboid faces and observe the die's motion. It will look as if an opposite force, with respect to the bounding box, is being applied to the die and this is the essential crux of this app/game. Of course, we will cheat a lot here and not model things precisely because we don't have to. Simple approximations do the trick just fine and can be parameterized well enough.
I'll componentize the entire app into atomic elements, which will help us achieve the big picture. Itemization helps with the planning and timely execution of the whole project. One can prioritize the items and build them independently of other units and submit a piece of code or model so that the game can slowly take shape.
The game has a scene graph to which objects are attached. Here, we need a static floor surrounded by invisible walls and a dynamic die object. And the GUI objects for insertion, changing the color of the die, adding SFX and particle effects.
3D assets require a mesh editor which can export the mesh definition out in some recognized format. Cocos2dx supports *.obj/mtl definitions and a proprietary readable/binary format *.c3t/c3b. I used Blender to create the 3D meshes and exported those in *.obj format. I had to overwrite the material to get the desired effect that I wanted. For simpler 2D models, I edited a sample *.c3t file.
It is a cube primitive from the point of view of the physics engine. We can add a more refined and realistic-looking low-poly mesh for the cube. I designed the object file in blender by differencing a low-poly icosphere with a cube. It chips away and rounds the corners. I edited the mesh further and beveled the edges to make it look even more natural. Since the mesh became more complicated, I used the Smart UV Unwrap to generate a guide for texture mapping and used Inkscape to add white dots and transparency.
The transparency is used to apply a color to the object via a GLSL uniform vec3. This was the best way to add custom colors and now users can add a new die with a color of their choice.
A simple quad or two triangles can model this. We can even write the mesh down ourselves and make counter-clockwise face assignments to be rendered as a front face. This object is the best sample to illustrate and learn about UV Mapping and its working. Two extreme corners should be assigned the UV coordinates (0, 0) and (1, 1) and when the quad is drawn, we should see the entire image drawn onto the quad.
We can use a large image with a high memory footprint to add the texture or go with a seamless/repeating texture that $$I(x+N*1, y+M*1) = I(x, y); \{0 \leq x, y \leq 1\}; N,M \in \mathbb{Z};$$ repeats itself. The repeating texture is a good strategy as it lets us use lesser memory and faster loading times. In this particular case, I looked for some wooden floor-like textures and found this one. I had to write a shader to make this work and I will cover it later.
This object was the simplest to manufacture as this only needs 8 points and six quads to generate. I built the first iteration by using corners of a cuboid(red box), which did a great job of containing the die object. However, it looked pretty weird because of perspective projection which makes parallel lines non-parallel. The die behavior was extremely unnatural around the edges and it looked strange.
I solved this problem using the camera frustum to give me the exact boundaries and a more natural look and feel. The frustum is a collection of 6 planes where we can abandon the near and far planes and use plane-plane intersection equations to figure out four extreme rays which project out of the camera(yellow lines). Then we put two more planes(green squares) right in front of the camera depending on the size of the containment needed and intersect these four rays with these two planes at 8 points to get the corners and use the corners to create six quads of containment.
The user interface is needed to set specific parameters like particle effects, the color of the die, add/remove dice. Also, some general usage guidelines and "how it works" have to be outlined.
I used a repeating crumpled paper texture as the UI background. The colors matched with the floor texture and it takes lesser space and I picked all the colors for the icons based on this background texture.
When the menu dropped, it looked very flat and boring as shown above, so I added a new quad with another texture on top of it and added a known color to add transparency for the top layer with a shader. I also had to limit the repetations of the top layer texture.
I created icons for Close, Add, Refresh. I also wanted the flexibility of changing the color of the dice and generated red, green and blue die icons. Also, I imagined some cool paricle effects when the dice collided with each other or the walls. I thought explosions and lightning effects might look cool and created icons for those. For the usage, I created an icon which shows how to hold the phone and the shaking motion which was required to get the best results.
Cocos2dx doesn't have good support for a layout, so I came up with my own which could be resized depending upon device resolutiuon. The plan was to keep 3 menu items with the Credits section on the left and the Main Menu in the center with the Usage being on the right. I had considered this before and designed my icons based on this layout. Essentially, rects were used to divide the usable screen space and the image was anchored at the center. I used a State pattern to handle different menu states in which the menu can only come down upon a downward swipe. In this active state, the user can swipe left, right or upwards, the left swipe shows the Credits and right swipe shows the Usage. The user has to go back to the main menu to enter the inactive state by swiping upwards.
A static menu looks boring, so I added new classes and implemented zoom and color shift actions. When the user touches a menu item, it will enlarge the image. For the color change die item, I created a special item which holds multiple images(3 supported as of now) and cycles through the opacity of each one with actions that created fine transitions. For particle effects, custom MenuItems were created and logic was put in place so that only one or none of the particle effects can be applied. The choice has to be communicated to all the active die objects via a weak implementation of the observer pattern.
The "game" doesn't have any complicated levels. Some might argue that this is not even a game as it lacks the basic elements like a scoring system and some form of objective. It's more of a VR app than a game which was built using a game engine. It has a utility though and it allows users to play other games when they lose or can't locate the real-world randomizer. This game didn't have too many design based opportunities and I've only used the State pattern for the Menu and a weak version of the Observer pattern to communicate the particle change effect.
The literal driving force for this app is the accelerometer data. In Cocos2dx, we have builtin platform dependent data fetching schemes and it returns the amount of forces acting on the device. Gravity is included by default so that, $$A = G + LA$$ where, A = total acceleration, G = acceleration due to gravity and LA = linear acceleration experienced by the device when the user moves it around.
For my application, I wanted the two forces to be separate, as I wanted the die to fall in the direction of gravity as the user rotates the device. I also wanted the die to react to the shaking movements as naturally as possible and the linear acceleration component would help out with that regard. We could use a low-pass filter to filter the gravity component out and it's a very general solution. However, Android devices return a filtered version of these values by modelling them as software sensors. It uses sensor fusion and multiple measurements from Accelerometer and the Gyroscope to compensate for the different types of noise in both sensors and return data which is very close to the actual truth.
Cocos2dx does this via the Cocos2dxAccelerometer java object and I created a new one called Cocos2dxDecomposedAccelerometer in which I generate 2 events, one for the Gravity sensor and one for the Linear Accelerometer sensor. Since cocos2dx is written in C++, we interface these via JNI native methods and threads. As I mentioned, a script based engine may be beneficial in improving the learning curve and reducing the time required in the development at the expense of flexibility. If I were using something like that, I would have to use a Low-pass filter to grab this data out which is available in its decomposed form with a little bit of extra code per application.
Cocos2dx is an event-based game engine and allows users to set callbacks in response to certain events. Every expected event must be registered with a callback function which tells it exactly what to do. For the acceleration events, I used the gravity to set the physics world gravity and the linear acceleration to apply an impulse force to all the dice present in the scene. This worked out well but in instances where the force of gravity and the linear acceleration add up. Even minute movements in that direction would over-accelerate the die, and it looked highly unnatural.
$$y = \beta x^3; \beta = y/x^3; \beta = 1/x^2 \mid y = x;$$ I weighed the linear acceleration component down using an exponential curve to avoid extreme behavior and put an upper limit to the maximum acceleration. The raw exponential curve had a very flat slope for lower values of acceleration magnitude. I settled on using a cubic polynomial which gave the best effect.
Now that we have forces enabled, the next best thing is to add collision detection. There are two types of objects in the scene; static, which are the containment walls and dynamic, which is the dice object which reacts to the force. There are two types of collision detection in this engine, one is an Entry-Exit mechanism which can track if an object has entered and subsequently exited the other object's boundary. In general, the other object is modelled by a btGhostObject class in bullet that can be queried for overlapping pairs. The other type is contact collision which returns the exact point of collision between to btRigidBody objects.
The real world does physics so elegantly we don't even bother to think about it too much. In a computer, we have to use a ton of tricks. The contact collision callbacks is designed to trigger every iteration as long as two objects are in contact. So if a body hits a wall by falling on it, it will always generate the trigger and doing one-time operations is impossible just with this approach. You could disable the trigger by masking the object but when the object leaves the boundary and comes into contact again, we won't be able to perform any action. The workaround is to reset the mask upon Exit functionality of a btGhostObject. This way, we disable further contact collision callbacks by setting object specific masks to the colliding entities and reset this mask when the other object leaves the boundary. For static-dynamic collision it works out well, but what if two dynamic objects collide? By default, the collision callback for both will be triggered and two sounds would be heard although it will happen so fast, humans won't be able to detect the phase shift. Even particle effects would be repeated and this is not a very elegant approach. We can detect the type of object and accordingly call the collision callbacks by modifying the game engine code. This provides finer control over the type of collisions and we can even use this to create different effects, like sounds for say dynamic-dynamic collision types.
This engine uses a Particle Universe plugin for 3D particle effects. The game samples had enough of these as a starting point. I already had explosions and lightning in mind and luckily enough, there were effects for these. The effects are stored in a text file and it is parsed to create an AST. Reading everything and trying to understand was going to get me nowhere with this. So, I tweaked some parameters until it made sense and doing it one at a time was incredibly helpful. I was able to convert the original explosion effect which had directional gravity to a radial direction. For the lightning, there was some smoke effect which I didn't need.
Every collision attached a new particle effect, and processing everything adds a lot of lag to the system. At first, I avoided this by limiting the number of particle effects attached to the dice. It worked okay, but when many collisions happen at once, it reaches the limit too fast. When further collisions happen, it will look like the dice are not reacting at all. To fix this, I kept a pointer to the last event and attached a newer event if a specific time had elapsed.
To complete the game, I had to polish it a bit more by adding a couple of cosmetic changes. I also wanted it to look more professional and the only thing missing from this is a Load Screen. Since I am using open-source software, I can come up with my own logo here. I've thought to name my one-man "company" Blood Eagle Studios as I watched Vikings recently and I am still a bit high on that.
Found a creepy intro sound that goes really well with this. To make this work, I created a new scene and pushed it on top of the original game scene. In Cocos2dx, the Director class suspends the previous scene and runs this one and I've created some action sequence in which the Sprite is faded-in, faded-out and then a callback to pop the current scene is called which runs the original game scene.
I am doing some insertions and deletions of objects in the main loop and I was worried about memory leaks. To be more confident, I ran a basic check to detect leaks and found some related to Label creation my game code. There were a lot more problems, especially related to fmod and AudioEngine but they were inherent to the Game Engine and I wanted to fix everything but that should go as pull requests to the game engine repo.
I recorded an early version which still is a little bit buggy but demonstrates the usage.Eventually, I managed to create something cool and lame at the same time. I wanted to keep it simple and for all ages because of the legalities associated with uploading the app on the Play Store. In this project, I didn't exercise too many design patterns because it wasn't worth it. This app/game has a straightforward use case, and it serves the purpose well enough. In upcoming projects, I will showcase those concepts more.
I have built this to simply showcase this as a complete project built from start to finish. If people actually like this and require some new types of dice, then I can think about including those as in-app purchases to get some monetization. I have absolutely no idea on how to integrate the purchases with Google Play Store which would be a good experience in itself.