The Journey

For Pre-Thesis

My thesis for this class revolves around the theory of cognitive behavioral therapy. Because we are facing a worldwide pandemic, I decided to shift my focus on creating an game/application people can use to calm their minds or at least put them in a different mood for the better. Through this class, I learned different approaches for deciding on what I want to make. My background is in psychology and game design, so I wanted to create something that combines both fields. What I wanted was a VR/AR experience that could focus on behavioral therapy for children with Autism. As I continued on, I realized that the concept is larger than I anticipated. I decided to change my thesis and pivot towards making something for myself.

In the event of Covid, I became quite aware of myself and had more emotions to deal with. Since I still have a strong interest in mental health for my thesis, I decided to play around with the idea of a game or application that lets you become aware of what you are feeling. I am interested in creating an aid to help you understand what emotions you are feeling. During the course of this class I developed two areas of my thesis, three working prototypes and an overall story concept.

Here are my three prototypes:

Unity WebGL Player | Mandelbrot

Unity WebGL Player |Mandelbulb

Github repos:

gtn223/Thesis-Prototype

gtn223/Mandelbuld-Proj

Why did I make these?

These prototypes fall under the aspect of my overall story I developed around the concept of discovering one’s emotions. These prototypes represent what the user will see as a representation of their “self”. I want to visually represent what “seeing” into your emotions would look like. The “emotions” are from a biased perspective (my own), which is why I am focusing on using this application on myself first.

Overall story:

• The main character is lost
• Stuck in the forest, trying to get out.
• Maybe hear something or see something that leads them deeper into the forest.
• The user will then find a doorway into the (Mandelbulb 3D) world.
• The player has to help another character escape to their world, too. (metaphor of emotion and awareness).
• Within the world, the user will notice how things change based on their interactions with the in-game character.
• Game feel mechanics:
• Exploration, finding your way back (creating the metaphor of finding yourself)
• Needing help from someone or something (being aware of your actions)
• Use each each other and help one another(learning to be more open minded).
• “Darmok and Jalad” TNG reference
• Epic of Gilgamesh Mesopotamian myth

Continues more on a previous post here: https://medium.com/@gtn223/story-concept-c4541dc4836b

Tech Research:

I investigated into more VR Autism apps that are out there for people to download and use.

Games in VR

Brain Power — Uses converted google glasses to help kids detect emotions and trains eye movement using AR. The images are overlaid onto the users perspective. The images look like 2D graphics. This is a company that has been increasing in popularity and in research, since 2013. The company is backed by a solid team of Doctors and medical professionals that serves as an advisory board. It looks like they are venturing into new technology incorporating Ai and machine learning into their products.

Floreo — This is a company that uses VR on a mobile device and focuses on joint attention for children with autism. Joint attention is a term used in psychology that defines two people engaged with an object in some form of communication. Non — verbal, verbal, eye gazing, gesturing can all be categorized as a form of communication.

Beam — Projection mapping. Similar to the Dynamicland interactive projections, Beam uses a similar technology for children with ASD. Not entirely clear on the objective of this company. There is an educational aspect to it, where kids can engage in games geared for social play and interaction, but not sure if the intention of the tech is supposed to be used for this. There is an entertainment value to this type of experience and it looks like the company is moving more into the space of entertainment for McDonalds and other larger corporations.

Emotional Game Applications

Mightier — A company that helps regulate their emotions through game play and biofeedback. IDEO also partnered with this company to create the design of this product. Article is here.

Pause — This is a mobile game that functions as a stress relieving app. The players drag their fingers on the blob and gather more dots to make it the blob bigger. This app was created by the same studio of Monument Valley, a puzzle mobile game. The creator of this app, Peng Cheng, wanted to create an app that helps ease the mind and get into a meditative state.

Physical Spaces

Snoezelen — A relaxation room developed in the Netherlands and became popular in Germany. The room was designed for behavioral therapy and provide a mental respite for people suffering from dementia, Alzheimers or brain damage. The room has been in the use for people with Autism, which has been proven quite effective. https://www.sciencedirect.com/science/article/abs/pii/S0891422206000461

Luxury Escapism — An experience located in Brooklyn. Basically like a VR spa and ASMR type of settings. The space is really a digital spa where people come in and relax and meditate with VR. The space inspired to think creatively in the realm of VR/AR and provide a therapeutic goal in mind.

Psychological Research:

Research on using geometric shapes:

Belin, Laurine et al. “Simple Shapes Elicit Different Emotional Responses in Children with Autism Spectrum Disorder and Neurotypical Children and Adults.” Frontiers in psychology vol. 8 91. 30 Jan. 2017, doi:10.3389/fpsyg.2017.00091

Research on using Virtual Reality Therapy:

Botella, Cristina et al. “Virtual reality exposure-based therapy for the treatment of post-traumatic stress disorder: a review of its efficacy, the adequacy of the treatment protocol, and its acceptability.” Neuropsychiatric disease and treatment vol. 11 2533–45. 3 Oct. 2015, doi:10.2147/NDT.S89542

Vincelli, F., et al. “Virtual reality assisted cognitive behavioral therapy for the treatment of Panic Disorders with Agoraphobia.” Studies in health technology and informatics 85 (2002): 552–559.

Combined Inspiration

After looking at existing applications, physical spaces and psychological research on virtual reality therapy, I came to the idea of creating an interactive geometric emotional based game/application. Based on the research, there is evidence that behavioral therapy can alter the mind to be in a changed state using a technological application. With therapeutic applications using more spatial technology like VR and AR, perhaps there could be more areas to explore the effectiveness of therapy represented in different ways.

I wanted to play around with the idea of using geometric shapes and finding a way to animate shapes. I got into the idea of using origami structures, mainly from architectural drawings. I am thinking of playing around with the idea of giving abstract geometric shapes a “personality”. I want to see what it’s like to put behavior onto a shape or pattern.

From origami, I came across the fractals in shaders and the use of shader toy. The idea of creating mandelbrot and a 3D mandelbulb takes a long time to render out. But within a shader, the “look” is the same but the result is a much more optimized version. In the story, the player is lost in the world aka, their emotion, I would like the world to be represented a living thing. Something that can react and help them reach enlightenment about themselves. When I first encountered the mandelbulb videos, I was completely mesmerized by the detail and awe of the beautiful design.

Shadertoy

Conclusion

The application I want to create relies heavily on the fact that people will “feel” something when they start the experience. I have been personally testing the interactive shaders on myself at the moment and tweaking them to fit my personal needs. Dan, brought the concept of self testing and I decided to try it out three times a day. I would like this to tie into my overall story arc of this experience. Where I can comfortably say that the application lead me into a new world and discovery. The “new world” being a digital representation of my current emotion and “discovery” being the awareness of feeling while I am going through the mandelbulb world.

I recently been getting into more techniques in shaders and becoming more interested in the technique of using ray marching for real time computing. I still want to incorporate the facial tracking prototype and link to the mandelbulb shader and possibly create a procedural ray marched character animation. Creating the link will be difficult, which I hope I can figure out how to do within the summer hopefully. I told myself and Dan, that I will not get distracted by the minutia of coding the concept, so I intend to approach the core mechanic of my application as well. I know I need to develop a strong story as well. There are still some grey areas for my story in how I intend to use the application for emotional awareness and what the means to the story.

References

Botella, Cristina et al. “Virtual reality exposure-based therapy for the treatment of post-traumatic stress disorder: a review of its efficacy, the adequacy of the treatment protocol, and its acceptability.” Neuropsychiatric disease and treatment vol. 11 2533–45. 3 Oct. 2015, doi:10.2147/NDT.S89542

Chiou, Erin K., Noah L. Schroeder, and Scotty D. Craig. “How we trust, perceive, and learn from virtual humans: The influence of voice quality.” Computers & Education 146 (2020): 103756.

Ellaway, Rachel, et al. “Virtual patient activity patterns for clinical learning.” The clinical teacher 12.4 (2015): 267–271.

Freeman, Daniel, et al. “Automated psychological therapy using virtual reality (VR) for patients with persecutory delusions: study protocol for a single-blind parallel-group randomised controlled trial (THRIVE).” Trials 20.1 (2019): 87.

Geraets, Chris NW, et al. “Virtual reality-based cognitive behavioural therapy for patients with generalized social anxiety disorder: a pilot study.” Behavioural and cognitive psychotherapy 47.6 (2019): 745–750.

Lin, Hsin, and Hua Wang. “Avatar creation in virtual worlds: Behaviors and motivations.” Computers in Human Behavior 34 (2014): 213–218.

M. Gonzalez-Franco, A. Steed, S. Hoogendyk and E. Ofek, “Using Facial Animation to Increase the Enfacement Illusion and Avatar Self-Identification,” in IEEE Transactions on Visualization and Computer Graphics.

McKee, Shari A., et al. “Effects of a Snoezelen room on the behavior of three autistic clients.” Research in developmental disabilities 28.3 (2007): 304–316.

Rifinski, Danielle, et al. “Human-human-robot interaction: robotic object’s responsive gestures improve interpersonal evaluation in human interaction.” Human–Computer Interaction (2020): 1–27.

Schofield, Pat. “Snoezelen: its potential for people with chronic pain.” Complementary Therapies in Nursing and Midwifery 2.1 (1996): 9–12.

Sonalkar, Neeraj, et al. “Augmenting Learning of Design Teamwork Using Immersive Virtual Reality.” Design Thinking Research. Springer, Cham, 2020. 67–76.

Lin, Hsin, and Hua Wang. “Avatar creation in virtual worlds: Behaviors and motivations.” Computers in Human Behavior 34 (2014): 213–218.

van Diepen, Erik, et al. “A Pilot Study of the Physiological and Behavioural Effects of Snoezelen in Dementia.” British Journal of Occupational Therapy, vol. 65, no. 2, Feb. 2002, pp. 61–66,