Interactions in Extended Reality

Consider for a moment how you interact with your physical reality, extended reality, and digital reality. Your physical reality is quite influenced by your extended reality. Those objects that you interact with on a daily basis usually have been influenced by your interactions within XR. See if you can point out five objects around you that might have been purchased online. Notice how many objects have been acquired by influences from a digital screen.

As these interactions blend together, keep in mind what is part of your actual reality.

What are Interactions?

When we think about the solutions that people create on a daily basis, we have a huge variety of options. Many solutions are created for problems that individuals manifested in their own reality. As many as those seem, there can be significantly more interactions. However, people might not know of the possible interactions that they could create. The interactions are only limited by your imagination.

Imagine what you could do with your hands, head, ears, nose, and mouth in your XR experiences. How often do your daily interactions come close to using any of those senses?

Here are a few suggestions: flipping, brushing, tossing, breaking, opening, and wiggling. Consider a few more questions for interactions in XR:

What could happen if you could undo something in a painting? 

What if you could quickly swap out colors for an outfit in a photo?

How do you create such an interaction?

How do you make it feel intuitive?

How do you bring it all together in an XR experience?

How often has moving or waving your hands had an impact in someone else's life?

It all starts with you and your imagination for XR interactions.

With interactions, experiences get created and people are conditioned to be perfect by imperfect people. Grab control of your reality and manifest the experiences that you want to be a part of.

Keep in mind that there some interactions that can be used across multiple senses. Augmented reality would be the closest to real life since there are digital overlays onto your actual reality. Virtual reality creates most of what you’re experiencing digitally to simulate your actual reality. Mixed reality involves quite a bit of what you’re experiencing on a daily basis. Consider how you’re inputting information with physical objects to create a digital experience. By separating these, you will be able to see how some interactions may translate exceptionally well in between a physical and digital reality.

There will be some interactions that do not translate well in other mediums. Similarly, some VR experiences that can’t exist in other mediums. The following is not a comprehensive list of interactions but it can get your imagination rolling for the five senses.

Vision

Visualize Reality

The vision sense makes XR technologies so exciting, fascinating, intriguing, and DECEPTIVE. With XR, you can be led to disbelieve your own eyes by external influences. If you don't believe me, search for comparison pictures of social media versus actual reality.

Quite frequently, individuals get influenced by sensational headlines, edited photographs, lifelong indoctrination, and fake news about our actual reality. It all starts with the eyes. They can't break free of the interactions that their vision shows them. Close your eyes for a moment. Envision something calm and peaceful. Nature, family, and food are the first few things that could come to mind.

Take a Look Around

With your head, you can look around, observe actual reality, and make note of things.

‍

It might feel unusual to read this, but really enjoy and focus on the photo above.

‍

‍What kind of visual interactions have you had that made you question what you're seeing with your own eyes?

XR is a fascinating medium that continues to influence and shape how we see reality. As a user moves their head, eye tracking software can gather insights on where they are looking at for future decisions. All of these data points can get some information on what a user is doing or intending to do. You might see this in games where a person needs to look around to enjoy the game.

Consider the number of daily experiences that could not happen without vision. Even with vision, there are many people who still can't see actual reality since it is shrouded by the influences of their extended reality. Wake up to your actual reality.

When your vision gets affected on a daily basis, you might trust your eyes less and rely on other "trusted" sources. Do you know what kind of reality that they are actually seeing? Be vigilant about those sources that are influencing you with computer-generated imagery (CGI) and embrace how your eyes discern actual reality from extended reality.

Open Your Eyes

Consider how much you have taken for granted with your vision. Try to move through your daily routine with your eyes closed for 5 seconds, 10 seconds, and 30 seconds. Work up to what you can actually see or visualize with your eyes closed. You might be surprised at how much you can achieve with your other senses. Then, open your eyes and pay much closer attention to the subtle details of actual reality around you. Observe what visuals are necessary in your XR experiences and what are unnecessary.

Touch

Feeling Reality

When it comes to touch interactions, let’s divide up the interactions between devices as 2D surfaces and 3D environments. This distinction is important since users might be limited to what they think they can do based on what an experience enables them to do. With many daily XR interactions, users who live in a 3D environment mainly interact on a 2D plane.

2D Surfaces

Typing on a physical or virtual keyboard, clicking on a mouse, or using a stylus to provide input to an experience, the user is using a 3D object to interact with a 2D plane. Keyboard, mouse, or controllers can provide input for specific interactions that take place on a 2D surface. These devices typically have buttons that can be pressed to execute an action. A mouse or joystick can be moved to simulate translational motions within an experience. Pens can be used to draw or interact with applications.

For applications with graphical user interfaces (GUI), users can rotate an object to make modifications with their preferred inputs. However, they are still touching a 2D plane to make changes to their 3D digital object. Then, the experience truly changes when the user is placed within a 3D environment.

3D Environments

Hands and controllers provide different methods of input for the user. It increases the complexity of what a user can do and attempt to do. The user's attempted actions can be a fascinating study on its own since it will shine light on new experiences that will be created. Consider these questions for building in a 3D environment:

• What are some key activities that can be done in an experience if a user has hands?
• How does that open up the experience?
• How does it change when we can move an object in 3D?
• What happens when we can literally get our hands on to everything?
• What are some experiences that are LESS intuitive than on a 2D plane?
• In what ways are virtual hands unnecessary?
• How many things it can be done better with just a mouse or a keyboard?

With a virtual keyboard, typing out something can be quite cumbersome. For these XR experiences, consider multiple modes of inputs between a mechanical controller, hand tracking, or other means of touch inputs.

Common Gestures

As you reflect upon your extended reality, there are quite a few gestures that you might have been accustomed to without realizing it. Many of these interactions take place on a 2D plane with your mobile device. Some of these interactions can translate well into a 3D medium, but some might be better situated in just a 3D environment.

• Grab
• Rotate
• Throw
• Catch
• Flick
• Push
• Tap
• Bounce
• Swipe
• Press
• Draw
• Pull
• Poke

 What are some other interactions that could be created to make an experience fascinating?

Touchless Experiments

Pick a location that is calm and peaceful. Close your eyes to help reduce the distractions. Count how many things do you feel with touch. Reach out in any way that you prefer and try to touch something else. Determine what you're touching without looking at it. Try to describe it. Consider how often you actually use your sense of touch on a daily basis. Evaluate what kind of touches that you will want in your XR experiences.

Hearing

Surround Sounds

Can you hear it?

Can you hear the music from your last concert?

Remember how the sounds made you feel.

For virtual reality, the hearing sense is one that is really noticeable when you DON'T have it in an experience. It adds tremendous depth to any experience, and it can easily control the mood of the experience and a user's emotions. In addition to visuals, it can provide confirmation that an action took place. This becomes quite critical for immersion within any VR experience. Try turning off the sound of a VR experience for a few minutes and evaluate how engaged you are within an experience.

For mixed reality and augmented reality, audio can also provide cues that an interaction took place. Consider the clicks on a virtual keyboard, voices that read out directions, or an audio jingle that starts at the beginning of a television show. We are consistently influenced by sounds that keep people asleep and in a trance. If you don't believe me, disconnect from it all with two or three days in nature. It will be a wonderful reset of all the noises that are actually around you.

Pay attention to some of the digital sounds that you hear on a daily basis. How many of those sounds are still kept to remind a user of a response that they are supposed to have?

Speech Recognition

Of the many interactions involved with hearing, speech recognition stands out for XR experiences. Messages can be read back to a user with a human-like voice. Messages can be sent between individuals with a digital recording. Talking heads are on TV and social media telling you how to think, what to believe, and what to do. In most of these cases, the transmission only happens in one direction. From the device to you. What if your XR experiences could use speech recognition to engage with the talking head? 

Speech recognition provides a mechanism for input from the user. This can be straightforward with requesting permission to access their microphone to record audio. From these inputs, the experiences can adapt to the users with their verbal inputs. It can be useful for commands, dialogues, practice, or additional methods of interacting with objects.

Listen Carefully

Find a location or experience that you are very familiar with. Visit it with earplugs in. If it is familiar, some of the sounds can be created for you by your brain. Spend 10 minutes, 30 minutes, or an hour with such an experience. Notice the sounds that you can't hear. Notice what you expected to have a sound. Observe how the experience feels without your hearing.

Try this experience twice. Once with the earplugs and once without them. Evaluate how much of the experience that your brain is creating based on past experiences. Consider what is essential to hear for your XR experiences.

Smell

Hint of Anticipation

How an experience smells can shift the user's expectations. Smells from a restaurant, product, or open area can drive business alone. It creates an anticipation or expectation for the XR experience. Consider how many experiences are driven by only smell. Fragrances, hospitality, textiles, medical, and manufacturing could be XR experiences that could elevate immersion with just smell.

Within AR and VR, there are new devices that could simulate the sense of smell for an experience. A burning scent could provide indication of a nearby or recent fire. Some food aromas could point the user in the direction of restaurants or cafes. Some unpleasant scents offer indications that an uncomfortable event or situation might be right around the corner.

It will be fascinating to see how different environments, scenes, and experiences can be enhanced with scents.

Whiff of Emotions

Within extended reality, the simulation of smell is a work in progress. The designers and developers have many years of life experience to study how our senses can influence emotions to take action. All of the senses can heavily influence our reality. They influence how we make decisions, how we take action, and how we ignore our intuitions. Consider how certain scents and mixtures will create a type of feeling such as calmness, relaxation, or positivity. These influences enable individuals to bend and shift another individual's extended reality without them realizing it.

Discover Some Scents

Close your eyes and walk with someone for an experience through a marketplace, mall, building, or in nature. Notice how other senses might heighten without your vision, but focus on your sense of smell. Determine if you can describe the location with the present scents. Identify what indicators provide that immersion for your experience. Articulate what are some smells that would be great to start with for an XR experience.

Taste

Sample of Reality

Consider the last time that you tried out some food based on another person's recommendation. Perhaps it did not turn out as magnificent as you expected it to be. Perhaps it exceeded expectations. But the taste interactions in extended reality have happened more often than you might expect.

Think about the last time that you went to try a food place WITHOUT reading any reviews. Try it out for a few times. You might be surprised of what you find out without many of the external influences.

Intriguing Cravings

Since all of the senses can influence and control emotions, taste is no exception. Think about the number of people you know who are not addicted to the taste of something, but "they enjoy it everyday." Their reliance on a particular taste that brings them happiness, energy, or positivity can influence their mood for the rest of the day. Notice how mentioning certain foods or products that have desirable tastes can shift the mood. Taste isn't just limited to food. It can apply to recreational substances, liquids, or other activities that focus on the sense of taste.

Savory Experiences

Put on a blindfold to your next meal. Your sense of smell can provide some insights on what you are about to taste. Take the moment to enjoy what you are about to taste. See if you can determine what components make up the food that you are tasting. Appreciate the textures and consistency of the dish. Contemplate how the sense of taste can augment and differentiate your XR experiences.

Acknowledge the journey that you are on with your senses. It will transform how you embrace the interactions that you have in XR.

As you continue the adventure into extended reality, notice how each of the senses can be used to influence emotions and immersion. Embrace each sense and spend some time focusing on how some senses work well in certain situations. When you unlock your imagination, remember who you are, and embrace your dreams, then there will be many more magical interactions to come. The future interactions are literally in your hands.