The please-don’t-sue-us disclaimer: This post is a personal interpretation of a scientific work applied to actual use cases, based on information available online. It is not an intention of the author, to belittle the technological achievements of any company mentioned below. Non-compliance with a definition of AR spread in the scientific community does not make any application less valuable to its user.

Augmented Reality is a pretty controversial topic. On the one hand, you have Google and Apple going out of their way to convince you that their AR-Framework is all advanced and futuristic. And “just think of the opportunities”, and “future begins today”. On the other hand, your own reality is still not augmented much, is it? Somehow, you don’t see facebook profile info hovering over your friends’ heads. And if that’s too much to expect unless you are in an episode of Black Mirror, you could at least have some product info show up when you scan your favorite grocery store’s racks. Or a nice AR-Dressing room for an online apparel store. Just anything other than that omnipresent IKEA Place app. But the market still doesn’t have that much to offer. Of course, we could put our patience pants on and wait, because the augmented future could come any minute now. But what’s the fun in that? Especially if instead we can dig out all the things that went wrong with the technology in the last couple of years and enjoy our ride to the AR-failure town. Now that’s what I call entertaining, so grab all the snobbism left over from that time you found an antipattern in someone else’s code, and let’s go.

The Theory

I know, I just said we are going to have fun. But first things first: we are not going after AR-apps only for them implementing a ridiculous use case. That would be a very subjective approach. For all you know, there might be someone who likes taking selfies with virtual perfume bottles (here is a link in case you are one of them). And I dare you to explain how Pokémon GO fans are more superior than those of AR Mosquito Hunter (App Store and Google Play, thank me later). I guess what I want to say, is that weird use cases are not the place to get judgy since you never know who for and how they might be useful. Instead, we are going to take the high road and go about judging others from a more objective, scientific point of view. To do so, we will have to build up our knowledge basis first, and this guy is going to help us out:

His name is Ronald Azuma, and if you think Augmented Reality is the next big thing, he should be your hero. In 1997 Azuma published a paper that has been referenced in almost all the AR-related literature ever since. A Survey of Augmented Reality reflects on the technology’s state of the art at the time and provides an exhaustive overview of applications, characteristics, and challenges of AR. But the reason Azuma’s paper got referenced over and over again was the definition, which managed to separate reality augmentation from whatever the hell people would call AR to make it sound cool. So let’s hear the definition.

Azuma defines AR as “systems that have the following three characteristics:

• 1) Combines real and virtual
• 2) Interactive in real time
• 3) Registered in 3-D”

Combining real and virtual seems obvious at first glance. You probably already have a Pokémon or an IKEA couch sitting on your living room floor in your mind. Or a bottle of perfume on a piece of paper next to your face, I am not judging. But this is only a small part of the picture since all of these are examples of visual AR. There are, however, many more ways to augment reality. One for each of your senses. Combinations are also possible and can provide even more convincing results. In theory mostly, because despite all the research we are still not that good at combining realities seamlessly.

Next on is the interactiveness characteristic. It is essential, that augmentation coexists with reality in real time. That means no photo-filters or video-editing will end up being a legit AR, no matter how realistic they look. You have to be able to interact with the superimposed components, even if the whole interaction comes down to moving around and seeing (or hearing, don’t forget about the first point we’ve made) virtual objects from different angles.

Last but far from least is the 3D characteristic. You have to be especially careful here because this is where most of the examples in the next section get it wrong. It is also not exactly clear from the way Azuma puts it in his definition. As a counterexample, he references 2D overlays, which do not register with the 3D world. Later in the paper, it becomes clear, that not all the augmentations have to represent 3D objects, just like on this picture:

You can see a 2D user interface window attached to a person. Azuma mentions it as an example of an AR application and underlines that the windows move with the person in 3D space. To us, this is a clear signal, that we can make our augmentations whatever we want them to be, as long as they live behind the screen, rather than stick to the other side of the monitor. This characteristic has some exciting implications for non-visual AR. You could google that. Or you could tell me in the comment below, and I will do the googling for you and make it into another AR-related post.

At this point, I finally feel comfortable to let you smirk at some unfortunate AR-experiments, which are only AR in the heads of their creators. Let’s take a moment to thank mister Azuma for this delightful opportunity and begin our journey.

There is something on my camera

Do you know that feeling, when you are trying to make a photo, and there is some dirt sticking to your camera’s lens? Well, turns out it is a microscopic Converse shoe! Download their app, and it will adjust the camera focus for you to see which model stuck to your smartphone. If you are lucky, you can even position your foot in front of the camera just the right way and make it look like you have some mediocre photoshop skills. Don’t forget to share your result on social media.

The app I am talking about was called “The Sampler” and launched at the end of 2010. It is hard to pin down the official announcement made by Converse themselves today, but multiple articles at the time claimed for the app to use Augmented Reality ^1,2. Interestingly, R/AG, the company which worked on this project together with Converse, still refers to the Sampler on its website but does not mention AR³. An intentional marketing move or an honest mistake, but the app got known as an example of AR in retail. A simple Azuma-check will easily disprove this thesis:

• yes, The Sampler does combine a virtual shoe with your real foot made of flesh and bone,
• and yes, it does so in real time, even allowing you to interact with the overlay by changing the shoe model,
• but there is no way it registers with the 3D space.

The shoe doesn’t care where your foot is. And it couldn’t give fewer sh*ts about the rest of the space around it. The shoe does its own thing, playing by its own rules. And don’t get me wrong, it has all the right to do so. Just not when someone’s calling it AR, and certainly not on Azuma’s watch.

Unfortunately, The Sampler is not the only example of these sticker-type apps actively marketed as AR. One of my favorites in this category is Webcam Social Shopper by Zugara.

The company did officially claim that their product uses AR and even gave it an inspiring slogan “Augmented Reality meets Utility”⁴. The best part is that, technically, they are right. Before the overlay freezes on the screen forever, you position it using a special AR-marker printed on a piece of paper. The augmentation sticks to the marker in real time and registers in 3D space, just to Azuma’s satisfaction (newer version from the picture above doesn’t need a marker). Disappointment awaits the moment it comes to the actual functionality, the dressing room part. In a blink of an eye, all the AR-euphoria ends, and, just like Cinderella’s carriage turning into a pumpkin, your virtual outfit turns into yet another sticker on the screen.

The Iron Man technology

Not all overlays ignorant of their surroundings look like they are glued into place. Some are using futuristic camouflage to trick you into thinking that they have something to do with AR. Porsche is an example of a company way too gullible in dealing with such insidious overlays.

Just recently, their brand new Tech Live Look technology appeared on the Roadshow’s YouTube channel. The application revolutionizes tech support combining smart glasses with modern video chat features. Using Tech Live Look, technicians can guide you through the repair works not only with their voices but also by drawing directly onto your field of view. Very innovative and undoubtedly useful, but not Augmented Reality. Why not?

The drawings are not aware of their surroundings, they do not stick to objects in space and can’t be interacted with. Same goes for the virtual screen appearing while wearing the glasses. So every time you see Tony Stark with all those complex graphs and diagrams in front of his face, remember about the guys from Porsche. They are just as cool. But neither of the two does Augmented Reality.

Shopping with Hiro Nakamura

For another truly amazing example of a misconception about AR, let’s travel back to February 2014. In a press release published on their website, IBM proudly announces their cooperation with Tesco to pilot an Augmented Reality mobile app. To completely overwhelm the reader with their high-tech solution, IBM links a video-report directly from the field. In 3 highly unspectacular minutes, this video makes it painfully obvious, that people at IBM are not as familiar with Ronald Azuma’s work as we are at this point.

The main purpose of the new app is to improve product availability for customers. A Tesco employee would take a photo of the racks with a tablet. It would then be sent to the server, which recognizes the aisle layout and compares it to the stored data. The server responses with that same picture, augmented with overlays at spots not compliant with the preferred layout. All fine, until you realize, that this right here is the entire reason the app is said to implement AR:

The problem here is that Scientists at IBM disregarded the second characteristic in Azuma’s list. Virtual objects have to be interactive in real time, which is not the case if you add some clickable checkboxes to a photo. I guess, technically, freezing time while presenting this type of overlay would make it Augmented Reality. But until Tesco starts getting help from superheroes instead of IBM, their app will have to stay on this list.

Let’s make one thing clear: not every app advertised as AR is an imposter. But now that we’ve covered a number of unfortunate examples coming from eminent companies with tremendous funding going into research every year, you can hopefully see, that nobody is perfect. If you can learn one thing from reading this post, this is what you should remember. People make mistakes. Being in tech for the last three years, I have often felt like I will never be able to learn enough and become at least somewhat like the professionals in the field, who are just never wrong. And some of them are happy to help you maintain this opinion. But fear of failure is the biggest obstacle on your way to learning. So never let it prevent you from trying new things, and if someone points at your mistakes, see it as a chance to grow.

Let’s learn from each other.
Dannynator.