The Apple Vision Pro helmet is the future of mixed reality

Before that, successes and failures alternated in the industry for more than ten years. Why didn’t the revolution come out?

On June 5, Apple introduced the Vision Pro mixed reality helmet - the company has developed it at least since 2015. The headset, despite the high cost (three and a half thousand dollars), caused positive reviews of specialized media and technological blocks that checked it in the case. However, Vision Pro will wait long for a long time : the release is planned only at the beginning of 2024.

Over the past ten years, several companies immediately tried to create a device that could build a whole market for new technology - but so far the results left much to be desired. The developers were looking for a form for a gadget (trying both almost weightless glasses and extremely heavy helmets), tried to find the perfect balance of the real with virtual and encountered constant complaints of wearing such devices. "Medusa" tells the history of Falstarts of promising technology - and lists the reasons why the promised revolution did not occur (but maybe with the release of Apple's headset).

What is mixed reality and why is it so called?

Until now, industry experts have not agreed on what to call the technology that they are trying to introduce. The term “Augmented Reality (AR) was originally spread: it was he who met in the descriptions of the Google Glass smart glasses and Pokémon Go . To create augmented reality, it is enough to have a device with a screen. The technology allows you to see digital objects built into real scenery. The simplest example of AR is an indicator on the windshield of aircraft or on a pilot helmet.

With the development of immersive technologies, experts started talking about mixed reality (Mixed Reality, MR), where virtual and real worlds interact. In the 1994 article , a team of scientists under the start of Paul Milgram from the University of Toronto introduced the idea of ​​a continuum of reality-virtuality: from ordinary reality through a supplemented to virtual. All intermediate stages on this scale were proposed to be called mixed reality. Today, MR is most often used as a concept that combines AR and VR (although for clarity many prefer the designation AR/VR). When talking about mixed reality sets (for example, Microsoft Hololens or Meta Quest Pro), manufacturers emphasize that their gadgets can both show fully virtual VR expirins and work in AR mode, combining digital objects with reality around the user.

In order to finally confuse everyone, in recent years, experts began to use the term “expanded reality”, which combines all varieties of digital manipulations with the world - AR, VR and MR. The new abbreviation has partly spread thanks to the OpenXR project-the standard of access to virtual and augmented reality platforms, introduced in 2019-and the Webxr application programming interface, which allows you to launch VR and AR-EXPIDIRIENS in the web browser.

Although smart glasses and other augmented reality devices have not yet gained wide popularity, local AR applications have long been a habit. Helen Papagiannis, author of the book “Augmented Human” (2017, translated “Augmented reality”), lists the five main areas of application of AR: healthcare, education, work, entertainment (including games) and trade. In the last area of ​​applications, there are especially many applications - from embedding virtual Ikeev furniture into interiors to trying on virtual clothing.

How exactly augmented reality works and how to manage it

Modern smartphones are equipped with cameras and a large number of sensors : in the presence of software, this already allows you to combine digital and real objects. In the case of full-fledged AR devices, new-wearable optical devices and control mechanisms are added to the basic requirements, in which it is not necessary to press on the touch screen.

In the field of optics, many options (glasses, helmets, contact lenses and retinal monitors) are tried, in which the image is projected onto the retina. After more than ten years of development, manufacturers mainly stopped on headset, which are fixed on their heads and close the field of view, like a ski mask. Companies are increasingly abandoning the completely opaque helmets, where the outside world is transmitted to the display using the camera system, the Apple Vision Pro and the latest QUEST line are just like that.

The imposition of virtual objects on a real environment is a complex technical task. The hardest thing is to make a three -dimensional model of the surrounding space, which is then complemented by digital elements. All this should exist in dynamics, responding to actions and movements. As Microsoft experts explain , two main directions in AR development, marker and non-macker technology requires image recognition, but in the first case, digital markers (for example, QR codes) have pre-added to determine the position and orientation of the camera. The AR object is then placed at the place of the marker. In the second case, the mechanism recognizes all objects in the field of view, and the binding occurs to other “anchor” elements - for example, a plane or geolocation.

Augmented reality devices are mainly controlled using physical buttons, wheels and touchpads on cases, controllers, voice commands, gestures, pupil movements and heads. Many of these interaction mechanisms have not been fully mastered. There is no single standard, as is the case with the sensory screen of the smartphone. So, each manufacturer is forced to invent his own gesture recognition system (for example, when using Vision Pro, you need to undergo training). In addition, each specific application may include its own mechanics (as in VR expirins)-for example, manipulation of virtual objects. One of the difficulties of introducing augmented reality is precisely in the variety of virtual interfaces and scenarios of interaction. Many of them are rejected by trial and error: each new device seemed to try to invent AR-Velovich.

The more widespread headsets of mixed reality become, where AR and VR complement each other, the more often the devices can use the achievements in the area of ​​virtual reality-for example, imitation of tactile sensations, complex gaming manipulators (like a light pistol) or even a VR coach .

The creators of gadgets with mixed reality seem to have made all possible mistakes - these are the main points

Augmented reality technologies were seriously spoke after Google introduced smart glasses in 2012. Along with this, the product failure for a long time “froze” the industry. The presentation of the gadget was accompanied by a loud media campaign. His show at the Google I/O conference began with paratroopers who jumped out of the airship with smart glasses in front of his eyes, broadcasting the picture directly to the conference room (see how it was here ). Designer Diana von Fürstenberg forced the models to wear Google Glass at the Fashion Week in New York (the head of the corporation, Sergei Brin, climbed the podium). Time magazine included smart glasses in the list of the best inventions of 2012. A photograph of Prince Charles in Google Glass scattered around the world. The creators of the Simpsons shot an episode dedicated to the gadget.

When the first wave of enthusiasm subsided, numerous disadvantages of the device were found. The Business Insider publication in May 2013 released an article with the heading “Verdict: Google Glass nobody likes”, mentioning the weak battery (withstanding from three to five hours), the abundance of bugs, the fading arches (it was meant that you wear glasses constantly), the low brightness of the screen, which was interfered imperfect voice recognition. Engadget journalists complained about the inability to control the settings: it was impossible to turn off Wi-Fi and Bluetooth with glasses, change the volume level or rearrange the icons in the interface. The device also reproached for causing a headache.

The emphasis on Google Glass was made on shooting photos and videos. But the quality by modern standards was low: five megapixels and 720p, respectively. The gadget was controlled through a touchpad on an arc and voice teams. With the help of Google Glass, it was possible to use a google-search engine, translate texts, receive and send messages (completely dictating), recognize the weather and lay routes. One of the disadvantages was called non -autonomous: the device worked only in conjunction with a smartphone. In addition, Google Glass did not create a full -fledged supplemented reality: text and graphic information simply appeared on the screen, without embedding into the surrounding “real scenery”.

In addition to difficulties in using and high cost (at the start of Google Glass cost one and a half thousand dollars), there were soon problems with privacy: nothing prevented the video in public places without the consent of others. The New York Times in 2015 noted that many establishments appeared in the United States, from bars and cinemas to casinos, which forbade entering smart glasses. The Guardian newspaper wondered how Google Glass can change etiquette and everyday habits - in particular, if the constant wearing of the gadget will lead to additional stress. The creators well understood the risks and therefore forbade the use of faces recognition in smart glasses, but quickly found hackers who went around these rules.

In Russia, since the premiere of smart glasses, rumors have gone that Google Glass could fall into the category of spy gadgets and be prohibited. When the authors of the blog about smart glasses sent the corresponding request to the FSB, the department said that the device "does not belong to the category of technical means for the unspoken receipt of information." In 2014, the media resumed a discussion of a possible ban, and TASS announced the plans of cinemas not to let the audience with Google Glass.

The discussions of Google Glass in Russia did not lead to anything, since in January 2015 the corporation unexpectedly announced the stop of production. Then the project was revived in the Google Glass Enterprise Edition (2017) and its second version (2019). In March 2023, development in this area was finally turned off - but the market of smart glasses is still developing.

The next milestone in the development of augmented reality was the release of Microsoft Hololens, headsets of mixed reality (it worked in VR and AR). Its prototype was presented in 2015 - after a three -year development period, including on the basis of the Kinect device. In 2019, the second version of the points was released, and the third, according to rumors , was laid on the shelf. Microsoft supports already released Hololens, but now positions them exclusively as a business tool.

In 2018, Microsoft entered into a contract for the supply of 100 thousand HoloLens headsets of the American army, but soon fifty corporation employees wrote an open letter demanding to stop cooperation with the military. Although the headset helped in navigation, coordination of missions and other combat missions, in 2022 there were reports on dizziness, eye fatigue and other problems that soldiers faced during the testing of the device.

Compared to Google Glass, Microsoft Hololens is a much more advanced gadget. The device is equipped with an Intel quad -core processor, and, in addition to the central and graphic processor, there is a holographic - developed specifically for the product. Hololens has four cameras (against one in Google Glass), four microphones, as well as a variety of sensors, including accelerometer, gyroscope and magnetometer. Optics is quite difficult: transparent holographic lenses are installed in the gadget. Their task is to properly conjure and send an image from micro -spokes located on the sides. The user can see the surrounding environment with added computer graphics (manufacturers call it “holograms”).

One of the main disappointments of the first Hololens users was a small field of view. Unlike virtual reality helmets, the picture in AR device is more like a normal computer display before your eyes. PCWORLD journalists compared the experience of using Hololens with a look through the helmet window that welders put on. This indicator was not officially called, but, according to the MSPOWERUSER website, the viewing angle is only 30 degrees horizontally and 17.5 vertically (for comparison: the horizontal of the visible area in the first Oculus Rift VR modes was 90 degrees, and it reaches 130 in the Valve Index helmet). During the presentation, Microsoft employees did not mention the shortage, which caused the discontent of a number of journalists. The Polygon publication called the demonstration of the Minecraft in augmented reality that the cube landscape was created on a regular table.

In the second generation, the Microsoft AR-bar increased the field of view to 43 degrees horizontally and 29 diagonally. But the VR-Schles were still far away, and the creators did not save from other problems.

The third big case in the modern history of augmented reality was the Magic Leap startup. In the early years, the product remained a secret (so that the performance of Founder Roni Abuttz at the TEDX conference in 2012 turned out to be an almost wordless performance with dancers in costumes of multi -colored monkeys - a parody of the film “2001: Cosmic Odyssey”). In 2014 alone, it became more or less clear that Magic Leap designed AR device. Then the Sundar of Pichai, the future head of Google, who was part of the board of directors of Magic Leap, said that the company from Florida would revolutionize trade, education and communications. The media enthusiastically wrote about a promising gadget ("Medusa" is no exception ). In 2015, the startup began to release demonstration videos , allegedly shot inside the augmented reality headset. I especially remember the video (remote from the official channel, but remaining with other users), on which the virtual whale jumped out of the floor in the physical education hall under the delighted exclamations of schoolchildren. Subsequently, part of the media called the video an advertising trick that has nothing to do with the real headset.

In 2018, the developers from Florida finally showed the Magic Leap One device: a set of AR chicks, a portable computer and a controller for $ 2,295. There were few available applications at the start-a virtual chat, an Internet browser, a system for placing digital objects in the real world and several more options in a digital store (they all work on the original Lumen OS operating system). The Verge review said that most of the Magic Leap headset applications are similar to mobile games or copied with Hololens. The company was primarily aimed at developers and creative people - the device was officially sold as Magic Leap One Creator Edition - but could not attract them. In the first six months, the creators sold about six thousand headsets (Aboitz promised investors a million sold devices for a year).

From a technical point of view, Magic Leap One was quite competitive. The device could boast of darkened lenses with excitements (engineers called them "photonic chips") and built -in speakers. Since the processor was put into a separate device, the glasses turned out to be light enough - only 316 grams. The battery worked without recharging for three hours. From the shortcomings of the media noted that the headset does not cope well with excess or insufficient lighting. And, just as in the case of Hololens, the viewing angle remained problematic - only 50 degrees. The Wired journalist spoke about the universal disappointment of colleagues from the fact that the device shown does not correspond to the promises of the revolutionary product.

In 2020, on the wave of pandemia, Magic Leap announced large -scale reductions. Roni Abotitz left the post of leader. Then skeptical articles about the company began to appear . Nevertheless, the work continued, and in September 2022 the presentation of the AR-hucks of Magic Leap 2 was held. Engineers reduced the size and weight of the device, and also added cameras, sensors and gesture recognition. The cost of the headset increased to 3299 dollars, and in full configuration (version of Developer Pro for developers) - up to 4099. The corporate version also appeared in almost five thousand dollars. According to the Engadget observer, such glasses were not suitable for ordinary users, but could be useful in healthcare, industry and business.

The new standard, apparently, set Oculus Quest - but not everything is perfect with it

Although the development of the VR was by and large in parallel AR (it turned out to be easier to create a completely virtual reality than mixed), these areas crossed - in 2019, when the first generation of the Oculus Quest helmet appeared on the market.

The startup bought by Mark Zuckerberg in 2014 for two billion dollars and subsequently became a META division, added Passthrough (can be translated as “passing”) to the device: a picture from cameras was simply shown on the screen. The outside world was presented in the gradations of gray, and the function was explained by safety requirements - it helped the owner of the headset not to crash into the walls and furniture. Прозрачный режим работал и во втором поколении гаджета — Oculus Quest 2, представленном в 2020 году и позже переименованном в Meta Quest 2. Гарнитура вплотную подбиралась к устройствам смешанной реальности: пользователям были доступны отслеживание рук, браузер и физическая клавиатура.

Наконец, в 2022 году вышла гарнитура Meta Quest Pro, которая уже позиционировалась как шлем смешанной реальности: цветная картинка с камер была в высоком разрешении. Правда, AR-функции устройства подверглись критике — например, за зернистое изображение на экране, ограничения при использовании дополненной реальности и перебои виртуального офиса. Даже приглашения в иммерсивный чат нужно было посылать по почте. Функция персонального офиса работала чуть лучше, но трудности с чтением интернет-страниц, необходимость синхронизации с компьютером и усталость от ношения гарнитуры сильно осложняли жизнь. Кроме того, Quest Pro оценили в полторы тысячи долларов — втрое выше, чем Quest 2. Возможно, часть проблем исправят в предстоящем Quest 3, который должен совместить низкую цену (499 долларов) и улучшенную AR-функциональность. Презентация ожидается на конференции Meta Connect в конце сентября 2023 года.

Устройства линейки Quest, с одной стороны, задали новый стандарт и запустили гонку среди крупных производителей. С другой — скромные успехи гарнитур (именно как устройств смешанной реальности) далеки от обещаний, сделанных Цукербергом в 2021 году, когда он объявил о переименовании Facebook в Meta, инвестициях в смешанную реальность и видении метавселенной. Новая стратегия Цукерберга до сих пор остается спорным шагом и послужила поводом для множества шуток ( особенно на фоне выхода Apple Vision Pro), но стала стимулом для развития AR-технологий.

Развитие дополненной реальности не ограничивалось очками и гарнитурами. Главный потребительский успех технологии — вышедшая в 2016 году игра Pokémon Go , которая не требовала ничего, кроме смартфона. Первые подобные игры появились еще в конце 2000-х — начале 2010-х, почти в одно время с Google Glass. Например, AR-игры Zombies, Run! (2012) и Ingress (2013), которая первой использовала данные GPS. Еще один нашумевший представитель жанра — Harry Potter: Wizards Unite (2019), где игроки в реальном мире искали мистические артефакты и волшебных существ. Разработки в области дополненной реальности на базе смартфонов и планшетов ускорились в 2017 году, после того как Appleпредставила ARKit — инструмент для программистов, облегчающий создание AR-приложений и внедренный в iOS 11. С помощью камеры, акселерометра и гироскопа iPhone и iPad научились определять положение пользователя, а также встраивать цифровые объекты в реальный мир. Большим подспорьем стало распознавание горизонтальных плоскостей в кадре — это позволяло виртуальным предметам и персонажам «стоять» на твердой поверхности.

ARKit стал первым шагом Apple в сторону собственных очков дополненной реальности — и спустя шесть лет этот «квест» был завершен. Еще в 2017 году специалист по дополненной реальности Мэтт Майсникс перечислил восемь пунктов, без которых AR-гарнитура от компании Тима Кука не может появиться:

• stylish appearance;
• качественная и компактная оптика;
• интуитивно понятная система управления, подразумевающая различные форматы ввода (жесты, движение глаз, голосовые команды, физические элементы на гарнитуре);
• мощный процессор;
• надежная система распознавания трехмерных объектов, которая позволит взаимодействовать с ними в дополненной реальности;
• возможность взаимодействия с другими пользователями, в том числе в рамках мультиплеерных AR-игр;
• удобная операционная система с интерфейсом, который будет обходиться без «рабочего стола»;
• наличие большого ассортимента AR-приложений и возможность выхода в интернет.

Майсникс предполагал, что Apple решит проблемы к 2021 году, но немного промахнулся — компания представила Vision Pro в июне 2023-го. Гарнитура уже отвечает большинству из этих критериев (особенно в части технических характеристик) — а также, судя по всему, прекрасно справляется с отслеживанием глаз, видеозвонками и веб-серфингом. Работа над отдельным App Store для шлема и созданием приложений дополненной реальности уже идет — Apple не просто так представила устройство на конференции разработчиков и дала им время, запланировав начало продаж аж на 2024 год. С первого дня владельцам гарнитуры будут доступны стриминговые сервисы Apple TV+ и Disney Plus.

Не менее важно партнерство с компанией Unity, разработчиком одноименного игрового движка — на его базе должны появиться приложения для Vision Pro. Множество разговоров, впрочем, ведется не о технических характеристиках, а о стоимости: Vision Pro будет в семь раз дороже, чем Quest 3 (три с половиной тысячи против 500 долларов). Ценник вполне может оказаться фактором, который перевесит все остальные и помешает широкому распространению гарнитуры.

Так, Марк Цукерберг после выхода Apple Vision Pro заявил подчиненным, что устройство от конкурентов его не особо впечатлило. По его словам, компания Тима Кука якобы не представила «никаких волшебных решений» по обходу «законов физики», которые бы уже не изучили специалисты Meta: «Apple использовала дисплеи с более высоким разрешением, но Vision Pro стоит в семь раз дороже [чем Quest 3] и требует столько энергии, что необходимо подключать аккумулятор на проводе». Тим Кук, напротив, счел стоимость Apple Vision Pro оправданной:

Сможет ли обычный человек позволить себе Apple Vision Pro? I don't know. Думаю, что люди сделают разный выбор в зависимости от финансового положения и иных факторов. Инженерные разработки и глубина технических решений поражают воображение. <…> Конечно, это не бесплатно. Это стоит определенных денег, но я считаю, что цена отличная.

Станет ли выход шлема от Apple поворотным моментом для индустрии

Незадолго до презентации нового продукта Apple издание The Verge предположило , что «дополненной реальности нужен iPhone-момент» — поворотный момент в развитии технологии, после которого устройство максимально быстро распространяется по миру. Для сравнения: именно так после выхода чат-бота ChatGPT профильные СМИ писали про искусственный интеллект. Теперь же технологические журналисты рассчитывают на то, что гарнитура Vision Pro учтет все недостатки предшественников, станет удобным пользовательским продуктом и создаст новые стандарты в индустрии.

The Verge также упомянула о подходе Apple «это просто работает» (it just works), согласно которому владелец устройства не должен заботиться о том, что происходит «под капотом». В отношении дополненной реальности этот принцип особенно важен, так как до сих пор владельцы AR-гарнитур нередко сталкивались с разнообразными трудностями и были вынуждены разбираться в сложном управлении, углах обзора и капризном поведении виртуальных объектов. С другой стороны, закрытая экосистема Apple означает навязывание пользователям и разработчикам своих правил.

Основное опасение специалистов заключается в том, что Vision Pro может не выйти на массовый рынок, повторив судьбу Microsoft HoloLens и Magic Leap, которые сейчас ориентированы в первую очередь на бизнес. Получилось ли у Тима Кука и его коллег создать «iPhone-момент» для индустрии дополненной реальности, станет ясно не раньше 2024 года, когда состоится презентация AR/VR-устройств от других компаний — а Vision Pro поступит в продажу.

"Jellyfish"