Future

Apple’s Next Game Changer: Augmented Reality

Apple CEO Tim Cook is betting on AR, which he sees as potentially as groundbreaking as the smartphone.

Augmented reality on an iPhone
Augmented reality on an iPhone
Mark Gurman*

Tim Cook has talked up a lot of technologies since becoming Apple's chief executive in 2011. Driverless cars. Artificial intelligence. Streaming television. But no technology has fired up Cook quite like augmented reality, which overlays images, video and games on the real world. Cook has likened AR's game-changing potential to that of the smartphone. At some point, he said last year, we will all "have AR experiences every day, almost like eating three meals a day. It will become that much a part of you."

Investors impatient for Apple's next breakthrough will be happy to know that Cook is very serious about AR. People with knowledge of the company's plans say Apple has embarked on an ambitious bid to bring the technology to the masses, an effort Cook and his team see as the best way for the company to dominate the next generation of gadgetry and keep people wedded to its ecosystem.

Apple has built a team combining the strengths of its hardware and software veterans with the expertise of talented outsiders, say the people, who requested anonymity to discuss internal strategy. Run by a former Dolby Laboratories executive, the group includes engineers who worked on the Oculus and HoloLens virtual reality headsets sold by Facebook and Microsoft as well as digital-effects wizards from Hollywood. Apple has also acquired several small firms with knowledge of AR hardware, 3D gaming and virtual reality software.

It's something they need to do to continue to grow and defend against the shift in how people use hardware.

As previously reported by Bloomberg, Apple is working on several AR products, including digital spectacles that could connect wirelessly to an iPhone and beam content — movies, maps and more — to the wearer. While the glasses are a ways off, AR features could show up in the iPhone sooner. Apple declined to comment.

It's an auspicious moment for Apple to move into augmented reality. The global market for AR products will surge 80 percent to $165 billion by 2024, according to researcher Global Market Insights. But Apple really has no choice, says Gene Munster, a founding partner at Loup Ventures who covered the company for many years as an analyst.

Over time, Munster says, AR devices will replace the iPhone. "It's something they need to do to continue to grow," he says, "and defend against the shift in how people use hardware."

Augmented reality is the less known cousin of virtual reality. VR gets more attention because it completely immerses users in an artificial world and has an obvious attraction for gamers. So far, however, headsets like the Oculus and HoloLens are niche rather than mainstream products. Apple believes AR will be an easier sell because it's less intrusive. Referring to VR headsets, Cook last year said he thought few people will want to be "enclosed in something."

Building a successful AR product will be no easy task, even for a company known for slim, sturdy devices. The current crop of AR glasses are either under-powered and flimsy or powerful and overwhelmingly large. Apple, the king of thin and light, will have to leapfrog current products by launching something small and powerful.

Adding AR features to the iPhone isn't a giant leap. Building glasses will be harder. Like the Watch, they'll probably be tethered to the iPhone. While the smartphone will do the heavy lifting, beaming 3D content to the glasses will consume a lot of power, so prolonging battery life will be crucial. Content is key too. If Apple's AR glasses lack useful apps, immersive games and interesting media content, why would someone wear them? The glasses will also require a new operating system and perhaps even a new chip. Finally, Apple will have to source the guts of the gadget cheaply enough to make it affordable for the mass market.

When it was developing the Watch, Apple put together a multi-disciplinary team drawn from inside and outside the company. It has done much the same with the AR effort. In 2015, Apple recruited Mike Rockwell, who previously ran the hardware and new technologies groups at Dolby, the iconic company known for its audio and video technology. Rockwell also advised Meta, a small firm that makes $950 AR glasses and counts Dolby as an investor.

Rockwell now runs the main AR team at Apple, reporting to Dan Riccio, who's in charge of the iPhone and iPad hardware engineering groups, the people said. "He's a really sharp guy," says Jack McCauley, who co-founded and worked at Oculus before it was sold to Facebook in 2015. "He could certainly put a team together that could get an Apple AR project going."

Apple CEO Tim Cook on Feb. 7 — Photo: Bernd Thissen/DPA via ZUMA

Last spring, in a sign that it's serious about taking products to market, Apple put some of its best hardware and software people on Rockwell's team, including Fletcher Rothkopf who helped lead the team that designed the Apple Watch, and Tomlinson Holman, who created THX, the audio standard made popular by Lucasfilm.

Apple has also recruited people with expertise in everything from 3D video production to wearable hardware. Among them, the people say: Cody White, former lead engineer of Amazon's Lumberyard virtual reality platform; Duncan McRoberts, Meta's former director of software development; Yury Petrov, a former Oculus researcher; and Avi Bar-Zeev, who worked on the HoloLens and Google Earth.

Apple has rounded out the team with iPhone, camera and optical lens engineers. There are people with experience in sourcing the raw materials for the glasses. The company has also mined the movie industry's 3D animation ranks, the people said, opening a Wellington office and luring several employees from Weta Digital, the New Zealand special-effects shop that worked on King Kong, Avatar and other films.

Besides hiring people, Apple has been busy making tactical acquisitions. In 2015, the company acquired Metaio, which developed AR software. Former Metaio CEO Thomas Alt now works on Apple's strategic deals team, which decides which technologies to invest in. Last year, Apple also bought FlyBy Media, which makes AR-related camera software. Cook even visited the offices of Magic Leap last summer and displayed interest in the secretive company's AR technology, the people say. Magic Leap declined to comment.

To be successful in AR, there is the hardware piece, but you have to do other stuff too: from maps to social to payments. Apple is one of the only companies that will be able to pull it off.

Hundreds of engineers are now devoted to the cause, including some on the iPhone camera team who are working on AR-related features for the iPhone, according to one of the people. One of the features Apple is exploring is the ability to take a picture and then change the depth of the photograph or the depth of specific objects in the picture later; another would isolate an object in the image, such as a person's head, and allow it to be tilted 180 degrees.

A different feature in development would use augmented reality to place virtual effects and objects on a person, much the way Snapchat works. The iPhone camera features would probably rely on a technology known as depth sensing and use algorithms created by PrimeSense, an Israeli company acquired in 2013. Apple may choose to not roll out these features, but such additions are an up-and-coming trend in the phone business.

The AR-enhanced glasses are further down the road, the people say. Getting the product right will be key, of course. Wearables are hard. Apple's first stab at the category, the Watch, has failed to become a mainstream hit. And no one has forgotten Google Glass, the much-derided headset that bombed in 2014. Still, time and again, Apple has waited for others to go first and then gone on to dominate the market. "To be successful in AR, there is the hardware piece, but you have to do other stuff too: from maps to social to payments," Munster says. "Apple is one of the only companies that will be able to pull it off."



*Bloomberg's Alex Webb and Mark Bergen contributed to this article.

Keep up with the world. Break out of the bubble.
Sign up to our expressly international daily newsletter!
Society

Debt Trap: Why South Korean Economics Explains Squid Game

Crunching the numbers of South Korea's personal and household debt offers a glimpse into what drives the win-or-die plot of the Netflix hit produced in the Asian country.

In the Netflix series, losers of the game face death

Yip Wing Sum

-Analysis-

SEOUL — The South Korean series Squid Game has become the most viewed series on Netflix, watched by over 111 million viewers and counting. It has also generated a wave of debate online and off about its provocative message about contemporary life.

The plot follows the story of a desperate man in debt, who receives a mysterious invitation to play a game in which the contestants gamble their lives on six childhood games, with the winner awarded a prize of 45.6 billion won ($38 million)... while the losers face death.


It's a plot that many have noted is not quite as surreal as it sounds, a reflection of the reality of Korean society today mired in personal debt.

Seoul housing prices top London and New York

In the polished streets of downtown Seoul, one sees endless cards and coupons advertising loans scattered on the ground. Since the outbreak of the pandemic, as the demand for loans in South Korea has exploded, lax lending policies have led to a rapid increase in personal debt.

According to the South Korean Central Bank's "Monetary Credit Policy Report," household debt reached 105% of GDP in the first quarter of this year, equivalent to approximately $1.5 trillion at the end of March, with a major share tied up in home mortgages.

Average home loans are equivalent to 270% of annual income.

One reason behind the debts is the soaring housing prices. In Seoul, home to nearly half of the country's population, housing prices are now among the highest in the world. The price to income ratio (PIR), which weighs the average price of a home to the average annual household income, is 12.04 in Seoul, compared to 8.4 in San Francisco, 8.2 in London and 5.4 in New York.

According to the Korea Real Estate Commission, 42.1% of all home purchases in January 2021 were by young Koreans in their 20s and 30s. For those in their 30s, the average amount borrowed is equivalent to 270% of their annual income.

Playing the stock market

At the same time, the South Korean stock market is booming. The increased demand to buy stocks has led to an increase in other loans such as credit. The ratio for Korean shareholders conducting credit financing, i.e. borrowing from securities companies to secure stock holdings, had reached 21.4 trillion won ($17.7 billion), further increasing the indebtedness of households.

A 30-year-old Seoul office worker who bought stocks through various forms of borrowing was interviewed by Reuters this year, and said he was "very foolish not to take advantage of the rebound."

In addition to his 100 million won ($84,000) overdraft account, he also took out a 100 million won loan against his house in Seoul, and a 50 million won stock pledge. All of these demands on the stock market have further exacerbated the problem of household debt.

42.1% of all home purchases in January 2021 were by young Koreans in their 20s and 30s

Simon Shin/SOPA Images/ZUMA

Game of survival

In response to the accumulating financial risks, the Bank of Korea has restricted the release of loans and has announced its first interest rate hike in three years at the end of August.

But experts believe that even if banks cut loans or raise interest rates, those who need money will look for other ways to borrow, often turning to more costly institutions and mechanisms.

This all risks leading to what one can call a "debt trap," one loan piling on top of another. That brings us back to the plot of Squid Game, "Either you live or I do." South Korean society has turned into a game of survival.

Keep up with the world. Break out of the bubble.
Sign up to our expressly international daily newsletter!
THE LATEST
FOCUS
TRENDING TOPICS
MOST READ