Why is Japan Falling Behind in AI?
And What Can Be Done?
Japan built the world’s first bullet train and the world’s most famously futuristic cities. So why is it struggling with the technology that will shape the next century?
Before we begin, you’ll see that this newsletter has a new name. I hope you like it! See my pinned post HERE for more.
An enduring tourist-board cliché about Japan is that it’s the place where tradition meets high technology. Travellers on the so-called Golden Route are whisked from a hyper-modern megalopolis via futuristic-feeling shinkansen to the ancient shrines and temples of Kyoto. Yet the shinkansen began life as 1960s technology - as, for the most part, did Japan’s famously ubiquitous vending machines.
Japan now has a problem. It was a world leader in late twentieth-century tech thanks to companies like Toyota, Sony and Panasonic. But then the economic bubble burst, and the centre of gravity for innovation shifted to America’s west coast and towards venture capital, platform-based business models and ‘fail fast’ startups as a new form of R&D.
Across Japan’s political spectrum, there is agreement that the country needs a productivity boost, stronger economic growth and greater self-sufficiency. It also needs to catch up with the United States and China in embracing AI. Nearly 70% of Americans and over 80% of Chinese have used generative AI. In Japan the figure is just 26.7%.
The jury is very much still out on what AI might ultimately offer in terms of productivity gains versus social harm. But Japan’s new prime minister, Sanae Takaichi, is pursuing a policy of rapid catch-up nonetheless. That includes ramping up semiconductor manufacturing in Japan and reducing reliance on China for rare-earth elements. One sign that the current diplomatic spat between China and Japan may yet cool is that Beijing has not pulled one of its most powerful levers: cutting off Japan’s supply of processed rare-earths.
If China and Japan did ever co-operate on AI, interesting things might happen. Both face two interconnected challenges. The first is linguistic: English has long been the lingua franca of the web, giving English-speaking AI developers an enormous advantage in training large language models (LLMs). Around 50% of web content is in English. Japanese accounts for up to 5%, and Chinese for less than 2%. The sheer scale and variety of English-language data available - from science to the arts - puts Japan and China at a disadvantage.
The second problem for China and Japan is a fascinating one – and it’s particularly acute for Japan.
How do you train LLMs when Japanese uses three scripts – kanji, alongside two syllabaries (hiragana and katakana) – a wide array of honorifics and varied levels of politeness in everyday speech.
It also operates in everyday life in ways that are heavily dependent on context and implied meaning.
Anyone who has studied Japanese knows that textbook sentences bear little resemblance to everyday speech. A few well-chosen words deployed at the right moment tend to do a great deal of the heavy lifting. You have to intuit the context - and AI is not yet very good at that.
LLMs are mathematical at heart. They transform words and punctuation into ‘tokens,’ tracking logical references within sentences and repeated combinations across training data.
Take as an example the sentence: Mō owatta kara, hon o ageta. Literally: ‘Because already finished, gave book.’
What’s finished? Who gave whom a book?
If you’re in the room, you might easily get the meaning from context plus people’s facial expressions, who’s looking at whom, who’s holding a book, etc. For an AI system, building in these contextual expectations requires vast and carefully designed datasets.
No Japanese LLM has yet come close to challenging the dominance of ChatGPT, which since GPT-3 has been trained multilingually and has improved its handling of non-English languages with every iteration.
Ongoing feedback from Japanese users, along with technical innovations, helps it to adjust to Japanese conversational norms: avoiding over-directness, misjudged humour and misunderstandings of ever-evolving slang.
University students are some of the biggest users of AI in Japan
Aside from the difficulty of building LLMs in Japan, why does a society known for being comfortable with technology - untroubled by humanoid robots - rank near the bottom internationally in willingness to use AI?
A common claim is that Shinto and Buddhism encourage comfort with the idea that non-human entities possess spirit or agency, reducing the ‘uncanny valley’ effect whereby trust in human-like robots declines the more human they seem. This may help explain why Japanese people often interact with robots more naturally than people in countries shaped by Abrahamic religions.
So far, so good. The problem seems to be that people in Japan don’t yet trust institutions to use artificial intelligence usefully or responsibly.
Private companies like Rakuten, which have spent decades building trust with consumers, have found people willing to take a bet on innovations like autonomous mobile robots (AMRs) - trialled in Tokyo earlier this year for delivering goods. The state, by contrast, has a poor reputation on trust and therefore a long way to go if it wants to persuade people to engage more with AI.
A Rakuten mobile delivery robot
One approach being tried is for the government to fund small startups such as Sakana.ai or Preferred Networks Inc. Another is for Japanese companies to work with Asia-Pacific startups that are already designing multilingual systems with Japanese, Chinese, Korean and Thai in mind.
Clarity on regulation would help. Will Japan mirror European caution or adopt the full-throttle, all-guns-blazing American approach?
There are modest signs of progress. Vending machines are being equipped with deep-learning systems that recommend products based on past purchases or weather conditions; some adjust prices dynamically based on footfall and perishability. Smart, unmanned convenience stores are being trialled too, taking advantage of Japan’s low crime rates.
Visionary investors such as SoftBank’s Masayoshi Son are betting heavily on OpenAI. And Toyota’s ‘Woven City,’ at the foot of Mount Fuji, is set to become a testbed for next-generation technologies in which hundreds - and later thousands - of residents will live alongside the entrepreneurs developing them.
Still, Japan is not yet close to regaining the technological lead it enjoyed in the 1960s and 70s. Like politicians elsewhere, Japanese leaders have a record of promising ambitious digital reforms while delivering them slowly. But as the potential benefits of AI to an ageing and shrinking society become clearer - and the risks of falling further behind grow harder to ignore - the pressure to accelerate may finally become strong enough to force a long-hoped-for return to form.
Thank you reading! Please hit the heart button if you enjoyed this piece, and don’t forget to have your say in the survey above.
This post is available to share, so if you know someone who’s interested in Japan or AI, then feel free!
—
Images
Sanae Takaichi: BBC (fair use).
University students: LSMU (fair use).
Great overview of this issue Chris, it really is mystifying why Japan apparently lags behind so much.
It took my mind back to 2015 when I began discussing the then ‘revolutionary’ Japanese govt vision for a ‘Society 5.0’ - complete with kawaii video (live) story - with my students, when it was first launched.
https://youtu.be/BoEl9K2v2B0?si=bwxRM-3KfvGWcV67
It seemed to be the future laid out before us, but I doubt there are many Japanese people today who have even heard of it!
It does seem there are many fold problems, linguistic, technological and governmental.
But it’s a real conundrum as to why a concept as clearly articulated was not really taken up. Seems somehow to parallel the ‘cool Japan’ policy that flopped too.