To develop technologies for searching and explaining the real world, we are working across a broad spectrum of fields, from the development of multimodal learning methods based on foundation models and large language models, as well as their applications in robotics, image captioning, fashion, and art. We won first place for a multimodal instruction-following task at the CVPR 2023 Embodied AI Workshop.

The increasing demand for support services for older and disabled people has spurred the development of domestic service robots as an alternative and credible solution to the shortage of caring labor. We won RoboCup@Home twice, which is the world's largest competition for DSRs.

In autonomous driving, it would be convenient if users could operate mobility with natural language instructions, such as "Stop behind the blue van." We have developed Trimodal Navigable Region Segmentation Model capable of handling three modalities and have achieved superior instruction understanding performance compared to existing methods.

It is estimated that the economic loss from a Carrington-class solar flare will be approximately US $163 billion in North America. We developed the Flare Transformer, a solar flare prediction model that handles both line-of-sight magnetograms and physical features, and achieved super-human performance.

Visual explanations for deep neural networks are important in terms of enhancing accountability about image processing and providing scientific insight to experts.
We developed the Lambda Attention Branch Networks, an extension of Lambda Networks that improves the explainability, and outperformed existing explanation generation methods in terms of interpretability.

Early deaths from air pollutants (PM2.5, etc.) are estimated to be 3.3 million per year worldwide. However, it is still difficult to predict them accurately. We developed an air pollutant prediction method in 2014, which is one of the earliest introduction of deep neural networks into this field.

Cloud computing of robot functions enables low-cost, high-performance processing. We have developed a cloud robotics platform, Rospeex, which was used by more than 50,000 unique users in 2013-18.

Imitation learning provides a user-friendly interface to teach motions to robots. Our studies on human motion analysis and imitation learning can reduce cost for implementing robot motion.

Optimizing sensor placement on the robot's body and in the environment enables faster learning and more efficient information collection. We proposed a method that maximizes mutual information using submodularity.

There is a demand for recommendations of tourist spots based on ambiguous preferences. We have developed a method to recommend tourist spots based on ambiguous preferences. Our algorithm has been adopted by Kyoto City.

Many papers and data are available on the Web, but their search accuracy is very low compared to web pages. We developed spatio-temporal pseudo relevance feedback method and released it as a web service.