Our research activities encompass a wide range of basic and applied research areas, including machine intelligence, intelligent robotics, spoken language processing, and machine learning. Recent research topics include domestic service robots, multimodal language understanding, language generation, time series prediction, space and environmental data prediction, imitation learning, and recommendation systems.

1. Multimodal Language Processing

Multimodal Language Processing

To develop technologies for searching and explaining the real world, we are working across a broad spectrum of fields, from the development of multimodal learning methods based on foundation models and large language models, as well as their applications in robotics, image captioning, autonomous driving, fashion, and art. We won first place for a multimodal instruction-following task at the CVPR 2023 Embodied AI Workshop.

2. Domestic Service Robots

Domestic Service Robots

The increasing demand for support services for older and disabled people has spurred the development of domestic service robots as an alternative and credible solution to the shortage of caring labor. We won RoboCup@Home twice, which is the world's largest competition for DSRs.

3. Solar Flare Prediction

Solar Flare Prediction

It is estimated that the economic loss from a Carrington-class solar flare will be approximately US $163 billion in North America. We developed the Flare Transformer, a solar flare prediction model that handles both line-of-sight magnetograms and physical features, and achieved super-human performance.

4. Explainable AI

Explainable AI

Visual explanations for deep neural networks are important in terms of enhancing accountability about image processing and providing scientific insight to experts.
We developed the Lambda Attention Branch Networks, an extension of Lambda Networks that improves the explainability, and outperformed existing explanation generation methods in terms of interpretability.

5. Air Pollutants Prediction

Air Pollutants Prediction

Early deaths from air pollutants (PM2.5, etc.) are estimated to be 3.3 million per year worldwide. However, it is still difficult to predict them accurately. We developed an air pollutant prediction method in 2014, which is one of the earliest introduction of deep neural networks into this field.

6. Cloud Robotics

Cloud Robotics

Cloud computing of robot functions enables low-cost, high-performance processing. We have developed a cloud robotics platform, Rospeex, which was used by more than 50,000 unique users in 2013-18.

7. Imitation Learning and Motion Analysis

Imitation Learning and Motion Analysis

Imitation learning provides a user-friendly interface to teach motions to robots. Our studies on human motion analysis and imitation learning can reduce cost for implementing robot motion.

8. Optimal Sensor Placement

Optimal Sensor Placement

Optimizing sensor placement on the robot's body and in the environment enables faster learning and more efficient information collection. We proposed a method that maximizes mutual information using submodularity.

9. User Profiling and Recommendation

User Profiling and Recommendation

There is a demand for recommendations of tourist spots based on ambiguous preferences. We have developed a method to recommend tourist spots based on ambiguous preferences. Our algorithm has been adopted by Kyoto City.

10. Open Data Search System

Open Data Search System

Many papers and data are available on the Web, but their search accuracy is very low compared to web pages. We developed spatio-temporal pseudo relevance feedback method and released it as a web service.