Multimodal Machine Learning Lab
Description
Multimodal machine learning concerns the research and development of learning methods that can process data provided in different types (i.e., modalities) such as images, videos, text, and audio. Key challenges include:
- learning vector representations shared across modalities (e.g., a text describing an image and the image itself should have similar representations),
- inferring new knowledge from multimodal information (multimodal inference),
- generating new texts, images, etc.,
- transferring knowledge between modalities, and
- the empirical analysis of multimodal models.
The project examines current research methods for problems related to these challenges.