Multimodal AI Lab
The Multimodal AI Solution for Your Language Learning

An AI-powered automated scoring and feedback system for real-life scene description tasks that apply multimodal information to accurately assess students’ language skills and provide immediate insights, enabling progressive improvement. The system incorporates level-specific adjustments, targeted at schools / teachers, which will solve the challenge of scalable and personalized language learning assessment. It has been successfully launched in December 2024. At the same time, our project was selected as the HKSTP Ideation Programme.

Lets Get Started →

What We Offer

Our goal and vision are to utilize AI technology to empower students to learn English self-directedly in authentic contexts through multisensory perception and interaction. Tailors feedback to different proficiency levels, helping different background student to build their language skills. Provides suggestions that meet students where they are, encouraging gradual improvement and building confidence in writing.

Authentic Situations & Daily-living Activities
Automatic Assessment & Timely Feedback
Multisensory Perception & Multimodal  Interaction

TRUST US

Cooperate with Schools in HKSAR and Mainland China

We have successfully cooperated with several middle schools in Hong Kong and Mainland China. Currently, over 22 schools are actively participating in this initiative.

The following are part of participating schools/association.

南京市揚子第一中學

Nanjing Yangzi No.1 Middle School

南京市第二十九中學

Nanjing No.29 Middle School

CHOOSE US

Why Make Us Your Trusted Partner

Choose us to leverage expertise and innovation for your language learning journey towards success.

Customized Multimodal Large Language Models for User Feedback
Provides  level-specific feedback to users.
Adapt to international and local school curricula.

Facilitating Self-directed Language Learning within Daily Life Scenarios
In daily life, AI can perceive real situations, engage in conversation with students, and provide timely feedback and responses.

Effective Learning Flow Design
Create tailored learning experiences to help students learn English effectively.

User-Friendly Interaction System
A user-friendly interface to interact with the Multimodal Large Language Models.

KNOWLEDGE CENTER

Latest Research & Blog

Zhao, R., Xie, Z., Zhuang, Y., Li, H., & Yu, P. L. H. (2024). Enhancing Language Learning through Multimodal AI-Driven Feedback on Picture Descriptions: An Eye-Tracking Study. In Proceedings of the 32nd International Conference on Computers in Education.

Zhuang, Y., Xie, Z. and Zhao, R. & Yu, P. L. H. (2024) Enhancing Language Learning through AI-Generated Feedback on Picture-Cued Writing. 21st AsiaCALL International Conference.

ZHAO, R., ZHUANG, Y., XIE, Z., & YU, L. H. P. (2024, April). AI-score: AI-assisted authentic language learning with automated feedback. In International Congress on Educational Futures 2024.

Zhao, R., Zhuang, Y., Xie, Z., & Philip, L. H. (2024). Facilitating self-directed language learning in real-life scene description tasks with automated evaluation. Computers & Education, 105106. (SCI / SSCI, Q1, IF 8.9)

Zhao, R*., Xie, Z*., Zhuang, Y., & Yu, P. L. H. (2023). Automated quality evaluation of large-scale benchmark datasets for vision-language tasks. International Journal of Neural Systems, 34(3), Article 2450009.  (* co-first authors, SCI,  Q1, IF 6.6)

Zhao, R., Zhuang, Y., Zou, D., Xie, Q., & Yu, P. L. (2023). AI-assisted automated scoring of picture-cued writing tasks for language assessment. Education and Information Technologies, 28(6), 7031-7063. (SCI / SSCI, Q1, IF 4.8)

Read All Articles →