Willy Chung

2025

1. VL-JEPA: Joint Embedding Predictive Architecture for Vision-Language

   Chen, Delong, Shukor, Mustafa, Moutakanni, Theo, Chung, Willy, Yu, Jade, Kasarla, Tejaswi, Bolourchi, Allen, LeCun, Yann, and Fung, Pascale

   arXiv preprint arXiv:2512.10942 2025

   Abs arXiv Bib PDF

   We introduce VL-JEPA, a vision-language model built on a Joint Embedding Predictive Architecture (JEPA). Instead of autoregressively generating tokens as in classical VLMs, VL-JEPA predicts continuous embeddings of the target texts. By learning in an abstract representation space, the model focuses on task-relevant semantics while abstracting away surface-level linguistic variability. In a strictly controlled comparison against standard token-space VLM training with the same vision encoder and training data, VL-JEPA achieves stronger performance while having 50% fewer trainable parameters. At inference time, a lightweight text decoder is invoked only when needed to translate VL-JEPA predicted embeddings into text. We show that VL-JEPA natively supports selective decoding that reduces the number of decoding operations by 2.85x while maintaining similar performance compared to non-adaptive uniform decoding. Beyond generation, the VL-JEPA’s embedding space naturally supports open-vocabulary classification, text-to-video retrieval, and discriminative VQA without any architecture modification. On eight video classification and eight video retrieval datasets, the average performance VL-JEPA surpasses that of CLIP, SigLIP2, and Perception Encoder. At the same time, the model achieves comparable performance as classical VLMs (InstructBLIP, QwenVL) on four VQA datasets: GQA, TallyQA, POPE and POPEv2, despite only having 1.6B parameters.

   @article{chen2025vl,
     title = {VL-JEPA: Joint Embedding Predictive Architecture for Vision-Language},
     author = {Chen, Delong and Shukor, Mustafa and Moutakanni, Theo and Chung, Willy and Yu, Jade and Kasarla, Tejaswi and Bolourchi, Allen and LeCun, Yann and Fung, Pascale},
     journal = {arXiv preprint arXiv:2512.10942},
     year = {2025},
   }

2. WorldPrediction: A Benchmark for High-level World Modeling and Long-horizon Procedural Planning

   Chen, Delong, Chung, Willy, Bang, Yejin, Ji, Ziwei, and Fung, Pascale

   arXiv preprint arXiv:2506.04363 2025

   Abs arXiv Bib PDF

   Humans are known to have an internal ’world model’ that enables us to carry out action planning based on world states. AI agents need to have such a world model for action planning as well. It is not clear how current AI models, especially generative models, are able to learn such world models and carry out procedural planning in diverse environments. We introduce WorldPrediction, a video-based benchmark for evaluating world modeling and procedural planning capabilities of different AI models. In contrast to prior benchmarks that focus primarily on low-level world modeling and robotic motion planning, WorldPrediction is the first benchmark that emphasizes actions with temporal and semantic abstraction. Given initial and final world states, the task is to distinguish the proper action (WorldPrediction-WM) or the properly ordered sequence of actions (WorldPrediction-PP) from a set of counterfactual distractors. This discriminative task setup enable us to evaluate different types of world models and planners and realize a thorough comparison across different hypothesis. The benchmark represents states and actions using visual observations. In order to prevent models from exploiting low-level continuity cues in background scenes, we provide ’action equivalents’ - identical actions observed in different contexts - as candidates for selection. This benchmark is grounded in a formal framework of partially observable semi-MDP, ensuring better reliability and robustness of the evaluation. We conduct extensive human filtering and validation on our benchmark and show that current frontier models barely achieve 57% accuracy on WorldPrediction-WM and 38% on WorldPrediction-PP whereas humans are able to solve both tasks perfectly.

   @article{chen2025worldprediction,
     title = {WorldPrediction: A Benchmark for High-level World Modeling and Long-horizon Procedural Planning},
     author = {Chen, Delong and Chung, Willy and Bang, Yejin and Ji, Ziwei and Fung, Pascale},
     journal = {arXiv preprint arXiv:2506.04363},
     year = {2025},
   }

2023

1. A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity

   Bang, Yejin, Cahyawijaya, Samuel, Lee, Nayeon, Dai, Wenliang, Su, Dan, Wilie, Bryan, Lovenia, Holy, Ji, Ziwei, Yu, Tiezheng, Chung, Willy, Do, Quyet V., Xu, Yan, and Fung, Pascale

   In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) Nov 2023

   Abs arXiv Bib PDF Code

   This paper proposes a framework for quantitatively evaluating interactive LLMs such as ChatGPT using publicly available data sets. We carry out an extensive technical evaluation of ChatGPT using 23 data sets covering 8 different common NLP application tasks. We evaluate the multitask, multilingual and multi-modal aspects of ChatGPT based on these data sets and a newly designed multimodal dataset. We find that ChatGPT outperforms LLMs with zero-shot learning on most tasks and even outperforms fine-tuned models on some tasks. We find that it is better at understanding non-Latin script languages than generating them. It is able to generate multimodal content from textual prompts, via an intermediate code generation step. Moreover, we find that ChatGPT is 63.41% accurate on average in 10 different reasoning categories under logical reasoning, non-textual reasoning, and commonsense reasoning, hence making it an unreliable reasoner. It is, for example, better at deductive than inductive reasoning. ChatGPT suffers from hallucination problems like other LLMs and it generates more extrinsic hallucinations from its parametric memory as it does not have access to an external knowledge base. Finally, the interactive feature of ChatGPT enables human collaboration with the underlying LLM to improve its performance, i.e, 8% ROUGE-1 on summarization and 2% ChrF++ on machine translation, in a multi-turn prompt engineering fashion. We also release codebase for evaluation set extraction.

   @inproceedings{bang-etal-2023-multitask,
     title = {A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity},
     author = {Bang, Yejin and Cahyawijaya, Samuel and Lee, Nayeon and Dai, Wenliang and Su, Dan and Wilie, Bryan and Lovenia, Holy and Ji, Ziwei and Yu, Tiezheng and Chung, Willy and Do, Quyet V. and Xu, Yan and Fung, Pascale},
     editor = {Park, Jong C. and Arase, Yuki and Hu, Baotian and Lu, Wei and Wijaya, Derry and Purwarianti, Ayu and Krisnadhi, Adila Alfa},
     booktitle = {Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
     month = nov,
     year = {2023},
     address = {Nusa Dua, Bali},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2023.ijcnlp-main.45},
     doi = {10.18653/v1/2023.ijcnlp-main.45},
     pages = {675--718},
   }

2. InstructTODS: Large Language Models for End-to-End Task-Oriented Dialogue Systems

   Chung, Willy, Cahyawijaya, Samuel, Wilie, Bryan, Lovenia, Holy, and Fung, Pascale

   In Proceedings of the Second Workshop on Natural Language Interfaces Nov 2023

   Abs arXiv Bib PDF Code

   Large language models (LLMs) have been used for diverse tasks in natural language processing (NLP), yet remain under-explored for task-oriented dialogue systems (TODS), especially for end-to-end TODS. We present InstructTODS, a novel off-the-shelf framework for zero-shot end-to-end task-oriented dialogue systems that can adapt to diverse domains without fine-tuning. By leveraging LLMs, InstructTODS generates a proxy belief state that seamlessly translates user intentions into dynamic queries for efficient interaction with any KB. Our extensive experiments demonstrate that InstructTODS achieves comparable performance to fully fine-tuned TODS in guiding dialogues to successful completion without prior knowledge or task-specific data. Furthermore, a rigorous human evaluation of end-to-end TODS shows that InstructTODS produces dialogue responses that notably outperform both the gold responses and the state-of-the-art TODS in terms of helpfulness, informativeness, and humanness. Moreover, the effectiveness of LLMs in TODS is further supported by our comprehensive evaluations on TODS subtasks: dialogue state tracking, intent classification, and response generation.

   @inproceedings{chung-etal-2023-instructtods,
     title = {InstructTODS: Large Language Models for End-to-End Task-Oriented Dialogue Systems},
     author = {Chung, Willy and Cahyawijaya, Samuel and Wilie, Bryan and Lovenia, Holy and Fung, Pascale},
     editor = {Chen, Kehai and Ku, Lun-Wei},
     booktitle = {Proceedings of the Second Workshop on Natural Language Interfaces},
     month = nov,
     year = {2023},
     address = {Bali, Indonesia},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2023.nlint-1.1},
     doi = {10.18653/v1/2023.nlint-1.1},
     pages = {1--21},
   }

3. Cross-Lingual Cross-Age Group Adaptation for Low-Resource Elderly Speech Emotion Recognition

   Cahyawijaya, Samuel*, Lovenia, Holy*, Chung, Willy*, Frieske, Rita, Liu, Zihan, and Fung, Pascale

   In INTERSPEECH 2023, 24th Annual Conference of the International Speech Communication Association, Dublin, Ireland, 20-24 August 2023 Nov 2023

   Abs arXiv Bib PDF Code

   Speech emotion recognition plays a crucial role in human-computer interactions. However, most speech emotion recognition research is biased toward English-speaking adults, which hinders its applicability to other demographic groups in different languages and age groups. In this work, we analyze the transferability of emotion recognition across three different languages–English, Mandarin Chinese, and Cantonese; and 2 different age groups–adults and the elderly. To conduct the experiment, we develop an English-Mandarin speech emotion benchmark for adults and the elderly, BiMotion, and a Cantonese speech emotion dataset, YueMotion. This study concludes that different language and age groups require specific speech features, thus making cross-lingual inference an unsuitable method. However, cross-group data augmentation is still beneficial to regularize the model, with linguistic distance being a significant influence on cross-lingual transferability.

   @inproceedings{cahyawijaya2023cross,
     title = {Cross-Lingual Cross-Age Group Adaptation for Low-Resource Elderly Speech Emotion Recognition},
     author = {Cahyawijaya, Samuel* and Lovenia, Holy* and Chung, Willy* and Frieske, Rita and Liu, Zihan and Fung, Pascale},
     year = {2023},
     booktitle = {INTERSPEECH 2023, 24th Annual Conference of the International Speech Communication Association, Dublin, Ireland, 20-24 August 2023},
     publisher = {ISCA},
   }

2022

1. Clozer: Adaptable Data Augmentation for Cloze-style Reading Comprehension

   Lovenia, Holy*, Wilie, Bryan*, Chung, Willy*, Min, Zeng*, Cahyawijaya, Samuel, Su, Dan, and Fung, Pascale

   In Proceedings of the 7th Workshop on Representation Learning for NLP Nov 2022

   Abs arXiv Bib PDF

   Task-adaptive pre-training (TAPT) alleviates the lack of labelled data and provides performance lift by adapting unlabelled data to downstream task. Unfortunately, existing adaptations mainly involve deterministic rules that cannot generalize well. Here, we propose Clozer, a sequence-tagging based cloze answer extraction method used in TAPT that is extendable for adaptation on any cloze-style machine reading comprehension (MRC) downstream tasks. We experiment on multiple-choice cloze-style MRC tasks, and show that Clozer performs significantly better compared to the oracle and state-of-the-art in escalating TAPT effectiveness in lifting model performance, and prove that Clozer is able to recognize the gold answers independently of any heuristics.

   @inproceedings{lovenia2022clozer,
     title = {Clozer: Adaptable Data Augmentation for Cloze-style Reading Comprehension},
     author = {Lovenia, Holy* and Wilie, Bryan* and Chung, Willy* and Min, Zeng* and Cahyawijaya, Samuel and Su, Dan and Fung, Pascale},
     booktitle = {Proceedings of the 7th Workshop on Representation Learning for NLP},
     pages = {60--66},
     year = {2022},
   }