Abstract

Visual dialog (VisDial) is a task of answering a sequence of questions grounded in an image, using the dialog history as context. Prior work has trained the dialog agents solely on VisDial data via supervised learning or leveraged pre-training on related vision-and-language datasets. This paper presents a semi-supervised learning approach for visually-grounded dialog, called Generative Self-Training (GST), to leverage unlabeled images on the Web. Specifically, GST first retrieves in-domain images through out-of-distribution detection and generates synthetic dialogs regarding the images via multimodal conditional text generation. GST then trains a dialog agent on the synthetic and the original VisDial data. As a result, GST scales the amount of training data up to an order of magnitude that of VisDial (1.2M → 12.9M QA data). For robust training of the synthetic dialogs, we also propose perplexity-based data selection and multimodal consistency regularization. Evaluation on VisDial v1.0 and v0.9 datasets shows that GST achieves new state-of-the-art results on both datasets. We further observe the robustness of GST against both visual and textual adversarial attacks. Finally, GST yields strong performance gains in the low-data regime.

Video

Visualization of the synthetic data

The synthetic visual dialog data generated by our proposed models is shown below. The red-colored text denotes incorrect answers.

Experiments

The student model trained on the synthetic visual dialog data achieves a new state-of-the-art performance on VisDial v0.9 and v1.0 datasets. Furthermore, the student model significantly improves the robustness against adversarial attacks compared with the teacher model. The teacher model is our baseline model which is not trained on the synthetic data.

Citation

Acknowledgements

This work was supported by the SNU-NAVER Hyperscale AI Center and the Institute of Information & Communications Technology Planning & Evaluation (IITP) (2021-0-01343-GSAI/40%, 2022-0- 00953-PICA/30%, 2022-0-00951-LBA/20%, 2021-0-02068- AIHub/10%) grant funded by the Korean government.

The website template was borrowed from Michaël Gharbi and Ref-NeRF.