Yang et al. (2016) investigated a grounded version of semantic role labeling. They annotated a set of video clips about cooking for predicate-argument structures and linking textual description with visual objects.
- Yang, S., Gao, Q., Liu, C., Xiong, C., Zhu, S.-C., & Chai, J. Y. (2016). Grounded Semantic Role Labeling. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 149–159). San Diego, California: Association for Computational Linguistics. Retrieved from http://www.aclweb.org/anthology/N16-1019