Search Results for author:
Found 44 papers, 26 papers with code
Character Grounding and Re-Identification in Story of Videos and Text Descriptions
We address character grounding and re-identification in multiple story-based videos like movies and associated text descriptions.
Language Models as Compilers: Simulating Pseudocode Execution Improves Algorithmic Reasoning in Language Models
Also, we show that compared to natural language, pseudocode can better guide the reasoning of LMs, even though they are trained to follow natural language instructions.
HyperCLOVA X Technical Report
We introduce HyperCLOVA X, a family of large language models (LLMs) tailored to the Korean language and culture, along with competitive capabilities in English, math, and coding.
Pearl: A Review-driven Persona-Knowledge Grounded Conversational Recommendation Dataset
Our experimental results demonstrate that utterances in PEARL include more specific user preferences, show expertise in the target domain, and provide recommendations more relevant to the dialogue context than those in prior datasets.
Can Large Language Models be Good Emotional Supporter? Mitigating Preference Bias on Emotional Support Conversation
Motivated by these, we explore the impact of the inherent preference in LLMs on providing emotional support, and consequently, we observe that exhibiting high preference for specific strategies hinders effective emotional support, aggravating its robustness in predicting the appropriate strategy.
Aligning Large Language Models by On-Policy Self-Judgment
Existing approaches for aligning large language models with human preferences face a trade-off that requires a separate reward model (RM) for on-policy learning.
Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback
Recent advancements in large language models have influenced the development of video large multimodal models (VLMMs).
SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Models
We introduce this new task to explain why people laugh in a particular video and a dataset for this task.
Localized Symbolic Knowledge Distillation for Visual Commonsense Models
Empirical results and human evaluations in a zero-shot setup demonstrate that our distillation method results in more precise VL models of reasoning compared to a baseline of passing a generated referring expression to an LLM.
Long Story Short: a Summarize-then-Search Method for Long Video Question Answering
Large language models such as GPT-3 have demonstrated an impressive capability to adapt to new tasks without requiring task-specific training data.
Ranked #1 on
Video Question Answering (Level 4)
on DramaQA
Reading Books is Great, But Not if You Are Driving! Visually Grounded Reasoning about Defeasible Commonsense Norms
NORMLENS consists of 10K human judgments accompanied by free-form explanations covering 2K multimodal situations, and serves as a probe to address two questions: (1) to what extent can models align with average human judgment?
VLIS: Unimodal Language Models Guide Multimodal Language Generation
Multimodal language generation, which leverages the synergy of language and vision, is a rapidly expanding field.
Dialogue Chain-of-Thought Distillation for Commonsense-aware Conversational Agents
Hence, our focus is to facilitate such multi-hop reasoning over a dialogue context, namely dialogue chain-of-thought (CoT) reasoning.
Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step
We release our corpus of chain-of-thought samples and code.
CLARA: Classifying and Disambiguating User Commands for Reliable Interactive Robotic Agents
In this paper, we focus on inferring whether the given user command is clear, ambiguous, or infeasible in the context of interactive robotic agents utilizing large language models (LLMs).
Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text
We release Multimodal C4, an augmentation of the popular text-only C4 corpus with images interleaved.
CHAMPAGNE: Learning Real-world Conversation from Large-Scale Web Videos
To train CHAMPAGNE, we collect and release YTD-18M, a large-scale corpus of 18M video-based dialogues.
Fusing Pre-Trained Language Models With Multimodal Prompts Through Reinforcement Learning
Language models are capable of commonsense reasoning: while domain-specific models can learn from explicit knowledge (e. g. commonsense graphs [6], ethical norms [25]), and larger models like GPT-3 manifest broad commonsense reasoning capacity.
SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization
Data scarcity has been a long standing issue in the field of open-domain social dialogue.
Learning Joint Representation of Human Motion and Language
In this work, we present MoLang (a Motion-Language connecting model) for learning joint representation of human motion and language, leveraging both unpaired and paired datasets of motion and language modalities.
Zero-shot Active Visual Search (ZAVIS): Intelligent Object Search for Robotic Assistants
In this paper, we focus on the problem of efficiently locating a target object described with free-form language using a mobile robot equipped with vision sensors (e. g., an RGBD camera).
ProsocialDialog: A Prosocial Backbone for Conversational Agents
With this dataset, we introduce a dialogue safety detection module, Canary, capable of generating RoTs given conversational context, and a socially-informed dialogue agent, Prost.
Ranked #1 on
Dialogue Safety Prediction
on ProsocialDialog
Multimodal Knowledge Alignment with Reinforcement Learning
Large language models readily adapt to novel settings, even without task-specific training data.
MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound
Given a video, we replace snippets of text and audio with a MASK token; the model learns by choosing the correct masked-out snippet.
Ranked #6 on
Action Classification
on Kinetics-600
(using extra training data)
NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics
To enable constrained generation, we build on NeuroLogic decoding (Lu et al., 2021), combining its flexibility in incorporating logical constraints with A*esque estimates of future constraint satisfaction.
Ranked #1 on
Text Generation
on ROCStories
Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer
In a difficult zero-shot setting with no paired audio-text data, our model demonstrates state-of-the-art zero-shot performance on the ESC50 and US8K audio classification tasks, and even surpasses the supervised state of the art for Clotho caption retrieval (with audio queries) by 2. 2\% R@1.
Pano-AVQA: Grounded Audio-Visual Question Answering on 360$^\circ$ Videos
However, previous benchmark tasks for panoramic videos are still limited to evaluate the semantic understanding of audio-visual relationships or spherical spatial property in surroundings.
Cycled Compositional Learning between Images and Text
Since this one-way mapping is highly under-constrained, we couple it with an inverse relation learning with the Correction Network and introduce a cycled relation for given Image We participate in Fashion IQ 2020 challenge and have won the first place with the ensemble of our model.
Transitional Adaptation of Pretrained Models for Visual Storytelling
In this work, we claim that a transitional adaptation task is required between pretraining and finetuning to harmonize the visual encoder and the language model for challenging downstream target tasks like visual storytelling.
Ranked #1 on
Visual Storytelling
on VIST
(ROUGE-L metric, using extra
training data)
MERLOT: Multimodal Neural Script Knowledge Models
As humans, we understand events in the visual world contextually, performing multimodal reasoning across time to make inferences about the past, present, and future.
ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning
We demonstrate that our approach finds videos with high audio-visual correspondence and show that self-supervised models trained on our data achieve competitive performances compared to models trained on existing manually curated datasets.
Self-Supervised Learning of Compressed Video Representations
We show that our approach achieves competitive performance on self-supervised learning of video representations with a considerable improvement in speed compared to the traditional methods.
Pano-AVQA: Grounded Audio-Visual Question Answering on 360deg Videos
However, previous benchmark tasks for panoramic videos are still limited to evaluate the semantic understanding of audio-visual relationships or spherical spatial property in surroundings.
Parameter Efficient Multimodal Transformers for Video Representation Learning
The recent success of Transformers in the language domain has motivated adapting it to a multimodal setting, where a new visual model is trained in tandem with an already pretrained language model.
Augmenting Data for Sarcasm Detection with Unlabeled Conversation Context
We present a novel data augmentation technique, CRA (Contextual Response Augmentation), which utilizes conversational context to generate meaningful samples for training.
CurlingNet: Compositional Learning between Images and Text for Fashion IQ Data
In order to learn an effective image-text composition for the data in the fashion domain, our model proposes two key components as follows.
Ranked #16 on
Image Retrieval
on Fashion IQ
A Joint Sequence Fusion Model for Video Question Answering and Retrieval
We present an approach named JSFusion (Joint Sequence Fusion) that can measure semantic similarity between any pairs of multimodal sequence data (e. g. a video clip and a language sentence).
Ranked #34 on
Video Retrieval
on LSMDC
A Memory Network Approach for Story-Based Temporal Summarization of 360° Videos
Second, we evaluate the temporal summarization with a newly collected 360° video dataset.
A Memory Network Approach for Story-based Temporal Summarization of 360° Videos
Second, we evaluate the temporal summarization with a newly collected 360{\deg} video dataset.
A Deep Ranking Model for Spatio-Temporal Highlight Detection from a 360 Video
We address the problem of highlight detection from a 360 degree video by summarizing it both spatially and temporally.
Supervising Neural Attention Models for Video Captioning by Human Gaze Data
The attention mechanisms in deep neural networks are inspired by human's attention that sequentially focuses on the most relevant parts of the information over time to generate prediction output.
Encoding Video and Label Priors for Multi-label Video Classification on YouTube-8M dataset
YouTube-8M is the largest video dataset for multi-label video classification.
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
In this paper, we focus on extending VQA to the video domain and contribute to the literature in three important ways.
Ranked #30 on
Visual Question Answering (VQA)
on MSRVTT-QA
End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answering
We propose a high-level concept word detector that can be integrated with any video-to-language models.
Ranked #37 on
Video Retrieval
on LSMDC
