Search Results for author:
Found 10 papers, 3 papers with code
Persona Extraction Through Semantic Similarity for Emotional Support Conversation Generation
We devise completeness loss and consistency loss based on semantic similarity scores.
Multilingual Visual Speech Recognition with a Single Model by Learning with Discrete Visual Speech Units
By using the visual speech units as the inputs of our system, we pre-train the model to predict corresponding text outputs on massive multilingual data constructed by merging several VSR databases.
AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation
To mitigate the problem of the absence of a parallel AV2AV translation dataset, we propose to train our spoken language translation system with the audio-only dataset of A2A.
DF-3DFace: One-to-Many Speech Synchronized 3D Face Animation with Diffusion
We contribute a new large-scale 3D facial mesh dataset, 3D-HDTF to enable the synthesis of variations in identities, poses, and facial motions of 3D face mesh.
Text-driven Talking Face Synthesis by Reprogramming Audio-driven Models
The visual speaker embedding is derived from a single target face image and enables improved mapping of input text to the learned audio latent space by incorporating the speaker characteristics inherent in the audio.
Exploring Phonetic Context-Aware Lip-Sync For Talking Face Generation
The contextualized lip motion unit then guides the latter in synthesizing a target identity with context-aware lip motion.
SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memory
It stores lip motion features from sequential ground truth images in the value memory and aligns them with corresponding audio features so that they can be retrieved using audio input at inference time.
Test-time Adaptation for Real Image Denoising via Meta-transfer Learning
We explore a different direction where we propose to improve real image denoising performance through a better learning strategy that can enable test-time adaptation on the multi-task network.
Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video
By learning the interrelationship through the associative bridge, the proposed bridging framework is able to obtain the target modal representations inside the memory network, even with the source modal input only, and it provides rich information for its downstream tasks.
Ranked #3 on
Lipreading
on CAS-VSR-W1k (LRW-1000)
Speech Reconstruction with Reminiscent Sound via Visual Voice Memory
Our key contributions are: (1) proposing the Visual Voice memory that brings rich information of audio that complements the visual features, thus producing high-quality speech from silent video, and (2) enabling multi-speaker and unseen speaker training by memorizing auditory features and the corresponding visual features.
