1 code implementation • 29 Jan 2024 • Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, Yang You
To help the open-source community have a better understanding of Mixture-of-Experts (MoE) based large language models (LLMs), we train and release OpenMoE, a series of fully open-sourced and reproducible decoder-only MoE LLMs, ranging from 650M to 34B parameters and trained on up to over 1T tokens.
no code implementations • 22 Oct 2023 • Rui Mao, Kai He, Xulang Zhang, Guanyi Chen, Jinjie Ni, Zonglin Yang, Erik Cambria
We connect the surveyed tasks with downstream applications because this may inspire future scholars to fuse these low-level semantic processing tasks with high-level natural language processing tasks.
2 code implementations • 22 May 2023 • Jinjie Ni, Rui Mao, Zonglin Yang, Han Lei, Erik Cambria
Specifically, the heads of MHA were originally designed to attend to information from different representation subspaces, whereas prior studies found that some attention heads likely learn similar features and can be pruned without harming performance.
1 code implementation • 21 Mar 2023 • Zonglin Yang, Xinya Du, Rui Mao, Jinjie Ni, Erik Cambria
This paper provides a comprehensive overview on a new paradigm of logical reasoning, which uses natural language as knowledge representation and pretrained language models as reasoners, including philosophical definition and categorization of logical reasoning, advantages of the new paradigm, benchmarks and methods, challenges of the new paradigm, possible future directions, and relation to related NLP fields.
no code implementations • 7 Mar 2023 • Jinjie Ni, Yukun Ma, Wen Wang, Qian Chen, Dianwen Ng, Han Lei, Trung Hieu Nguyen, Chong Zhang, Bin Ma, Erik Cambria
Learning on a massive amount of speech corpus leads to the recent success of many self-supervised speech models.
no code implementations • 2 Sep 2022 • Jiaxing Xu, Jinjie Ni, Sophi Shilpa Gururajapathy, Yiping Ke
In this paper, we propose a Class-Aware Representation rEfinement (CARE) framework for the task of graph classification.
1 code implementation • 9 Sep 2021 • Tom Young, Frank Xing, Vlad Pandelea, Jinjie Ni, Erik Cambria
It features inter-mode contextual dependency, i. e., the dialogue turns from the two modes depend on each other.
Ranked #1 on Dialogue Generation on FusedChat
no code implementations • 10 May 2021 • Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, Erik Cambria
To the best of our knowledge, this survey is the most comprehensive and up-to-date one at present for deep learning based dialogue systems, extensively covering the popular techniques.
1 code implementation • 27 Dec 2020 • Fuzhao Xue, Aixin Sun, Hao Zhang, Jinjie Ni, Eng Siong Chng
Dialogue relation extraction (RE) is to predict the relation type of two entities mentioned in a dialogue.
Ranked #9 on Dialog Relation Extraction on DialogRE