publications

2023

1. ConferencePreprint

   LoBaSS: Gauging Learnability in Supervised Fine-tuning Data

   Zhou, Haotian,  Liu, Tingkai, Ma, Qianli, Yuan, Jianbo, Liu, Pengfei, You, Yang, and Yang, Hongxia

   In arXiv, Oct 2023

   LLM Finetuning Training

   URL

   Supervised Fine-Tuning (SFT) serves as a crucial phase in aligning Large Language Models (LLMs) to specific task prerequisites. The selection of fine-tuning data profoundly influences the model’s performance, whose principle is traditionally grounded in data quality and distribution. In this paper, we introduce a new dimension in SFT data selection: learnability. This new dimension is motivated by the intuition that SFT unlocks capabilities acquired by a LLM during the pretraining phase. Given that different pretrained models have disparate capabilities, the SFT data appropriate for one may not suit another. Thus, we introduce the term learnability to define the suitability of data for effective learning by the model. We present the Loss Based SFT Data Selection (LoBaSS) method, utilizing data learnability as the principal criterion for the selection SFT data. This method provides a nuanced approach, allowing the alignment of data selection with inherent model capabilities, ensuring optimal compatibility and learning efficiency. In experimental comparisons involving 7B and 13B models, our LoBaSS method is able to surpass full-data fine-tuning at merely 6% of the total training data. When employing 16.7% of the data, LoBaSS harmonizes the model’s capabilities across conversational and mathematical domains, proving its efficacy and adaptability.

2. ConferencePreprint

   Let’s reward step by step: Step-Level reward model as the Navigators for Reasoning

   Ma, Qianli, Zhou, Haotian,  Liu, Tingkai, Yuan, Jianbo, Liu, Pengfei, You, Yang, and Yang, Hongxia

   In arXiv, Oct 2023

   LLM Reward-Model Inference

   URL

   Recent years have seen considerable advancements in multi-step reasoning with Large Language Models (LLMs). The previous studies have elucidated the merits of integrating feedback or search mechanisms during model inference to improve the reasoning accuracy. The Process-Supervised Reward Model (PRM), typically furnishes LLMs with step-by-step feedback during the training phase, akin to Proximal Policy Optimization (PPO) or reject sampling. Our objective is to examine the efficacy of PRM in the inference phase to help discern the optimal solution paths for multi-step tasks such as mathematical reasoning and code generation. To this end, we propose a heuristic greedy search algorithm that employs the step-level feedback from PRM to optimize the reasoning pathways explored by LLMs. This tailored PRM demonstrated enhanced results compared to the Chain of Thought (CoT) on mathematical benchmarks like GSM8K and MATH. Additionally, to explore the versatility of our approach, we develop a novel method to automatically generate step-level reward dataset for coding tasks and observed similar improved performance in the code generation tasks. Thus highlighting the robust nature of our reward-model-based approach to inference for reasoning tasks.

3. ConferencePreprint

   Video-CSR: Complex Video Digest Creation for Visual-Language Models

   Liu, Tingkai, Tao, Yunzhe, Liu, Haogeng, Fan, Qihang, Zhou, Ding, Huang, Huaibo, He, Ran, and Yang, Hongxia

   In arXiv, Oct 2023

   Multimodal Video-Language Benchmark

   URL

   We present a novel task and human annotated dataset for evaluating the ability for visual-language models to generate captions and summaries for real-world video clips, which we call Video-CSR (Captioning, Summarization and Retrieval). The dataset contains 4.8K YouTube video clips of 20-60 seconds in duration and covers a wide range of topics and interests. Each video clip corresponds to 5 independently annotated captions (1 sentence) and summaries (3-10 sentences). Given any video selected from the dataset and its corresponding ASR information, we evaluate visual-language models on either caption or summary generation that is grounded in both the visual and auditory content of the video. Additionally, models are also evaluated on caption- and summary-based retrieval tasks, where the summary-based retrieval task requires the identification of a target video given excerpts of a corresponding summary. Given the novel nature of the paragraph-length video summarization task, we perform extensive comparative analyses of different existing evaluation metrics and their alignment with human preferences. Finally, we propose a foundation model with competitive generation and retrieval capabilities that serves as a baseline for the Video-CSR task. We aim for Video-CSR to serve as a useful evaluation set in the age of large language models and complex multi-modal tasks.

4. ConferencePreprint

   Video-Teller: Enhancing Cross-Modal Generation with Fusion and Decoupling

   Liu, Haogeng, Fan, Qihang,  Liu, Tingkai, Yang, Linjie, Tao, Yunzhe, Huang, Huaibo, He, Ran, and Yang, Hongxia

   In arXiv, Oct 2023

   Multimodal LLM Video-Language Model

   URL

   This paper proposes Video-Teller, a video-language foundation model that leverages multi-modal fusion and fine-grained modality alignment to significantly enhance the video-to-text generation task. Video-Teller boosts the training efficiency by utilizing frozen pretrained vision and language modules. It capitalizes on the robust linguistic capabilities of large language models, enabling the generation of both concise and elaborate video descriptions. To effectively integrate visual and auditory information, Video-Teller builds upon the image-based BLIP-2 model and introduces a cascaded Q-Former which fuses information across frames and ASR texts. To better guide video summarization, we introduce a fine-grained modality alignment objective, where the cascaded Q-Former’s output embedding is trained to align with the caption/summary embedding created by a pretrained text auto-encoder. Experimental results demonstrate the efficacy of our proposed video-language foundation model in accurately comprehending videos and generating coherent and precise language descriptions. It is worth noting that the fine-grained alignment enhances the model’s capabilities (4% improvement of CIDEr score on MSR-VTT) with only 13% extra parameters in training and zero additional cost in inference.

5. ConferencePreprint

   SimVLG: Simple and Efficient Pretraining of Visual Language Generative Models

   Jian, Yiren,  Liu, Tingkai, Tao, Yunzhe, Vosoughi, Soroush, and Yang, Hongxia

   In arXiv, Oct 2023

   Multimodal LLM Image-Language Video-Language Model

   URL

   In this paper, we propose “SimVLG”, a streamlined framework for the pre-training of computationally intensive vision-language generative models, leveraging frozen pre-trained large language models (LLMs). The prevailing paradigm in vision-language pre-training (VLP) typically involves a two-stage optimization process: an initial resource-intensive phase dedicated to general-purpose vision-language representation learning, aimed at extracting and consolidating pertinent visual features, followed by a subsequent phase focusing on end-to-end alignment between visual and linguistic modalities. Our one-stage, single-loss framework circumvents the aforementioned computationally demanding first stage of training by gradually merging similar visual tokens during training. This gradual merging process effectively compacts the visual information while preserving the richness of semantic content, leading to fast convergence without sacrificing performance. Our experiments show that our approach can speed up the training of vision-language models by a factor ×5 without noticeable impact on the overall performance. Additionally, we show that our models can achieve comparable performance to current vision-language models with only 1/10 of the data. Finally, we demonstrate how our image-text models can be easily adapted to video-language generative tasks through a novel soft attentive temporal token merging modules.

6. Journal

   The functional logic of odor information processing in the Drosophila antennal lobe

   Lazar, Aurel A.,  Liu, Tingkai, and Yeh, Chung-Heng

   PLOS Computational Biology, Apr 2023

   Neuroscience Theory Olfaction

   URL

   Recent advances in molecular transduction of odorants in the Olfactory Sensory Neurons (OSNs) of the Drosophila Antenna have shown that the odorant object identity is multiplicatively coupled with the odorant concentration waveform. The resulting combinatorial neural code is a confounding representation of odorant semantic information (identity) and syntactic information (concentration). To distill the functional logic of odor information processing in the Antennal Lobe (AL) a number of challenges need to be addressed including 1) how is the odorant semantic information decoupled from the syntactic information at the level of the AL, 2) how are these two information streams processed by the diverse AL Local Neurons (LNs) and 3) what is the end-to-end functional logic of the AL? By analyzing single-channel physiology recordings at the output of the AL, we found that the Projection Neuron responses can be decomposed into a concentration-invariant component, and two transient components boosting the positive/negative concentration contrast that indicate onset/offset timing information of the odorant object. We hypothesized that the concentration-invariant component, in the multi-channel context, is the recovered odorant identity vector presented between onset/offset timing events. We developed a model of LN pathways in the Antennal Lobe termed the differential Divisive Normalization Processors (DNPs), which robustly extract the semantics (the identity of the odorant object) and the ON/OFF semantic timing events indicating the presence/absence of an odorant object. For real-time processing with spiking PN models, we showed that the phase-space of the biological spike generator of the PN offers an intuit perspective for the representation of recovered odorant semantics and examined the dynamics induced by the odorant semantic timing events. Finally, we provided theoretical and computational evidence for the functional logic of the AL as a robust ON-OFF odorant object identity recovery processor across odorant identities, concentration amplitudes and waveform profiles.

7. Patent

   Method and apparatus for encoding odorants

   Aurel A. Lazar, Chung-Heng Yeh, and Liu, Tingkai

   US Patent US11674937B2, Columbia Technology Ventures, Jun 2023

   NeuroScience Model Olfaction

   URL

2022

1. JournalPreprint

   Divisive Normalization Circuits Faithfully Represent Auditory and Visual Sensory Stimuli

   Lazar, Aurel A.,  Liu, Tingkai, and Zhou, Yiyin

   bioRxiv, Sep 2022

   Neuroscience Theory Audition Vision

   URL

   Divisive normalization is a canonical neural computation employed for sensory adaptation in vision, olfaction and attention modulation. While Divisive Normalization has been proposed to be an efficient coding algorithm, it remains unclear whether such transformation results in information loss in a dynamic setting. Leveraging a previously proposed general mathematical framework called the Divisive Normalization Processor (DNP), we first show that the DNP circuit describes a wide class of neural circuits as well as phenomenological models including the Linear-Nonlinear cascade model. We then demonstrate both theoretically and computationally that the DNP is an invertible operator that could faithfully represent input information given sufficient output samples.

2. JournalPreprint

   Odorant Mixture Separation in Drosophila Early Olfactory System

   Lazar, Aurel A.,  Liu, Tingkai, Yeh, C.-H., and Zhou, Yiyin

   bioRxiv, Sep 2022

   Neuroscience Theory Olfaction

   URL

   Natural odorant scenes are complex landscapes comprising mixtures of volatile compounds. It was previously proposed that the Antennal Lobe circuit recovers the odorant identity in a concentration-invariant manner via divisive normalization of Local Neurons. It remains unclear, however, how identities of odorant components in a mixture is represented or recovered in the fruit fly early olfactory pathway. In the current work, we take a different approach from the traditional steady-state analyses that classify odorant mixture encoding into configural vs. elemental schemes. Instead, we focus on the spatio-temporal responses of the early olfactory pathway at the levels of the Antennal Lobe and the Mushroom Body, and formulate the odorant demixing problem as a blind source separation problem - where the identities of each individual odorant component and their corresponding concentration waveforms are recovered from the spatio-temporal PSTH of Olfactory Sensory Neurons (OSNs), Projection Neurons (PNs), and Kenyon Cells (KCs) respectively. Building upon previous models of the Antenna and the Antennal Lobe, we advanced a feedback divisive normalization architecture of the Mushroom Body Calyx circuit comprised of PN, KC and the giant Anterior Paired Lateral (APL) neuron. We demonstrate that the PN-KC-APL circuit produces a high dimensional representation of odorant mixture with robust sparsity, and results in greater odorant demixing performance than the PN responses.

2021

1. Journal

   Accelerating with FlyBrainLab the Discovery of the Functional Logic of the Drosophila Brain in the Connectomic and Synaptomic Era

   Lazar, Aurel A.,  Liu, Tingkai, Turkcan, Mehmet K., and Zhou, Yiyin

   eLife, Feb 2021

   Neuroscience Tool

   URL

   In recent years, a wealth of Drosophila neuroscience data have become available including cell type and connectome/synaptome datasets for both the larva and adult fly. To facilitate integration across data modalities and to accelerate the understanding of the functional logic of the fruit fly brain, we have developed FlyBrainLab, a unique open-source computing platform that integrates 3D exploration and visualization of diverse datasets with interactive exploration of the functional logic of modeled executable brain circuits. FlyBrainLab’s User Interface, Utilities Libraries and Circuit Libraries bring together neuroanatomical, neurogenetic and electrophysiological datasets with computational models of different researchers for validation and comparison within the same platform. Seeking to transcend the limitations of the connectome/synaptome, FlyBrainLab also provides libraries for molecular transduction arising in sensory coding in vision/olfaction. Together with sensory neuron activity data, these libraries serve as entry points for the exploration, analysis, comparison, and evaluation of circuit functions of the fruit fly brain.

2. Conference

   FlyBrainLab: A Complete Programming Environment for Discovering the Functional Logic of the Fruit Fly Brain

   Lazar, Aurel A.,  Liu, Tingkai, Turkcan, Mehmet Kerem, and Zhou, Yiyin

   In Neuroinformatics Assembly 2021, Apr 2021

   Neuroscience Tool

2020

1. Conference

   FlyBrainLab: An Interactive Computing Platform for Exploring the Drosophila Brain

   Lazar, Aurel A.,  Liu, Tingkai, Turkcan, Mehmet K., and Zhou, Yiyin

   In PyData Global, Nov 2020

   NeuroScience Tool
2. Conference

   Comparative Study of Fruit Fly Brain Circuit Models with FlyBrainLab

   Lazar, Aurel A.,  Liu, Tingkai, Turkcan, Mehmet K., and Zhou, Yiyin

   In neuromatch 3.0, Oct 2020

   NeuroScience Tool Olfaction Vision
3. Conference

   FlyBrainLab: Interactive Computing in the Connectomic/Synaptomic Era

   Lazar, Aurel A.,  Liu, Tingkai, Turkcan, Mehmet K., and Zhou, Yiyin

   In Jupytercon, Oct 2020

   NeuroScience Tool
4. Conference

   Functional Identification of the Odorant Transduction Process of Drosophila Olfactory Sensory Neurons

   Lazar, Aurel A.,  Liu, Tingkai, and Zhou, Yiyin

   In BMC Neuroscience 2020, 21(Suppl 1):P225, Jul 2020

   NeuroScience Theory Olfaction
5. Conference

   Comparing Drosophila Neural Circuit Models with FlyBrainLab

   Lazar, Aurel A.,  Liu, Tingkai, Turkcan, Mehmet Kerem, and Zhou, Yiyin

   In BMC Neuroscience 2020, 21(Suppl 1):P112, Jul 2020

   NeuroScience Tool Olfaction Vision
6. Conference

   The Geometry of Spatio-Temporal Odorant Mixture Encoding in the Drosophila Mushroom Body

   Lazar, Aurel A.,  Liu, Tingkai, and Yeh, Chung-Heng

   In BMC Neuroscience 2020, 21(Suppl 1):218, Jul 2020

   NeuroScience Theory Olfaction
7. Conference

   An Odorant Encoding Machine for Sampling, Reconstruction and Robust Representation of Odorant Identity

   Lazar, Aurel A.,  Liu, Tingkai, and Yeh, Chung-Heng

   In ICASSP 2020, May 2020

   NeuroScience Theory Olfaction

   URL
8. Conference

   Time-Dependent Simultaneous Configural and Elemental Odorant Mixture Processing

   Liu, Tingkai, Yeh, Chung-Heng, and Lazar, Aurel A.

   In COSYNE*2020, Feb 2020

   NeuroScience Theory Olfaction

2019

1. JournalPreprint

   The Fruit Fly Brain Observatory: From Structure to Function

   Ukani, N. H., Yeh, C.-H., Tomkins, A., Zhou, Y., Florescu, D., Ortiz, C. L., Huang, Y.-C., Wang, C.-T., Turkcan, M. K.,  Liu, Tingkai, Richmond, P., Lo, C.-C., Coca, D., Chiang, A.-S., and Lazar, A. A.

   bioRxiv, Mar 2019

   Neuroscience Tool

   URL
2. Conference

   FlyBrainLab: An Interactive Computing Environment for the Fruit Fly Brain

   Turkcan, Mehmet K.,  Liu, Tingkai, Yeh, Chung-Heng, Zhou, Yiyin, and Lazar, Aurel A.

   In Society of Neuroscience, Oct 2019

   NeuroScience Tool
3. Conference

   Divisive Normalization Circuits Faithfully Represent Visual, Olfactory and Auditory Stimuli

   Lazar, Aurel A.,  Liu, Tingkai, and Zhou, Yiyin

   In BMC Neuroscience 2019, 20 (Suppl 1): P353, Oct 2019

   NeuroScience Theory Vision Olfaction Audition

2017

1. Conference

   An Open-Source Model of the Fruit Fly Larval Mushroom Body

   Lazar, Aurel A., and Liu, Tingkai

   In Neurobiology of Drosophila, Oct 2017

   NeuroScience Model Olfaction