Publications

2023

1. 

   LALM: Long-Term Action Anticipation with Language Models

   Sanghwan Kim, Daoji Huang, Yongqin Xian, and 3 more authors

   arXiv preprint arXiv:2311.17944, 2023

   arXiv Abs Bib

   Understanding human activity is a crucial yet intricate task in egocentric vision, a field that focuses on capturing visual perspectives from the camera wearer’s viewpoint. While traditional methods heavily rely on representation learning trained on extensive video data, there exists a significant limitation: obtaining effective video representations proves challenging due to the inherent complexity and variability in human activities.Furthermore, exclusive dependence on video-based learning may constrain a model’s capability to generalize across long-tail classes and out-of-distribution scenarios. In this study, we introduce a novel approach for long-term action anticipation using language models (LALM), adept at addressing the complex challenges of long-term activity understanding without the need for extensive training. Our method incorporates an action recognition model to track previous action sequences and a vision-language model to articulate relevant environmental details. By leveraging the context provided by these past events, we devise a prompting strategy for action anticipation using large language models (LLMs). Moreover, we implement Maximal Marginal Relevance for example selection to facilitate in-context learning of the LLMs. Our experimental results demonstrate that LALM surpasses the state-of-the-art methods in the task of long-term action anticipation on the Ego4D benchmark. We further validate LALM on two additional benchmarks, affirming its capacity for generalization across intricate activities with different sets of taxonomies. These are achieved without specific fine-tuning.

   @article{kim2023lalm,
     title = {LALM: Long-Term Action Anticipation with Language Models},
     author = {Kim, Sanghwan and Huang, Daoji and Xian, Yongqin and Hilliges, Otmar and Van Gool, Luc and Wang, Xi},
     journal = {arXiv preprint arXiv:2311.17944},
     year = {2023},
   }

2. 

   Distilling ODE Solvers of Diffusion Models into Smaller Steps

   Sanghwan Kim, Hao Tang, and Fisher Yu

   arXiv preprint arXiv:2309.16421, 2023

   arXiv Abs Bib

   Distillation techniques have substantially improved the sampling speed of diffusion models, allowing of the generation within only one step or a few steps. However, these distillation methods require extensive training for each dataset, sampler, and network, which limits their practical applicability. To address this limitation, we propose a straightforward distillation approach, Distilled-ODE solvers (D-ODE solvers), that optimizes the ODE solver rather than training the denoising network. D-ODE solvers are formulated by simply applying a single parameter adjustment to existing ODE solvers. Subsequently, D-ODE solvers with smaller steps are optimized by ODE solvers with larger steps through distillation over a batch of samples. Our comprehensive experiments indicate that D-ODE solvers outperform existing ODE solvers, including DDIM, PNDM, DPM-Solver, DEIS, and EDM, especially when generating samples with fewer steps. Our method incur negligible computational overhead compared to previous distillation techniques, enabling simple and rapid integration with previous samplers. Qualitative analysis further shows that D-ODE solvers enhance image quality while preserving the sampling trajectory of ODE solvers.

   @article{kim2023distilling,
     title = {Distilling ODE Solvers of Diffusion Models into Smaller Steps},
     author = {Kim, Sanghwan and Tang, Hao and Yu, Fisher},
     journal = {arXiv preprint arXiv:2309.16421},
     year = {2023},
   }

3. 

   Boosting Radiology Report Generation by Infusing Comparison Prior

   Sanghwan Kim, Farhad Nooralahzadeh, Morteza Rohanian, and 5 more authors

   The Association for Computational Linguistics (ACL’23), BioNLP Workshop, 2023

   arXiv Abs Bib

   Recent transformer-based models have made significant strides in generating radiology reports from chest X-ray images. However, a prominent challenge remains: these models often lack prior knowledge, resulting in the generation of synthetic reports that mistakenly reference non-existent prior exams. This discrepancy can be attributed to a knowledge gap between radiologists and the generation models. While radiologists possess patient-specific prior information, the models solely receive X-ray images at a specific time point. To tackle this issue, we propose a novel approach that leverages a rule-based labeler to extract comparison prior information from radiology reports. This extracted comparison prior is then seamlessly integrated into state-of-the-art transformer-based models, enabling them to produce more realistic and comprehensive reports. Our method is evaluated on English report datasets, such as IU X-ray and MIMIC-CXR. The results demonstrate that our approach surpasses baseline models in terms of natural language generation metrics. Notably, our model generates reports that are free from false references to non-existent prior exams, setting it apart from previous models. By addressing this limitation, our approach represents a significant step towards bridging the gap between radiologists and generation models in the domain of medical report generation.

   @article{kim2023boosting,
     title = {Boosting Radiology Report Generation by Infusing Comparison Prior},
     author = {Kim, Sanghwan and Nooralahzadeh, Farhad and Rohanian, Morteza and Fujimoto, Koji and Nishio, Mizuho and Sakamoto, Ryo and Rinaldi, Fabio and Krauthammer, Michael},
     journal = {The Association for Computational Linguistics (ACL’23), BioNLP Workshop},
     year = {2023},
   }

4. 

   Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learning

   Sanghwan Kim, Lorenzo Noci, Antonio Orvieto, and 1 more author

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023

   arXiv Abs Bib

   In contrast to the natural capabilities of humans to learn new tasks in a sequential fashion, neural networks are known to suffer from catastrophic forgetting, where the model’s performances on old tasks drop dramatically after being optimized for a new task. Since then, the continual learning (CL) community has proposed several solutions aiming to equip the neural network with the ability to learn the current task (plasticity) while still achieving high accuracy on the previous tasks (stability). Despite remarkable improvements, the plasticity-stability trade-off is still far from being solved and its underlying mechanism is poorly understood. In this work, we propose Auxiliary Network Continual Learning (ANCL), a novel method that applies an additional auxiliary network which promotes plasticity to the continually learned model which mainly focuses on stability. More concretely, the proposed framework materializes in a regularizer that naturally interpolates between plasticity and stability, surpassing strong baselines on task incremental and class incremental scenarios. Through extensive analyses on ANCL solutions, we identify some essential principles beneath the stability-plasticity trade-off.

   @inproceedings{kim2023achieving,
     title = {Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learning},
     author = {Kim, Sanghwan and Noci, Lorenzo and Orvieto, Antonio and Hofmann, Thomas},
     booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
     pages = {11930--11939},
     year = {2023},
   }

2020

1. 

   Learning Boost by Exploiting the Auxiliary Task in Multi-task Domain

   Jonghwa Yim, and Sanghwan Kim

   arXiv preprint arXiv:2008.02043, 2020

   arXiv Abs Bib

   Learning two tasks in a single shared function has some benefits. Firstly by acquiring information from the second task, the shared function leverages useful information that could have been neglected or underestimated in the first task. Secondly, it helps to generalize the function that can be learned using generally applicable information for both tasks. To fully enjoy these benefits, Multi-task Learning (MTL) has long been researched in various domains such as computer vision, language understanding, and speech synthesis. While MTL benefits from the positive transfer of information from multiple tasks, in a real environment, tasks inevitably have a conflict between them during the learning phase, called negative transfer. The negative transfer hampers function from achieving the optimality and degrades the performance. To solve the problem of the task conflict, previous works only suggested partial solutions that are not fundamental, but ad-hoc. A common approach is using a weighted sum of losses. The weights are adjusted to induce positive transfer. Paradoxically, this kind of solution acknowledges the problem of negative transfer and cannot remove it unless the weight of the task is set to zero. Therefore, these previous methods had limited success. In this paper, we introduce a novel approach that can drive positive transfer and suppress negative transfer by leveraging class-wise weights in the learning process. The weights act as an arbitrator of the fundamental unit of information to determine its positive or negative status to the main task.

   @article{yim2020learning,
     title = {Learning Boost by Exploiting the Auxiliary Task in Multi-task Domain},
     author = {Yim, Jonghwa and Kim, Sanghwan},
     journal = {arXiv preprint arXiv:2008.02043},
     year = {2020},
   }