Seung Hyun Lee

Ph.D. Student

University of Michigan

CV | Google Scholar | |

Starting in Fall 2024, I will join Prof. Stella Yu's group at the University of Michigan as Ph.D. student.

Previously, I got master degree in the Department of Aritificial Intelligence at the Korea University. I interned at Naver Papago, Yonsei Severance CCIDS.

I also worked as Google Student Researcher (MTV, CA).

My research interests lie in the intersection field of computer vision, especially on the tasks of multi-modal generative models, artificial general intelligence, sound-guided visual syntheis, multimodal representation learning.


  • seungle [at] umich [dot] edu

  • easter3163 [at] korea [dot] ac [dot] kr


  • Ph.D. in CSE, 2024~

    University of Michigan

  • MS in Artificial Intelligence, 2022~2024

    Korea University, Korea

  • BS in Computer Science, 2016~2022

    University of Seoul, Korea

Work Experiences

  • Google Research, MTV, CA
    Jul 2023 - Mar 2024

    Student Researcher
  • Naver Papago, Korea
    Dec 2020 - Jun 2021

    Research Intern, Computer Vision

Selected Publications

    @ Google Research

  • Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for Text-to-Image Generation

    Seung Hyun Lee, Yinxiao Li, Junjie Ke, Innfarn Yoo, Han Zhang, Jiahui Yu, Qifei Wang

    Fei Deng, Glenn Entis, Junfeng He, Gang Li, Sangpil Kim, Irfan Essa, Feng Yang

    ECCV 2024 / Google Student Research Project

    [ paper ]

  • @ Korea University

  • Sound-Guided Semantic Image Manipulation

    Seung Hyun Lee, Wonseok Roh, Wonmin Byeon, Sang Ho Yoon, Chanyoung Kim, Jinkyu Kim*, Sangpil Kim*

    CVPR 2022

    [ project / paper / code ]

  • Sound-Guided Semantic Video Generation

    Seung Hyun Lee, Gyeongrok Oh, Wonmin Byeon, Chanyoung Kim, Won Jeong Ryoo, Sang Ho Yoon, Jihyun Bae, Jinkyu Kim*, Sangpil Kim*.

    ECCV 2022

    [ paper / dataset ]

  • Robust Sound-Guided Image Manipulation

    Seung Hyun Lee*, Hyung-gun Chi*, Gyeongrok Oh, Wonmin Byeon, Sang Ho Yoon, Hyunje Park, Wonjun Cho, Jinkyu Kim*, Sangpil Kim*

    Neural Networks 2024

    [ paper ]

  • The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusion

    Yujin Jeong, Wonjeong Ryoo, Seung Hyun Lee, Dabin Seo, Wonmin Byeon, Jinkyu Kim

    ICCV 2023

    [ project ]

  • Functional Hand Type Prior for 3D Hand Pose Estimation and Action Recognition from Egocentric View Monocular Videos

    Wonseok Roh, Seung Hyun Lee, Wonjeong Ryoo, Gyeongrok Oh, Jakyung Lee, Soo Yeon Hwang, Hyung-gun Chi, Sangpil Kim

    BMVC 2023

  • Soundini: Sound-Guided Diffusion for Natural Video Editing

    Seung Hyun Lee, Sieun Kim, Innfarn Yoo, Feng Yang, Donghyeon Cho, Youngseo Kim, Huiwen Chang, Jinkyu Kim*, Sangpil Kim*

    Under review

    [ project / paper / code ]

  • LISA: Localized Image Stylization with Audio via Implicit Neural Representation

    Seung Hyun Lee, Chanyoung Kim, Wonmin Byeon, Sang Ho Yoon, Jinkyu Kim*, Sangpil Kim*

    Under review

    [ paper ]

Awards & Honors

  • Pytorch Open Source Contribution, Pytorch Tutorial Translation (lead menti) 2021
  • Google Developer Student Clubs – University of Seoul, Korea 2021
  • Software Maestro 11th – Best Software talent discovery program, Korea

Academic Activities

Teaching Assistant

  • Machine Learning at Korea University (Spring 2023)