Jin Woo Lee on "Differentiable Physical Modeling for Sound Synthesis: From Design to Inverse Problems"

Jin Woo Lee is a PhD Candidate at Seoul National University advised by Prof. Kyogu Lee (CCRMA PhD 2008). His research interests are focused on (1) physical modeling for musical instrument sound synthesis and (2) differentiable rendering for immersive and efficient sound simulation. His recent works broadly cover topics in musical sound synthesis, spatial audio rendering, loudspeaker control, and speech quality analysis. He has interned at Meta Reality Labs Research and Supertone. Prior to his PhD, Jin conducted research in computational fluid dynamics during his undergraduate years in Mechanical Engineering from POSTECH. For more information, please visit personal website (http://jnwoo.com/).


Abstract:

Synthesizing sounds and simulating performance based on an instrument's shape, physical properties, and playing technique is a long-standing research topic. Much of the research based on these signal processing and numerical analysis techniques, which is still evolving, has led to the development of numerous virtual musical instruments and instrument designs. In recent years, the ease of computer-based gradient back-propagation, especially in machine learning research, has led to the development of rendering engines based on "differentiable" physical modeling in many fields, including sound synthesis and computer graphics. In this talk, I will present my research related to these differentiable rendering techniques, with a focus on physical modeling-based music sound synthesis, and discuss some of the challenges we've faced and the ones we're still working on. Specifically, the talk will cover: design choices for differentiable physical modeling for sound synthesis, the non-convexity issues that can be encountered in parameter optimization, and the approaches that have been taken in various fields to solve them.


Reference:

Lee, J. W., Park, J., Choi, M. J., & Lee, K. (2024). Differentiable Modal Synthesis for Physical Modeling of Planar String Sound and Motion Simulation. arXiv preprint arXiv:2407.05516.

Jin, X., Xu, C., Gao, R., Wu, J., Wang, G., & Li, S. (2024, July). DiffSound: Differentiable Modal Sound Rendering and Inverse Rendering for Diverse Inference Tasks. In ACM SIGGRAPH 2024 Conference Papers (pp. 1-12).

Condor, J., Speierer, S., Bode, L., Bozic, A., Green, S., Didyk, P., & Jarabo, A. (2024). Volumetric Primitives for Modeling and Rendering Scattering and Emissive Media. arXiv preprint arXiv:2405.15425.

Torres, B., Peeters, G., & Richard, G. (2024, April). Unsupervised Harmonic Parameter Estimation Using Differentiable DSP and Spectral Optimal Transport. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1176-1180). IEEE.

Xing, J., Luan, F., Yan, L. Q., Hu, X., Qian, H., & Xu, K. (2022). Differentiable rendering using rgbxy derivatives and optimal transport. ACM Transactions on Graphics (TOG), 41(6), 1-13.


Zoom:

https://stanford.zoom.us/j/7733389381?pwd=Z2xkVUEvNTA3dTRSclBHUlNRZGFZdz09
Meeting ID: 773 338 9381
Passcode: 601462