EditSinger: Zero-Shot Text-Based Singing Voice Editing System with Diverse Prosody Modeling

Abstract

Zero-shot text-based singing voice editing enables users to edit the singing content by just performing text operations on the lyrics, while without any additional data from the target singer. However, due to the different demands, challenges occur when applying existing speech editing methods to singing voice editing task, mainly including the lack of systematic consideration concerning prosody in insertion and deletion, as well as the trade-off between the naturalness of pronunciation and the preservation of prosody in replacement. In this paper we propose EditSinger, which is a novel singing voice editing model with specially designed diverse prosody modules to overcome the challenges above. Specifically, 1) a general masked variance adaptor is introduced for the comprehensive prosody modeling of the inserted lyrics and the transition of deletion boundary; and 2) we further design a fusion pitch predictor for replacement. By disentangling the reference pitch and fusing the predicted pronunciation, the edited pitch can be reconstructed, which could ensure a natural pronunciation while preserving the prosody of the original audio. In addition, to the best of our knowledge, it is the first zero-shot text-based singing voice editing system. Our experiments conducted on the OpenSinger prove that EditSinger can synthesize high-quality edited singing voices with natural prosody according to the corresponding operations.

Introduction:

In the first two sections (Audio Samples & Method Analyses), there are some samples of common performance demonstration and comparison experiments.

In the third section (More Samples), we provide more samples of different aspects (e.g., comparisons of different editing positions).

Our research is based on the open-source dataset OpenSinger, and all experiments conducted in the paper have been authorized by the publisher. This project is currently only used for research, and aims to make contributions and provides some ideas for the community. Please do not used for commercial purposes.

1 Audio Samples

Notes:

GT denotes the original audio(the input audio to be edited).

The red part represents the editing region.

Words —— Phonemes

Exp. 1:

GT	GT(Mel+PWG)	EditSinger(insertion)	EditSinger(replacement)	EditSinger(deletion)

Exp. 2:

GT	GT(Mel+PWG)	EditSinger(insertion)	EditSinger(replacement)	EditSinger(deletion)

Exp. 3:

GT	GT(Mel+PWG)	EditSinger(insertion)	EditSinger(replacement)	EditSinger(deletion)

Exp. 4:

GT	GT(Mel+PWG)	EditSinger(insertion)	EditSinger(replacement)	EditSinger(deletion)

Exp. 5:

GT	GT(Mel+PWG)	EditSinger(insertion)	EditSinger(replacement)	EditSinger(deletion)

Exp. 6:

original lyrics: 在昏暗中的我 —— z ai # h un | an # zh ong # d e # w o
insertion: 在那时昏暗中的我 —— z ai # n a | sh i # h un | an # zh ong # d e # w o
replacement: 在昏暗中与你(的我) —— z ai # h un | an # zh ong # y u # n i (~~d e # w o~~)
deletion: 在昏暗(中)的我 —— z ai # h un | an # ( ~~zh ong #~~ ) d e # w o

GT	GT(Mel+PWG)	EditSinger(insertion)	EditSinger(replacement)	EditSinger(deletion)

2 Method Analyses

2.1 Insertion

Exp. 1:

GT	GT(Mel+PWG)	EditSinger(insertion)	w/o CVA	w/o ML-GAN

Exp. 2:

GT	GT(Mel+PWG)	EditSinger(insertion)	w/o CVA	w/o ML-GAN

2.2 Deletion

Exp. 1:

original lyrics: 你何苦非为他等在雨中 —— n i # h e | k u # f ei | w ei # t a # d eng # z ai # y u # zh ong
deletion: 你(~~何苦非~~)为他等在雨中 —— n i # (~~h e | k u # f ei |~~ ) w ei # t a # d eng # z ai # y u # zh ong >

GT	GT(Mel+PWG)	EditSinger(deletion)	w/o CVA	w/o ML-GAN

Exp. 2:

GT	GT(Mel+PWG)	EditSinger(deletion)	w/o CVA	w/o ML-GAN

2.3 Replacement (Dang Test)

Note: In this part, it can fully demonstrate the superior performance of Editsinger(replacement), and it even supports the replacement of entire sentences, which is not available in previous work. “Dang” here can be understood as any character. We have tested many other characters and done experiments including part and whole sentence replacement experiments, which are also very effective. Directly migrating the original prosody (Direct) to the new word without considering the attributes of the word will lead to a decrease in the sense of hearing, and ignoring the prosody of the corresponding position (w/o FPIP) will lead to a decrease in the similarity with the original song.

Exp. 1:

GT	GT(Mel+PWG)	EditSinger(replacement)	Direct	w/o FPIP	w/o VQVAE	w/o ML-GAN

Exp. 2:

GT	GT(Mel+PWG)	EditSinger(replacement)	Direct	w/o FPIP	w/o VQVAE	w/o ML-GAN

3 More Samples

Editing at Different Positions (Begining/Middle/End of the Sentence)

original lyrics: 朋友爱得那么苦痛 —— p eng | y ou # ai | d e # n a | m e # k u | t ong

Type	Begining	Middle	End
insertion
	如果朋友爱的那么苦痛	朋友如果爱的那么苦痛	朋友爱的那么苦痛如果
deletion
	(朋友)爱的那么苦痛	朋友爱的(那么)苦痛	朋友爱的那么苦(痛)
replacement
	认真(朋友)爱的那么苦痛	朋友爱的认真(那么)苦痛	朋友爱的那么认真(苦痛)