[GSOC] ref-filter code clean

Hey,
Due to the existence of dummy ref_format in ref-filter will hinder some
subsequent improvements, I hope this patch [1] gets attention, thanks.

--
ZheNing Hu

[1] https://lore.kernel.org/git/pull.1025.git.1629882532.gitgitgadget@gmail.com/ (local)