However, as soon as I rmmod the driver, set_ftrace_filter becomes
empty automatically.

Out-of-tree means the module is not build together with the kernel.

So how do I do a trace on out-of-tree modules I have ?

If you built your driver without building it together with the kernel, 
you can trace it as any other module after you insmod it.

Cheers,
Abel