it seems to me that the swap file interface doesn't make sense; to
"eliminate the middleman" and provide a direct swap tree type would be
the way to go, so a btrfs can hold swapped pages. Doing this would
AIUI require:
1) direct interface to btrfs trees, not via file-system interface
2) modifying swapping to use (1) instead of essentially the
swap-to-a-partition mechanisms to the "swap file" presented as a loop
device

I also wonder if you explicitly create a loop device for your swap
file and mount that as swap, will that work and if so how poorly.