I'm a bit confused about what PageTransCompoundMap() is supposed to do.
What it actually does is check that the specific page (which may or
may not be a head page) is not mapped by a PTE.  I don't understand why
you'd care how some (other?) process does or does not have it mapped.
What I _think_ you want to know is "Can I map this page with a PMD entry
in the guest".  And the answer to that is simply:

bool kvm_is_transparent_hugepage(kvm_pfn_t pfn)
{
	struct page *head = compound_head(pfn_to_page(pfn));
	return compound_order(head) >= HPAGE_PMD_ORDER;
}

but maybe there's some reason you don't want to map hugetlbfs or other
sufficiently large compound pages with PMDs?

Looking at the one caller of kvm_is_transparent_hugepage(), I'd be
tempted to inline the above into transparent_hugepage_adjust()
and call get_page() directly instead of indirecting through
kvm_get_pfn().