quoted

I am right it sounds like we can drop vivt cache support with highmem?
vivt is alot easier to understand if kmap isn't running around
creating virtual addresses aliases!

I think so, yes. I've prototyped a patch for it to add a dependency
on !VIVT and the drop the code paths that are clearly no longer
needed without it. This obviously needs proper testing, but the
diffstat is promising:

diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index f14bac913d50..0a31cfaf3818 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig

@@ -1214,6 +1214,7 @@ config ARCH_SPARSEMEM_ENABLE
 config HIGHMEM
        bool "High Memory Support"
        depends on MMU
+       depends on !CPU_CACHE_VIVT
        select KMAP_LOCAL
        select KMAP_LOCAL_NON_LINEAR_PTE_ARRAY
        help
 arch/arm/Kconfig                    |  1 +
 arch/arm/configs/gemini_defconfig   |  1 -
 arch/arm/configs/multi_v5_defconfig |  1 -
 arch/arm/configs/mvebu_v5_defconfig |  1 -
 arch/arm/include/asm/highmem.h      | 56 +-----------------------
 arch/arm/mm/cache-feroceon-l2.c     | 31 +------------
 arch/arm/mm/cache-xsc3l2.c          | 47 +++-----------------
 arch/arm/mm/dma-mapping.c           | 14 ++----
 arch/arm/mm/flush.c                 | 19 ++------
 mm/highmem.c                        | 86 +++++--------------------------------
 10 files changed, 27 insertions(+), 230 deletions(-)

I did get an email from Nicolas Schichan (added to Cc here),
and he is still supporting a widely deployed Kirkwood based
platform that uses 1GB RAM configurations. He should get
a chance to test that with CONFIG_VMSPLIT_3G_OPT, but I
would expect that to continue working, possibly with minor
bugfixes.