I am working on implementing the 'perf mem' command for Power
systems. This would for instance, let us know where in the memory
hierarchy (L1, L2, Local RAM etc) the data for a load/store
instruction was found (hit).

On Power7, if the mcmcra[DCACHE_MISS] is clear _and_ the
instruction is a load/store, then it implies a L1-hit.

Unlike on Power8, the Power7 event vector has no indication
if the instruction was load/store.

In the context of a PMU interrupt, is there any way to determine
if an instruction is a load/store ?

You could read the instruction from memory and work it out.  

We do something similar to this in power_pmu_bhrb_to() where we read the
instruction and work out where the branch is going to.

If you do this, please use and/or extend the functions in
arch/powerpc/lib/code-patching.c

Mikey