About IO Access Ordering

Saw an interesting discussion at
https://groups.google.com/forum/#!topic/linux.kernel/AlYdjqXOJCs



> You can find this info in ARMv7 ARM spec[1] named
> "DDI0406B_arm_architecture_reference_manual_errata_markup_8_0.pdf", on
> page A3-45. There is a para that goes:
>
> "Accesses must arrive at any particular memory-mapped peripheral or
> block of memory in program order, that is, A1 must arrive before A2.
> There are no ordering restrictions about when accesses arrive at
> different peripherals or blocks of memory, provided that the accesses
> follow the general ordering rules given in this section."

That is news to me.  My DDI0406B does not have this paragraph, so it's
something that ARM has sprung upon us without telling *anyone* about it.
It's not unreasonable or even unexpected.  That is exactly the same
condition which applies on buses like PCI due to write posting on bridges
downstream of the CPU, and issuing memory barriers will not help with
that.

Consider two PCI devices each behind their own P2P bridge.  Device A's
bridge is really lazy and takes time to empty its write post buffer.
Device B's bridge is really fast at getting writes.  If you write to
device A then device B, they'll arrive at device B before device A.
Again, let me stress that memory barriers will not allow you to solve
this problem.

The only way to solve this is to read back from the device, because reads
to device memory can not bypass writes to device memory, otherwise the
system is unpredictable.  With PCI, it's recommended to read back from
the exact same address which you've written to _provided_ there's no
side effects from doing so.  If there are, you have to find some other
solution to it.