0

I've asked this question in various places, and I'm still trying to get any kind of answer. Please forgive the cross posting if you encounter it.

This question boils down to whether it is possible to access main memory sequentially at full speed on an ARM Cortex A9, with specific problems about my proposed approach.

I'm working on some performance critical numerical code running on a 667 MHz Cortex A9 (in a Xilinx Zynq) that is currently cache limited. I can achieve ~1 neon multiply per memory access on small datasets (~530 MFlops), dropping to far less when the data size gets bigger (~180MFlops in the limit). This is on ARM Linux with the GCC stack.

The 180 MFlops figure above is with PLD instructions (which definitely help), but I'm still nowhere near the theoretical memory bandwidth of the main memory (DDR2 @533MHz). Given the problem is essentially operating on sequential memory accesses that are fully defined a priori, I was wondering if I could speed things up with the L2 preload engine.

Now, I've experimented somewhat with this, writing a small kernel driver that flips the PLEUAR register bit to enable userspace access to the PLE (as well as fiddling a bit with the PLEPCR register to reduce the cycles between PLE operations), and then performing the PLE operation (something like MCRR p15, 0, %[vect_addr], %[vect_config], c11;).

Various things are apparent from this:

  1. The program runs successfully about 20% of the time, the remaining times it yields an "Illegal instruction" almost immediately on the first attempt to set up the PLE.
  2. The run times are not affected in the slightest. If this is expected, I'm very interested to know whether it's possible to ever operate on data accessed at the memory bandwidth.
  3. I can monitor the (16 length) FIFO and observe it decreasing in space as the instructions are added and then reverting to empty.

My observations would suggest I'm not doing what I think I'm doing, or at least, something is thwarting my attempts to do what I want to do.

Am I missing something fundamental here? Am I treading all over the kernel's carefully protected space?

Are the virtual addresses different as viewed by the PLE compared to the running process?

Why might the illegal instruction occur?

1 Answer 1

1

Xilinx reliably informs me that all models of Zynq have a dual-core Cortex-A9, so I'm going to guess that you're running SMP Linux and your kernel module isn't taking that into account. Unless you use one of the on_each_cpu* calls to perform the enable on every core, you're setting yourself up for a game of Russian roulette against the scheduler. Note also that if you have cpuidle or any other power/thermal management over which a core may lose state, that's going to add to the fun, because there's no guarantee that the idle/hotplug code will bring things back in their meddled-with state.

Now, with that out of the way, the Preload Engine is a crazy thing specific to certain Cortex-A9 configurations, which nobody uses, especially not in Linux. I don't know for sure its original design intent, but I suspect it's got more to do with feeding a GPU or other device hung off the ACP than with the cores themselves, which are perfectly capable of managing their own prefetching. It also doesn't address your real problem anyway: all the bandwidth in the world between DRAM and L2 still isn't going to stop you missing at L1, it only makes it hurt a bit less. Achieving maximum processing bandwidth on older cores is all about unrolling your loops by just the right amount, and fine-tuning your PLDs in search of the sweet spot which keeps L1 as hot as possible - that obviously depends on plenty of system-specific factors, but will typically work out to a prefetch distance of around 2-4 cache lines ahead.

4
  • This is a wonderfully helpful answer, clearing up well my understanding, so thank you. I've actually messed around quite a bit with PLD twiddles and I think I've reached the limit of what is possible. I'm going to play with a newer CPU and see how that improves things. Mar 3, 2016 at 17:06
  • @HenryGomersall Heh, on Zynq there's also presumably the option of synthesizing your own hardware block to do the processing and wiring it to the ACP, at which point the Preload Engine might actually become more useful :D Mar 4, 2016 at 22:14
  • Absolutely. The bandwidth of DMA over the ACP channel is not quite enough to get much benefit sadly - We can get ~600 MB/s sustained, which would translate to 150 MFlops, more or less what I get on the CPU. Actually, that might well explain the limit :) Another solution is to build a custom board with the ram connected directly to the PL and get the full memory bandwidth sustained. Both those solutions are notably harder to work with that a simple CPU software solution though. Mar 7, 2016 at 9:46
  • The Preload Engine isn't specific to the Cortex-A9, it's an evolution of the one present (non-configurable) on the Cortex-A8, which itself is heavily based on the ARM11 DMA engine used to transfer between TCM and external memory.
    – Matthijs
    Dec 18, 2023 at 21:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.