Testers wanted: Testing DRAM reliability on BPi M2+ and NanoPI M1

[tkaiser]

Update: BPi M2+ is done but further results would still be interesting.

 

Now we need results for NanoPi M1. The most simple way is using the Armbian image as outlined in post #4 below. The only thing you would've to change is the following and can then run the lima-memtester binary as outlined below:

ln -sf /boot/bin/nanopim1.bin /boot/script.bin
echo nanopim1 > /etc/hostname
reboot

Dear BPi M2+ users. I just tested DRAM reliability with my BPi M2+ just to realize that this board doesn't run stable even with just 624 MHz clockspeed (currently testing 600 MHz for an additional hour or so).

 

In case you have a BPi M2+ it would really help if you could do the same. Everything is outlined here: http://linux-sunxi.org/Xunlong_Orange_Pi_Plus_2E#DRAM_clock_speed_limit

 

Just grab the referenced fel-boot-lima-memtester-on-orange-pi-h3-v3.tar.bz2 archive that now also contains stuff for BPi M2+ and then use the contained fel-boot-lima-memtester-on-banana-pi-m2plus script (I would also start with 624 and if that succeeds then check increasing DRAM clockspeed in 24 MHz steps). Please be aware that since SinoVoip saved a second led on BPi M2+ the red led will blink and you get no notification by a solid lighting 2nd led so you should let the test run at least for 1 hour.

 

It's important to connect a HDMI display and ensure that a spinning cube can be seen with gray background (if the background is glowing red then something is wrong). Some more information can be found here: https://linux-sunxi.org/Hardware_Reliability_Tests#Reliability

 

Please get back ASAP with results since Chen-Yu is currently preparing upstream u-boot support and DRAM timing is important!

[lvmc]

I can provide SSH access to multiple M2+ boards if needed.

[vlad59]

Hi,

 

Finally I made a full hour run on my NanoPI M1 at 624MHz, the culprit was definitely the SOC temperature / cooling strategy. It's not shown on the screenshot by the 2 cores killed came back after a little while.

 

I'll make other runs at higher frequencies, but the graph made me wonder about the actual cooling table.

 

As soon as 2 cores were killed the temperature dropped from 93°c to 85°C and more or less stayed around 84°C for the rest of the lima run. CPU Frequency even jumped to 628MHz for small periods of time. I think there is some enhancement possible here. What if (I know I don't like sentences starting like that) we changed the cooling strategy :

 * 1st action : limit the CPU frequency to ~672 MHz so that the CPU is powered with 1.1V

 * 2nd action : kill 2 cores but stay with the same CPU frequency limit

 * 3rd action : stay with 2 cores (that seems possible if I understand correctly the graph) but limit the freq to 480 or 320

 

Playing with frequency alone seems not optimal here. It also seems that killing one core only has a very limited impact on the SOC temperature.

 

Is there any documentation about ths_para / cooling_table (the comments are mostly fine, but the more the better) ?

[tkaiser]

I can provide SSH access to multiple M2+ boards if needed.

 

Maybe I get back to you soon regarding this.

 

Playing with frequency alone seems not optimal here. It also seems that killing one core only has a very limited impact on the SOC temperature.

 

Be careful since we're talking here about another part of the SoC that overheats badly: In lima-memtester's case the idea is to run heavy stuff on the GPU cores (2 of them available in H3 and clocked by us at 600 MHz by default) to let GPU and CPU cores start a battle over available DRAM (they have to share access to DRAM anyway and if both are busy then DRAM access/timing problems will be found more easily).

 

So in this example we're talking about heavy GPU activity and Allwinner's 'budget cooling' jumping in. This 'budget cooling' stuff starts to throttle CPU cores down even if heat is generated mostly by the GPU cores. So we face two challenges:

• check in kernel sources whether this budget cooling stuff really does what it should (I doubt anyone will do that)
• compare with more realistic (lighter) workloads (be it CPU or GPU bound)

It seems our new BSP kernel shows also way better GPU performance compared to older OS images (or the fresh ones from @sinovoip for their M2+ since they simply took loboris' aging kernel sources for their new board): http://forum.banana-pi.org/t/bpi-m2-armbian-n64-emulator-test/1836  (+30 percent more GPU performance for free isn't that bad by exchanging kernel sources and caring about settings)

 

So after installing RPi-Monitor and running such game emulators and watching temperatures we know about 'GPU' and running the usual cpuburn-a7 stuff we could check 'worst case CPU load' conditions. And these should be the basis for throttling strategies and not the rather unrealistic lima-memtester approach (that's only needed to put as much stress on the board as possible to check for DRAM reliability as we've to do now).

 

So what we already know thanks to your test is that NanoPi M1 is unfortunately also prone to heavy overheating (for yet unknown reasons) and that we need to take care of this by allowing downclocking to the minimum.

 

Now what's missing are performance and thermal figures with cpuburn-a7. And then we can decide about a perfect throttling strategy since while you're right that in lima-memtester example it seems suboptimal that's due to most of the heat generated by the GPU (where CPU downclocking is rather useless except of preventing insane overheating so that the board totally freezes)

 

Edit: It would be great if you could parse dmesg output for this budget cooling stuff and put it on pastebin.com since at least with the older BSP driver throttling changes for both CPU and GPU cores were sent to syslog. 

[vlad59]

Here is attached the graph for an hour long test of my NanoPI M1 with cpuburn-A7

 

It's actually not that bad, most of the time it was using 3 cores at 648MHz at 85°C. IMHO it could maybe be using a slightly higher clock speed while staying at 1.1V (going up to 90~95°C).

 

For a little while the forth core was used again but then the frequency went to 320MHz (it seems that 240MHz don't work with my Nanopi for now) which way too low comparing with the performances with 3 cores.

 

I'll start another test with lima_memtester (with a higher DRAM speed) and check your other questions afterwards.

[tkaiser]

It's actually not that bad, most of the time it was using 3 cores at 648MHz at 85°C.

 

Thx for the numbers. But that's somewhat sad and means NanoPi M1 will also be way slower than the various Oranges in full load situations (please compare here with results without heatsink from OPi One, PC and Plus 2E)

[vlad59]

Yes I know, I was trying to cheer myself up

[vlad59]

At last one good news. I made three one hour runs of lima_memtester at 696, 720 and 744 MHz and they all passed without any crash or redish background.

 

Tomorrow I'll remove the heatsink, clean the CPU and remake a cpuburn-a7 to be absolutely sure that my previous graph is trustable.

 

 

@tkaiser

Let me know if you need something else.

[vlad59]

Here is the graph for more than an hour of cpuburnA7 with my Nanopi M1 without heatsink.

 

So my conclusion :

 * No crash also so the setting from Banana Pi M2+ seems good. I'll make a lima_memtest run but I think it'll be fine.

 * The heatsink does help (the CPU stayed mostly at 648MHz with 3 cores with the heatsink and was much erratic without)

 * The Nanopi M1 performance under load is way lower than Orange Pi (see tkaiser post)

 

I think I'll use my NanoPI with Openelec to replace my Raspberry Pi 2. I'll make a test this weekend and ask my daughters to watch Frozen / Any Ghibli movie one or two times on it . That's not what I have planned but that'll do.

 

@tkaiser

Can I make a PR on github to update the nanopim1.fex or do you want to fix differently ?

Should I add the nanopi M1 page on linux-sunxi's Wiki ?

 

That should close the Nanopi case for now unless anybody need another test (once the nanopi are in the hands of my daughters, tests will be way harder to plan)

[tkaiser]

Thx for the test! Yes, please send a PR with updated thermal settings. And creating a page on linux-sunxi wiki would also be great (please don't forget the entry in this table too).

 

Thx for your contribution!

[tkaiser]

Ah, eventually found it: Quoting Olimex (they started with first H3 board prototypes last year just to realize they were overheating like crazy):

 

We learned our lessons from the first H3-OLinuXino prototypes, now this board consumes less power, DDR3L memory is used at lower voltage 1.35V and this helps the processor to not overheat like on our first proto.

 

In our tests the only two boards that use DDR3 instead of DDR3L (as used on every Orange Pi) also overheat like hell. So maybe that's the culprit (the more primitive/unused voltage regulators and PCBs with less heat spreading capabilities adding to the problem).

 

Based on these experiences BPi M2+ and NanoPi M1 aren't the devices of choice when it's about running constantly challenging or even heavy workloads.

[Tido]

On the Testpoints I measured:

1,22 V SoC ?

1,30 V SoC ?

1,50 V DDR3 ?

 

As this RAM gets hotter than DDR3L it can also influence the temperature of the surrounding like air, PCB.

And maybe than also the Bus is powered with 1,50 V and this will hit the SoC in some part of it.

[tkaiser]

Thx for reminding the test point values. Currently I'm a bit curious how Beelink X2 behaves. According to fex file contents there LPDDR3 is used (low power) together with an unregulated VDD_CPUX supply. 

[zador.blood.stained]

As this RAM gets hotter than DDR3L it can also influence the temperature of the surrounding like air, PCB.

And maybe than also the Bus is powered with 1,50 V and this will hit the SoC in some part of it.

 

This voltage may affect DRAM controller in SoC in terms of generating more heat. It may be possible to compare DDR3 and LPDDR3 with some memory-intensive tests ("stress -m") while making other settings (CPU frequency and voltage) identical.

[tkaiser]

It may be possible to compare DDR3 and LPDDR3 with some memory-intensive tests ("stress -m") while making other settings (CPU frequency and voltage) identical.

 

Just remembered where my Beelink X2 has been put into. Since we're talking about LPDDR3 I think the only device we have equipped with this is the X2. Original fex file contents:

[dram_para]
dram_clk = 672
dram_type = 7
dram_zq = 0x3bfb
dram_odt_en = 1
dram_para1 = 283377664
dram_para2 = 0
dram_mr0 = 6208
dram_mr1 = 64
dram_mr2 = 8
dram_mr3 = 2
dram_tpr0 = 0x48a192
dram_tpr1 = 0x1b1b18d
dram_tpr2 = 0x76052
dram_tpr3 = 0x0
dram_tpr4 = 0x0
dram_tpr5 = 0x0
dram_tpr6 = 0x0
dram_tpr7 = 0x2a0782a0
dram_tpr8 = 0x1
dram_tpr9 = 0x5008888
dram_tpr10 = 0x8888
dram_tpr11 = 0x5005
dram_tpr12 = 0x0
dram_tpr13 = 0x2000

I use now my Xenial-lima-memtester image from this thread, the most recent fex settings (treating the LPDDR3 as DDR3/DDR3L but that doesn't matter now since we're using mainline u-boot where it's treated like that anyway) and have a look. Currently using u-boot settings with 648MHz, let's see how the little box behaves (no heatsink, only a heatpad and inside pretty small plastic enclosure)

[zador.blood.stained]

Just remembered where my Beelink X2 has been put into. Since we're talking about LPDDR3 I think the only device we have equipped with this is the X2.

I meant we could compare all 3 types - 1.5V DDR3 (M2+), 1.35V DDR3L (H3 Oranges) and 1.2V LPDDR3.

 

DRAM Datasheet linked at X2 linux-sunxi wiki page is for 1.5V DDR3, so do we have any device with LPDDR3?

 

Edit: Orange Pi One has DDR3L, not LPDDR3

[tkaiser]

I meant we could compare all 3 types - 1.5V DDR3 (M2+), 1.35V DDR3L and 1.2V LPDDR3 (H3 Oranges).

 

DRAM Datasheet linked at X2 linux-sunxi wiki page is for 1.5V DDR3, so do we have any device with DDR3L?

 

Nope, I've fooled myself by relying on the original fex file (thinking type 7 would be LPDDR3). Since I opened the device just to be sure I will now let lima-memtester run to see how temperatures behave. The thermal pad and a rather huge metal plate used inside the enclosure did at least a pretty good job compared to now without any heatsink but lying flat on the table: 480 MHz with only 2 cores (2 more cores before)

[zador.blood.stained]

This is not directly related to the topic, but still interesting: I noticed this on Oranges schematics

Vout = 0.6*(1+Rd1/Rd2)
VDRAM=1.5V/1A,R2=100K-1%
VDRAM=1.35V/1A,R2=120K-1%

and decided to check and verify

 

So OPi One and LIte use 1.5V powered DRAM (DDR3?) (1_5V testpoint voltage is 1.5V)

OPi PC, PC Plus and Plus 2E use 1.35V powered DRAM (DDR3L?) (1_5V testpoint voltage is 1.35V)

 

So wiki info on One and Lite about using DDR3L is not correct unless I didn't understand this properly.

[Tido]

If you look for this PDf from ARM Cortex A7, I searched like this in google: cortex a7 memory controller filetype:pdf

 

Look for this file, on page 34 of 268
This book is for the Cortex-A7 MPCore processor. This is a multiprocessor device that has between one to four processors.
DDI0464D_cortex_a7_mpcore_r0p3_trm.pdf

 

2.3 Clocking and resets

 

2.4 Power management

I don't know if this is implemented into H3, but it may be help to understand what the 'switches' are for.

[tkaiser]

So OPi One and LIte use 1.5V powered DRAM (DDR3?) (1_5V testpoint reads 1.5V)

OPi PC, PC Plus and Plus 2E use 1.35V powered DRAM (DDR3L?) (1_5V testpoint reads 1.35V)

 

So wiki info on One and Lite about using DDR3L is not correct unless I didn't understand this properly.

 

No idea but since most probably I have written the wiki entry it might be not correct at all (but doesn't the type of DRAM matter, according to the datasheet it's DDR3L, isn't it?)

 

Anyway: This is Beelink X2 measured without heatsink lying flat on the table:

 

 

lima-memtester with 648 MHz DRAM clock on the left (ending up with only one CPU core @ 480 MHz), cpuburn-a7 on the right (3 CPU cores @ 480 MHz).

[zador.blood.stained]

One and Lite have K4B2G1646Q-BCK0 (DDR3); PC and PC Plus have K4B4G1646Q-HYK0 (DDR3L).

[tkaiser]

One and Lite have K4B2G1646Q-BCK0 (DDR3); PC and PC Plus have K4B4G1646Q-HYK0 (DDR3L).

 

My bad, will correct that in the wiki asapissimo. But then we're back at 'BPi M2+, NanoPi M1, Olimex H3 prototype and Beelink X2 overheat for yet unknown reasons' (and have one possible explanation why thermal results for the larger Oranges are slightly better compared to One/Lite)?

[zador.blood.stained]

But then we're back at 'BPi M2+, NanoPi M1, Olimex H3 prototype and Beelink X2 overheat for yet unknown reasons' 

First two have fixed CPU voltage and we don't have schematics for other two, so it's possible that they overheat simply because lowering frequency only is not enough at 100% CPU load.

 

(still this is a bit off topic of DRAM reliability)

[Tido]

 

First two have fixed CPU voltage

This is, what I also wrote to TK in private message.

 

@Zador you have Orange Pi One. There are several 'Test Points' to measure voltage.

Can you take some samples (volt) while the One is just idle?

Can you also take some samples on load?

 

We are just guessing here.

[tkaiser]

First two have fixed CPU voltage and we don't have schematics for other two, so it's possible that they overheat simply because lowering frequency only is not enough at 100% CPU load.

 

Sure, but in case you would modify fex for One/Lite in a way that VDD_CPUX remains at 1.3V, then let cpuburn-a7 run and confirm VDD_CPUX by measuring the test point we would get one step further (agreeing on being slightly off-topic here )

[zador.blood.stained]

@Zador you have Orange Pi One. There are several 'Test Points' to measure voltage.

Can you take some samples (volt) while the One is just idle?

Can you also take some samples on load?

I already measured CPU voltage on 2 Orange Pi Ones and on Orange Pi Lite, it's OK. I don't see how any other voltage may be significantly affected under load, but I may perform some measurements in the future (I don't want to unscrew a "rack" of 3 Oranges to get access to all test points for now)

 

Sure, but in case you would modify fex for One/Lite in a way that VDD_CPUX remains at 1.3V, then let cpuburn-a7 run and confirm VDD_CPUX by measuring the test point we would get one step further (agreeing on being slightly off-topic here )

I thought about this. I'll test this setup and will post the results.

 

Edit: Hm. "New" OPi One on the left, "Old" on the right, both have heatsinks.

[tkaiser]

Edit: Hm. "New" OPi One on the left, "Old" on the right, both have heatsinks.

 

Hmm... I wouldn't trust in the readouts of the 'Old' one at all

 

BTW: Did lima-memtester on Beelink X2 with 672 MHz (passed but lima-memtester is really heavy and H3 throttled down to 240/312 MHz with only 3 CPU cores left). So we choose now 624 MHz for the X2. On the right cpuburn-a7 in the enclosure (thermal pad + large metal plate) where clockspeeds/throttling look better compared to no heatsink at all:

 

[slinde]

Here is the result of my Beelink X2 running "lima-memtester 100M" and the setting "dram_clk = 648". Test went well, cube was spinning with grey background the whole time. I can see it disabled one cpu-core after about 30min.

 

[Tido]

I already measured CPU voltage on 2 Orange Pi Ones and on Orange Pi Lite, it's OK. I don't see how any other voltage may be significantly affected under load, but I may perform some measurements in the future (I don't want to unscrew a "rack" of 3 Oranges to get access to all test points for now)

Well, I don't know either but there must be a reason that they stay 'cool' compared to the other.

So I thought we may get a guidance from this aspect.

On the other hand it is not such an urge that I would ask you to take your things apart.

[tkaiser]

Here is the result of my Beelink X2 running "lima-memtester 100M" and the setting "dram_clk = 648". Test went well, cube was spinning with grey background the whole time.

 

 

Ok, then we should take 624 MHz as with the other H3 boards (these 624 MHz were used in normal Armbian builds anyway since here u-boot starts DRAM initilization and the value assigned here will be used all the time until the first time throttling occurs since then the kernel switches to the value defined in script.bin).

 

Thank you for testing!

[slinde]

I did another test over night with 624MHz. It pretty much looks the same.