board support - general discussion / project aims

[tkaiser]

Note: discussion started here: https://forum.armbian.com/topic/6650-toolchain-banana-pi-m2-zero-h3/?do=findComment&comment=50530

 

41 minutes ago, Igor said:

Ignore the fact its there or remove it

 

 

Igor, I was not talking about the board in question. I'm talking about something else like the last 18 months already: policies and processes and why this stuff is ignored (by you). And how we can end the horrible user experience with u-boot/kernel updates constantly breaking user installations.

 

When part of 'adding a CSC board' is a patch that tries to modify drivers/mmc/sunxi_mmc.c in a general way that affects all our sunxi boards I consider this dangerous. Currently the patch does not work, without the (fixed) patch the resulting OS image will not work. If the patch will be 'fixed' it affects all other sunxi boards. Why is the whole stuff there in the first place? Especially since for whatever bizarre reasons we still have not something like a testing or beta branch to be able to test out stuff without breaking installations.

 

Do you think https://github.com/armbian/build/blob/0df0209db90855355abab5326ef075f8feb89ba5/patch/u-boot/u-boot-sunxi/fix-sdcard-detect-bpi-m2z.patch would have been accepted by mainline u-boot folks as part of adding support for one single new board?

[Igor]

25 minutes ago, tkaiser said:

policies and processes and why this stuff is ignored (by you)

Try to imagine how to enforce policies to volunteer developers. Do count them and then proceed to this question.

 

27 minutes ago, tkaiser said:

Do you think https://github.com/armbian/build/blob/0df0209db90855355abab5326ef075f8feb89ba5/patch/u-boot/u-boot-sunxi/fix-sdcard-detect-bpi-m2z.patch would have been accepted by mainline u-boot folks as part of adding support for one single new board?

Of course not. I don't pay much attention to stuff which is CSC and since you discovered a potential bug which is actually not working ... why do we waste time discussing this? Remove it.

 

Our work is somewhere in between as you can see mainline support is months behind and some features might never get there. I don't see this as a problem.

[tkaiser]

3 minutes ago, Igor said:

Try to imagine how to enforce policies to volunteer developers

 

Ahem, that's how open source projects usually work (see Linux kernel or u-boot for example -- @zador.blood.stained already commented on the real issue here)

 

We have no testing branch, we have constant upgrade troubles breaking user installations and interpreting your answer this won't change anytime soon or at all. I can not use Armbian any longer for my own needs without preventing Armbian packages being updated. That's just bizarre and makes me sad.

 

It's not about this specific case with this specific board. I'm talking about how we can prevent this horrible update experience happening again and again and again. Given the patch would've not been broken but really messed up sunxi u-boot... what would've happened? Why has this to happen?

 

Why can't we agree on policies every developer is bound to? Before we add a new device a 'Board bring up' thread will be started where pros/cons and problems are discussed. I have the ugly hack they 'needed' to apply on my list since months but had waited for someone starting a Board Bring Up thread (since instructed to stop starting such threads when the device should NOT be supported by Armbian).

 

If I understand correctly @Icenowy addressed the problem with changed card detect GPIO between pre-production and production boards already back in November last year (see mmc0 definition in her v2 patch series) and then SinoVoip as usual ignored everything and came up three weeks later with their ugly hack instead of relying on her work.

 

18 minutes ago, Igor said:

I don't pay much attention to stuff which is CSC

 

Well maybe that's why policies are useful or even mandatory. IMO there's no need to suck in patches sitting in a board vendor's repo (known for having never contributed upstream anything so 'port and forget' code style has to be expected) especially if Armbian does still not provide any means of a sane testing/beta environment. It's all done in master branch which is IMO terrible.

[Igor]

49 minutes ago, tkaiser said:

We have no testing branch, we have constant upgrade troubles breaking user installations and interpreting your answer this won't change anytime soon or at all. I can not use Armbian any longer for my own needs without preventing Armbian packages being updated. That's just bizarre and makes me sad.

Back in the days, I made many many many tests to secure that, but in today's perspective, this is not possible. The answer to this is - try to: limit board count, enlarge team, enforce better organization. All at once. 

Until we don't have help (more people for covering at least board statuses) or do it radically different (and risk opening a pandora box of currently unknown troubles), nothing will change. Automated testings are one thing, but still, we need truly dedicated persons who would have an eye on this and that board - the new webpage is being designed with this in mind. 
 

49 minutes ago, tkaiser said:

We have no testing branch

... of a build engine which is grabbing sources from elsewhere? Hmm. If we move to a development branch (perhaps even to start with a clean fork as Zador proposed not long ago), someone needs to maintain the old one. Can we afford to maintain both ATM?

 

The last upgrade was a bit messy, yes. I wasn't expecting that much problems from upstream. Prevention would be possible only with extensive testing which would provide a more accurate big picture before putting anything out.

 

We still have an unfinished task: recruiting and manage testers. I tried but failed. Writing announcement or pleads helps absolutely nothing. 

[Tido]

1 hour ago, Igor said:

.. of a build engine which is grabbing sources from elsewhere? Hmm. If we move to a development branch (perhaps even to start with a clean fork as Zador proposed not long ago), someone needs to maintain the old one. Can we afford to maintain both ATM?

YES.

a slowly updating, but stable branch

I can build my installation, I can run updates automatically without fear of a total crash 

Who wouldn't love that - ARCH users, but the majority uses UBUNTU and LTS  that has a very clear reason.

 

Based thereof we can get testers as well, but you ( @Igor, @zador.blood.stained, @tkaiser, @TonyMac32 to name a few) have to decide on a new branch first.

The old stays stable or the new will become the stable - or at the end frome the new you design stable/testing

1 hour ago, Igor said:

a clean fork as Zador proposed

Let's go for it !

[Igor]

13 minutes ago, Tido said:

I can run updates automatically without fear of a total crash

If you mean testing repository. Well, we do have it -> beta.arnbian.com ... I understand the previous hint as a build engine branch and then we do it this way:

https://github.com/armbian/build -b stable -> apt.armbian.com

https://github.com/armbian/build -b development -> beta.armbian.com

This will solve some problems, but not all and I am not sure if we can set some strict merging policy on this schema.

 

If we are ok with proposed, development continues from there, current master is only getting bugfixes from development branch and other branches are duped?

[TonyMac32]

Would it be a horrible thing to add a patch directory for a specific board, assuming we're not going to flat out refuse to support buggy boards?  

 

This way we "packetize" a csc board, or any other board for that matter.  Same kernel, all common kernel issues get resolved as they do now (see Amlogic patchset) but individual board deficiencies stay with the board (Tinker changing SD driver to toggle power supply lines, I have a messy workaround using the device tree platform string keeping it from breaking the MiQi)

 

As far as a stable branch, I like it.  Roll out a patch/change to "development" branch, test it, get feedback where possible, move to stable after passing some review.  Not always possible and not foolproof, but it should help prevent big and (in hindsight perhaps) obvious bugs from making it through.  

[Tido]

20 hours ago, Igor said:

stable -> apt.armbian.com

 

Can we agree on a definition - and once we have it outlined put it into the readme.md and documentation?

I suggest future upgrade procedure:

1. stable only receives security updates - low maintenance effort but high stability
2. after 6,7,8,9,10 (to be defined) months a freeze in development -> beta.armbian.com 
3. the community does tests of the BETA (2 weeks), (still freeze in dev) report bugs
4. bugfixes period for 3 weeks (devs cannot work day and night, family, job, etc.) = RC1
5. the community does tests RC1 (2 weeks), (still freeze in dev) report bugs
6. another round as in 4.  or ready for stable release update

Update Distribution:

Google first delivers new Android to 1-5% of the devices for us this could mean only one SBC per SoC (ie. OPi Zero H2+ | OPi PC H3 | BPi A20 etc.) for one week. We get information on  different-update-scenario's (from us untested, who knows) and SBC usage, diverse/unforeseen feedback (as in my past the IBM AS400 welcomed me: No News - Good News)

 

This sounds now very professionell AND time consuming, so what.. it is very simple and it is easy to destroy a good name, but very hard to get a good reputation.

 

What is your take on that ?

[chwe]

6 hours ago, TonyMac32 said:

Would it be a horrible thing to add a patch directory for a specific board, assuming we're not going to flat out refuse to support buggy boards?  

When we talk about buggy there are always different opinions.  The tinker and in general micro-USB powered boards can be seen as 'buggy' cause part of our userbase doesn't understand that such boards needs some attention to run stable.. On the other hand I still like my OPi0 for lightweight applications and the tinker might be one of the few boards for 'multimedia' applications cause legacy kernel allows hardware accelerated videodecoding & the 'magical mali support'. 

 

What does having a 'board-specific' patch directory mean in case of efforts needed to maintain it? As I understand, in case of the MIQI/Tinker it would save you a lot of work but in case of the whole project? I can imagine that having patches 'here and there' might be hard to maintain..

 

16 hours ago, tkaiser said:

16 hours ago, Igor said:

Try to imagine how to enforce policies to volunteer developers

 

Ahem, that's how open source projects usually work (see Linux kernel or u-boot for example -- @zador.blood.stained already commented on the real issue here)

In case there is enough interest in bringing up a 'board support' policy. I can offer to write this down to a readable text/document. But I only do it if you as devs actively peer-review it and if I get enough input from your side to make it happen.  I don't feel comfortable to write a draft on without inputs from your side... 

 

[TonyMac32]

6 minutes ago, chwe said:

When we talk about buggy there are always different opinions.  

 

Well for me it is the boards with actual hardware errors, which, sadly, is also the Tinker Board with that reboot issue, it was quite a pain initially to fix without breaking any other board supported by the same kernel.  I am unconcerned about people not understanding electricity and getting confused, as long as we have moderators *ahem* to deal with them. 

 

12 minutes ago, chwe said:

I can imagine that having patches 'here and there' might be hard to maintain..

 

Well possibly, although I would argue that any patches going into the main kernel patch directory for a given kernel should be "board agnostic" and be things like device driver additions/updates, dtsi/dts bugfixes, etc.  For a specific board then, come the "ugly" hack job patches, that would be a problem to maintain no matter where they were.  That BPi patch, the tinker reboot, tinker touchscreen (before I stuffed it into the device tree), probably tinker camera, etc. 

[TonyMac32]

With the addition of the development branch, how do we want to handle bugfixes?  Should I apply to development and then cherry pick to the master branch?

[Igor]

1 hour ago, TonyMac32 said:

With the addition of the development branch, how do we want to handle bugfixes?  Should I apply to development and then cherry pick to the master branch?

If some urgent kernel update is needed, we do it directly from development branch? ... and/or completely merge development to master when new features are thoughtfully tested?

 

I am currently finishing a desktop packaging and networking cleanup. Starting with debugging. 

[TonyMac32]

OK, I have that GPIO base update for rk3288 default, Rockchip went to a base 1000 which breaks all the GPIO libs people use, I wanted to get that pushed through.  But, as it is now 3:00 AM here (time changed on me while I was working), I'll have to wait. 

 

 

[tkaiser]

2 hours ago, TonyMac32 said:

With the addition of the development branch, how do we want to handle bugfixes?

 

Well, usually it's the other way around. First developers discuss what they think a problem is, then they try to find a solution for this problem involving an appropriate strategy to deal with weighing pros and cons (e.g. discussion how to handle bug fixes and hot fixes first, then implementing something that suits these needs).

 

IMO our problem is lack of communication and coordination (see the 5.38 update). The main area we should focus on IMO is 'upgrade experience' since while it's quite OK if new images don't work or are malfunctioning users get it that there's something wrong rather quick -- they might only loose some time. But with updates ending up with bricked boards it's a real mess since we destroy productive (server) installations. Currently Armbian is an example how to not do Debian: combining the disadvantages of stable/oldstable (everything outdated as hell) with the disadvantages of testing (do an update and we will brick your board).

 

Addressing this would require getting an idea how to deal with upstream updates (since those introduce an awful lot of problems constantly). Do we need to update u-boot every two months? Can we afford doing this given that 'Team Testers' is obviously an illusion while testing all possible upgrade scenarios would be very time consuming? Do we need to force bleeding edge mainline kernel always? There's a reason Debian and Ubuntu use their own kernels and backport fixes. That's not a solution for us of course since we are both lacking manpower and try the impossible: supporting an insanely high amount of different platforms.

 

IMO these are the problems that need solutions or at least a strategy since obviously the way we do it now does not work at all (not at this scale with way too many boards and platforms and that amount of people willing to spend time on the project). What were the results of trying to solve these problems so far? Many hours/days spent on discussions and then... nothing (some actionism and the usual lack of communication and coordination again).

 

 

[Igor]

7 minutes ago, tkaiser said:

Do we need to force bleeding edge mainline kernel always?

No, we don't. Current Allwinner next branch has yet another problem. sun8i support was merged into sunxi and now it is a bit difficult to go back until upstream gets solved. The idea is to stay still on 4.14.y and apply bugfix patches only. OK, but unfortunately there is a mess outside our power atm and we will probably need to jump further when the time is right. 

 

Regarding u-boot I vote for a conservative approach. We don't update it unless it's tested very well or necessary. This will require switching repository upgrading to semi-manual mode. 

 

11 minutes ago, tkaiser said:

But with updates ending up with bricked boards it's a real mess since we destroy productive (server) installations.

I picked two boards (C2 and Opi 2e), where I will do all possible upgrade scenarios together with new/changed features part. This is manageable but I didn't cover all problems this way, well at least most of the userspace ones. Even we don't have dedicated testers I hope someone will join at least testing part to find problems.

[TonyMac32]

36 minutes ago, tkaiser said:

discussion how to handle bug fixes and hot fixes first, then implementing something that suits these needs

Well, right.  In this case, I think it's easy enough to say everything gets dumped into development by default, I was just trying to make sure I in particular didn't create a giant git shitstorm.  . 

 

I agree that this sort of thing needs discussion and agreement, with an emphasis on scaling to this project.  Professionally working in the automotive industry, I will be the first to throw a flag if anything is too complex or overbearing, since I would rather this not go down into that abyss.

 

For u-boot, I'd honestly never change it unless our scripting required some new feature or a board was still in the process of gaining support (Tinker S needed a patch to U-boot on Rockchip, so whenever it is properly supported I'd ideally test then update.  Of course rk3288 is exactly 2 boards, not exactly a Herculean task to test.  With H3 I can see it being a royal pain.   

 

16 minutes ago, Igor said:

Regarding u-boot I vote for a conservative approach. We don't update it unless it's tested very well or necessary.

 

I would go conservative enough to say it would have to have a proven benefit.  Especially when our boot packages only cover a portion of the story, the boot scripts themselves depend on the binary, and user edits are something whose compatibility with upgrade you cannot test.

16 minutes ago, Igor said:

The idea is to stay still on 4.14.y and apply bugfix patches only.

That is what I intended with Meson64 and Rockchip, jumping LTS to LTS.

[chwe]

I try to update this:

 

A20:

@Igor

- legacy kernels should go out due too overall bad kernel quality

- reboot/poweroff problems on some boards

 

H3:

@chwe

- CSI (camera) doesn't work on mainline, gc2035 needs an small script in legacy to start up properly

-

Amlogic:

@TonyMac32

- Mainline support effort is still relatively new.  Things like USB, audio, etc are WIP.

- "Legacy" kernel that is stable is old and EOL (3.14) and I haven't fooled with it.  In that world and all others Amlogic, @balbes150 is the master. 

- Newer legacy kernel is still a work in progress (4.9 w/bugs)

- u-boot blobs are lame and make using mainline u-boot difficult (need both legacy 2015 and mainline u-boot to get a working image)

- Lower quality of open source information, all more or less community driven by a very small group, unlike with say Rockchip with a large amount of published info from Rockchip themselves.  Rumor/commits say this may change with newer IoT processors.

 

Freescale:

@Igor

- the legacy kernel is getting obsolete. No point to have it.

- possible to implement hw video acceleration on mainline kernel

- Solidrun has new 4.9 kernel, but can be safely ditched for mainline

Rockchip (I suggest only the one we support at the moment, not RK3399):

@chwe

- CSI (tinkerboard, rk3288) is supported (RK 4.4 bsp kernel) by kernel (OV6547 OV5640, imx219 - RPI-cam V1.3 resp. V2) but doesn't show up properly when booting). See github issue and related thread.

-Wifi driver seems to be problematic for some 'configurations'. 

@TonyMac32

- Igor and tkaiser are already seeing this, but mkimage/uboot/trust can be finicky.  luckily that's  a "figure it out and move on" situation, but it is likely to cause headaches.

- 4.4 kernel is an increasingly old kernel, now 2 LTS's in the past, however still much better than a lot of others (also has a lot of activity)

- RKwifi is something I don't know the reason for.  I don't know why it exists, I don't know what it does (other than cause problems).  It is a wrapper for some wifi drivers/an rfkill system

- Rockchip likes to use their own subsystems in many things, like the "MPP" system for media playing.  This has different capabilities per SoC, so may not behave predictably to simply be able to say we know how to use it.

 

 Marvell ARMADA:

@Igor

- missing SFP support in mainline

 

Armbian related scripts:

@chwe

- h3consumption is legacy only

-

[jeanrhum]

Concerning the testing phase of Armbian, I was participating in previous "effort" using https://desk.armbian.com/public/index.php.

I can continue to make some end-user tests (installation, upgrade, functionality test) if a process or something else is defined. I owned a few "armbian" boards based on A20, H3 and A64 and maybe planned to buy a more recent one based on rk3399 if it is cheap enough. Currently, I have not enough knowledge and time to come to the dev side.

[chwe]

Since we have now the development branch implemented in armbians build script, we may need some 'rules' and procedures how and when we merge things to master to ensure they don't break something but don't get lost in the dev branch forever. A small annotation for external contributors when they should send PRs to which branch could also be helpful.

[TonyMac32]

1 hour ago, chwe said:

we may need some 'rules' and procedures

I think a gerneral one is to have very granular branches dedicated to specific things, but that isn't so practical I think.  I saw we rolled out new Meson64 images, but the changes haven't been rolled into Master.  My Git is not amazingly strong, however, would this be the proper case to use tags?

[zador.blood.stained]

13 minutes ago, TonyMac32 said:

I think a gerneral one is to have very granular branches dedicated to specific things, but that isn't so practical I think. 

Having at least per SoC family branches + a general build script improvements branch would be easier to test and merge one by one IMO. Currently we will have to cherry-pick a lot of separate commits if we wanted to put tested parts to master.

 

13 minutes ago, TonyMac32 said:

My Git is not amazingly strong, however, would this be the proper case to use tags?

No, tags only mark specific commits in the repository. We may need branches (which contain commit chains).

[TonyMac32]

2 minutes ago, zador.blood.stained said:

Currently we will have to cherry-pick a lot of separate commits if we wanted to put tested parts to master.

 

I was worried about that.  Well, I can get to work on my part of the mess.

[Igor]

11 hours ago, TonyMac32 said:

I can get to work on my part of the mess.

I need a week or two to get my mess into the order. Perhaps we write down changes on the way?

[TonyMac32]

OK.  Do you want to handle the upstream patches part?  The first ones on Tinker were in a joint commit with 4.4 and 4.9 upstream. 

 

Should we write them down in the 5.42 thread?

[Igor]

9 minutes ago, TonyMac32 said:

OK.  Do you want to handle the upstream patches part?  The first ones on Tinker were in a joint commit with 4.4 and 4.9 upstream. 

Was this meant for Zador or me?

 

10 minutes ago, TonyMac32 said:

Should we write them down in the 5.42 thread?

Yes, better there.

[TonyMac32]

22 minutes ago, Igor said:

Was this meant for Zador or me?

 

It's about the commits with upstream patches, If I cherry pick the ones for tinker board there's other stuff in them.

[chwe]

Do we have some policies how to handle such questions?

 

 

on the other hand we have PRs like this one:

https://github.com/armbian/build/pull/927

 

Where we clean up the kernel. For the SoC related stuff it's obvious that Armbian might differ from SoC to SoC with modules, for BSP kernels it's also obvious that some drivers aren't available due to not available for this kernel or nobody wrote a patch to implement them. But is there a way to keep the user experience as similar as possible in case of compiled drivers? Means if it's added to one, it's added to all (where possible). Adding a driver to default might then need valid reasons why it should be default and when this is not the case, users are forced to build their own kernel with the drivers they need. 

[Igor]

6 hours ago, chwe said:

Do we have some policies how to handle such questions?

Nothing special. I encourage people to create a PR but since this is less work to just add then explaining to people what to do. I would propose to add this to new web page's FAQ, somewhere among medium important questions. 

 

6 hours ago, chwe said:

Means if it's added to one, it's added to all

 

Cleaning as https://github.com/rfrht made should be done on all kernels and if there is some request we usually translate it to other kernels ... but sometimes enabling on one might break compilation, while it will work on some other kernel. 

 

[zador.blood.stained]

Instead of finding and cherry-picking specific commits it may be easier to just check out the "development" branch on top of the "master" without switching branches and then manually git add/git add -p and commit stable and tested changes for specific topics (i.e. rk32xx, rk33xx, mvebu64, cubox, etc.). When using a GUI application (i.e. gitg on Linux and git-gui on Windows) it's relatively easy to see and group necessary changes (i.e. config/sources/... + config/kernel/... + patch/kernel/... for any kernel related updates).

 

And I hope that we can revert all Bionic related hacks from the development branch and try to add it the proper way (using this to simplify things), since even though I want to work on some things using the development branch I don't want to turn my build host OS into a mix of Xenial and Bionic packages, especially since I'm using this system not only for the Armbian.

[Igor]

2 hours ago, zador.blood.stained said:

And I hope that we can revert all Bionic related hacks from the development branch and try to add it the proper way

OK. Going safer way can save us unexpected troubles. I am fine with such approach and I will migrate main build host to Bionic ASAP that it is possible to put out Bionic (nightly) builds. If everything is working correctly we move recommendation to Bionic and voila. Other things should be O.K. and perhaps we should just merge the rest, dump development branch for a while and focus fixing bugs on master only. The development branch is reinitiated for next development cycle?