apt armbian mirrors wonky, how to mitigate?

[Nitrax]

Very often when i do apt update or apt dist-upgrade some of the armbian mirrors acts up and pretty much denies connections (it isnt the generic debian mirrors as other distros are working fine). I assume they are just overloaded and theres not much to do about that. But it has caused one of my system to be out of date for a few weeks I assume because the unattended upgrade script doesnt retry.

 

How do I get this to work? for example some times i have to retry up to 10 times before I can get apt to work (when I try manually). The apt configuration in etc is a bit of magic to me so Id appreciate if someone can guide me to get unattended upgrades to retry connections. 

 

 

[Werner]

Hm this should not happen. We do have an error count for existing mirrors. Maybe try to fix to a static one?

https://github.com/armbian/mirror/

[Igor]

17 hours ago, Nitrax said:

Very often when i do apt update or apt dist-upgrade some of the armbian mirrors acts up

Which ones do you have troubles with? Any specific mirror or in general?

 

Check also here:
https://github.com/armbian/dl-router/issues

 

@lanefu

 

There is indeed a bug in mirror redirector - for some reason it doesn't reload mirror table and broken ones are still in it. Will be fixed ASAP.

[lanefu]

Hey sorry I had the apt running on a test instance.   Been trying to solve the big that changes the request protocol after a while.

[Nitrax]

Ok thanks! But Ill also try to set a static one, for some reason i often get stuff that looks chinese even though Im in sweden.

[Set3]

Hi ( Had some account problems, long time not logged in.... )

 

I found mismatches in at least one mirror

I use apt-mirror to get a local apt copy so I can run without internet access and still update/upgrade/install at will

apt-mirror uses file date for comparison and that is where it goes wrong

Every now and then my Orange Pi One devices complain that there is a problem in file size :

Err:8 http://MYLOCALREPO/mirror/apt.armbian.com bionic/main armhf Packages
  File has unexpected size (153052 != 303677). Mirror sync in progress?

A few times I deleted the whole armbian mirror folder and reloaded ( 60GB :-( few hours) , which solver the problem

Not this time

Did some digging

Normally I mirror apt.armbian.com So nowI also mirrored the http://mirror.armbian.de/apt/ ( Another yes 60GB download )

and found a difference in CONTENT in 1 file, but they had the same time stamp : problem !

That is a problem in at least this mirror :

https://stpete-mirror.armbian.com/apt/dists/bionic/

 

That site has an InRelease file dated 28-11 but inside the file it says

    Date: Mon, 15 Nov 2021 14:22:59 UTC

 

All these mirrors are OK, at least for this specific file, this time that is....

http://armbian.12z.eu/apt/
http://armbian.hosthatch.com/apt/
http://armbian.systemonachip.net/apt/
http://armbian.tnahosting.net/apt/
http://imola.armbian.com/apt/
http://mirror.12z.eu/pub/linux/armbian/apt/
http://mirror.armbian.de/apt/
http://mirrors.bfsu.edu.cn/armbian/
http://mirrors.dotsrc.org/armbian-apt/
http://mirrors.nju.edu.cn/armbian/
http://mirrors.tuna.tsinghua.edu.cn/armbian/
http://mirrors.ustc.edu.cn/armbian/
http://stpete-mirror.armbian.com/apt/
http://xogium.performanceservers.nl/apt/

 

So now trying :

Just entered the URL apt.mirror.com/dists/boinic

then we get forwarded to one of the mirrors, then downloaded InRelease and checked

no others found yet

Hope this helps you find the error !

[Set3]

So, well, yes, someone could every night (or more often as you wish),  run apt-mirror on all mirrors (works incremental) , then run a compare files on content and presto, you have your mirror health in one go.

at least until the problems are solved

 

an apt-mirror on one mirror take s a few minutes, until an update is needed

 

60GB includes arm32 and arm64

brute force : file compare with ffs takes 1.5hr on a network share 80-100 M Byte p s

can be done more intelligent

[lanefu]

4 hours ago, Set3 said:

So, well, yes, someone could every night (or more often as you wish),  run apt-mirror on all mirrors (works incremental) , then run a compare files on content and presto, you have your mirror health in one go.

 

Need to integrate

 

https://github.com/armbian/mirror

 

With

 

https://github.com/armbian/dl-router

 

4 hours ago, Set3 said:

Normally I mirror apt.armbian.com

 

 

 

Rsync.armbian.com

Or mirrors.dotsrc.org

 

Dotsrc is fast

 

 

[lanefu]

On 12/7/2021 at 11:24 AM, Nitrax said:

Ok thanks! But Ill also try to set a static one, for some reason i often get stuff that looks chinese even though Im in sweden.

 

It does it's best via geoip but data can be old.  setting a mirror via armbian-config is an option

[lanefu]

4 hours ago, Set3 said:

That is a problem in at least this mirror :

https://stpete-mirror.armbian.com/apt/dists/bionic/

 

That site has an InRelease file dated 28-11 but inside the file it says

    Date: Mon, 15 Nov 2021 14:22:59 UTC

 

Thanks for catching that

[Igor]

23 hours ago, Set3 said:

That is a problem in at least this mirror

Thank you for your research.

 

Here is how we do it:
 

# upload to NAS
rsync -arH --exclude _h5ai/ --exclude "dists" "${HOME_BASE}"/output/repository/. igorp@${NAS}:/tank/armbian/repository

# we sync index only when old enough or at once if we have beta repository
if [[ `stat --format=%Y "${HOME_BASE}"/output/repository/db/LOG` -le $(( `date +%s` - ${WAITINSEC} )) || $1 == "beta" ]]
then
rsync -arH --exclude _h5ai/ --exclude "stats/" --delete "${HOME_BASE}"/output/repository/. igorp@${NAS}:/tank/armbian/repository
fi

After repository is recreated, we synch everything but index. Index got synched only if 24h nothing is changed. Which also needs around 1 day to get around mirrors. Now if this principle can be made better, I am all ears. Beta repository is currently hosted on one machine, which is why we don't need this.

[Set3]

Hi Igor,

 

As said, "can be done more intelligent", but sync is not the main issue here I now realize.

 

My problem is solved for now

     I switched to the .de mirror, and changed all apt files, in my Orange Pis, via my DIY bash based distribution system and all works.

     But we should use the apt-armbian.com as much as possible right ? so until solved.

 

But lets repair this for all.

 

Problem

    So if today ( file is still in error ) someone does apt update bionic armhf or arm64 and get this stpete mirror, they cant update/upgrade/install

    Somewhere in the creation/sync process on stpete mirror, something went terribly wrong

 

    Same for the  InRelease and Release file

 

I guess that this specific error might be solved when a new InRelease /Release is created, fingers crossed ?

 

It is a different file : differences:

 

1- the date in the content of the file does not match the file datetime

    in file         Date: Mon, 15 Nov 2021 14:22:59 UTC

    vs

    file date    2021-11-28 18:57

 

3. Complete different file sizes inside the InRelease / Release  file, look like 2 - 3 times larger ?

    e.g. file Contents-arm64 : 559420517 vs 204947365

 

    That is what apt update is complaining about : wrong file size

 

 

how did this happen, Is InRelease / Release created locally on stpete ? Are they using a different source to copy / run a different program to create this file ?

 

---------------------------------------------------------------------------------------------------------------------------------------

on .de mirror

 

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Origin: Armbian
Label: Armbian
Suite: bionic
Codename: bionic
Date: Sun, 28 Nov 2021 18:57:36 UTC
Architectures: arm64 armhf
Components: bionic-desktop bionic-utils main
Description: Generated by aptly
MD5Sum:
 713f40b776be3e429bf4d490078fcebb 204947365 Contents-arm64
 62aa28ca7533d9b851cee738e05aeb0c 11483603 Contents-arm64.gz
 12b178bf3f7b725bd794dab5a91ac53f 192630392 Contents-armhf
 420271a676b17787f5e066bca4b802d5 11234465 Contents-armhf.gz

...

---------------------------------------------------------------------------------------------------------------------------------------

on stpete

 

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Origin: Armbian
Label: Armbian
Suite: bionic
Codename: bionic
Date: Mon, 15 Nov 2021 14:22:59 UTC
Architectures: arm64 armhf
Components: bionic-desktop bionic-utils main
Description: Generated by aptly
MD5Sum:
 c352cdecd45eaea57e1487fcff099476 559420517 Contents-arm64
 870b03f3bcba70dfaba9c91ba1e4ea78 31939133 Contents-arm64.gz
 8da79c257b6cf1d57b25d8126d2a773c 543748921 Contents-armhf
 aa6b322f000304e98f0d4f3b218988f0 32387711 Contents-armhf.gz

...
---------------------------------------------------------------------------------------------------------------------------------------

 

So who can solve that ?

 

Hope this helps.

 

[lanefu]

3 hours ago, Set3 said:

So who can solve that ?

 

Hope this helps.

 

I've pulled armbian.de mirror out of rotation for now... can you try another mirror?

http://apt.armbian.com/mirrors

 

 

[Set3]

Hmm, Not sure I understand you, but I think you pulled the wrong one.

 

Just to be clear : I think

    mirror.armbian.de  is ok

    stpete-mirror.armbian.com  has the wrong InRelease file

 

Few days ago I made a wget script to pull all InRelease files from all mirrors in the mirrors list you referred to, all match, except the one from stpete

 

 

Now don't get confused below : ( please read twice :-)

 

I don't mirror stpete directly (yet), I currently I mirror

   apt.armbian.com

   mirror.armbian.de

Just ran a compare again, and there is a mismatch only in the InRelease file

 

I guess that I got that file from stpete via apt.armbian.com allocating me to stpete  :-)

 

 

 

[lanefu]

18 hours ago, Set3 said:

apt.armbian.com

 

are you using rsync or a web spider?   apt.armbian.com is a redirect service.

[lanefu]

On 12/16/2021 at 9:18 AM, Set3 said:

how did this happen, Is InRelease / Release created locally on stpete ? Are they using a different source to copy / run a different program to create this file ?

 

St Pete was syncing from a secondary source.    Im not sure what happened but we were in the process of converting it to our primary source so must have been something amiss.     It looks okay looking at the file dates now.   Still having issues?

[Set3]

Let me try again :

 

Did you say you pulled .de out or the mirror rotation ?

In my humble opinion : I think that is a mixup

 

 

stpete should be taken out of the  mirror rotation

.de mirror is ok

 

stpete stil has the wrong file

 

So users still can get the error message mentioned.

 

For now I stay on the .de mirror, that is working fine

 

 

[Igor]

On 12/24/2021 at 12:44 AM, Set3 said:

stpete should be taken out of the  mirror rotation

.de mirror is ok

.de is synching from stpete ...  It is normal that index is different since ... if we would sync all at once, more critical problems would happen.

 

For example:

Main rsync and mirror X are having latest and greatest index, but not others. Files are also still synching and this can take up to 24h. You will run apt update and index will tell you that there are new files. OK, lets upgrade them. Upgrade process will request a file and redirector (apt.armbian.com) will select some mirror (at each file request can be different) which has no files yet and you will run into fatal error "Fetch error ... "

 

Index is different (for some time) by purpose. It's a feature, not a bug.

[Matthijs Kooijman]

I've also been seeing the size mismatch errors during builds and investigated a bit further. I believe the problem is caused by the combination of mirrors being out of sync (which is unavoidable, I do not think it is feasible to sync them all at the same time), and apt.armbian.com redirecting to different mirrors between requests (as Igor also pointed out).

I created a github issue with some more details and a suggested fix here: https://github.com/armbian/dl-router/issues/29

 

I believe the fix for this would be to make apt.armbian.com do consistent redirects (i.e. redirect the same IP to the same mirror all the time, or at least for longer periods of time).

@Igor, you said above that "It's a feature, not a bug", are you referring to the unavoidability of mirrors being out of sync, or also the inconsistent redirects? IOW, are you suggesting this is not fixable (or should not be fixed)?