Nitrax Posted December 6, 2021 Posted December 6, 2021 Very often when i do apt update or apt dist-upgrade some of the armbian mirrors acts up and pretty much denies connections (it isnt the generic debian mirrors as other distros are working fine). I assume they are just overloaded and theres not much to do about that. But it has caused one of my system to be out of date for a few weeks I assume because the unattended upgrade script doesnt retry. How do I get this to work? for example some times i have to retry up to 10 times before I can get apt to work (when I try manually). The apt configuration in etc is a bit of magic to me so Id appreciate if someone can guide me to get unattended upgrades to retry connections.
Werner Posted December 7, 2021 Posted December 7, 2021 Hm this should not happen. We do have an error count for existing mirrors. Maybe try to fix to a static one? https://github.com/armbian/mirror/
Igor Posted December 7, 2021 Posted December 7, 2021 17 hours ago, Nitrax said: Very often when i do apt update or apt dist-upgrade some of the armbian mirrors acts up Which ones do you have troubles with? Any specific mirror or in general? Check also here: https://github.com/armbian/dl-router/issues @lanefu There is indeed a bug in mirror redirector - for some reason it doesn't reload mirror table and broken ones are still in it. Will be fixed ASAP.
lanefu Posted December 7, 2021 Posted December 7, 2021 Hey sorry I had the apt running on a test instance. Been trying to solve the big that changes the request protocol after a while.
Nitrax Posted December 7, 2021 Author Posted December 7, 2021 Ok thanks! But Ill also try to set a static one, for some reason i often get stuff that looks chinese even though Im in sweden.
Set3 Posted December 12, 2021 Posted December 12, 2021 Hi ( Had some account problems, long time not logged in.... ) I found mismatches in at least one mirror I use apt-mirror to get a local apt copy so I can run without internet access and still update/upgrade/install at will apt-mirror uses file date for comparison and that is where it goes wrong Every now and then my Orange Pi One devices complain that there is a problem in file size : Err:8 http://MYLOCALREPO/mirror/apt.armbian.com bionic/main armhf Packages File has unexpected size (153052 != 303677). Mirror sync in progress? A few times I deleted the whole armbian mirror folder and reloaded ( 60GB :-( few hours) , which solver the problem Not this time Did some digging Normally I mirror apt.armbian.com So nowI also mirrored the http://mirror.armbian.de/apt/ ( Another yes 60GB download ) and found a difference in CONTENT in 1 file, but they had the same time stamp : problem ! That is a problem in at least this mirror : https://stpete-mirror.armbian.com/apt/dists/bionic/ That site has an InRelease file dated 28-11 but inside the file it says Date: Mon, 15 Nov 2021 14:22:59 UTC All these mirrors are OK, at least for this specific file, this time that is.... http://armbian.12z.eu/apt/ http://armbian.hosthatch.com/apt/ http://armbian.systemonachip.net/apt/ http://armbian.tnahosting.net/apt/ http://imola.armbian.com/apt/ http://mirror.12z.eu/pub/linux/armbian/apt/ http://mirror.armbian.de/apt/ http://mirrors.bfsu.edu.cn/armbian/ http://mirrors.dotsrc.org/armbian-apt/ http://mirrors.nju.edu.cn/armbian/ http://mirrors.tuna.tsinghua.edu.cn/armbian/ http://mirrors.ustc.edu.cn/armbian/ http://stpete-mirror.armbian.com/apt/ http://xogium.performanceservers.nl/apt/ So now trying : Just entered the URL apt.mirror.com/dists/boinic then we get forwarded to one of the mirrors, then downloaded InRelease and checked no others found yet Hope this helps you find the error !
Set3 Posted December 12, 2021 Posted December 12, 2021 So, well, yes, someone could every night (or more often as you wish), run apt-mirror on all mirrors (works incremental) , then run a compare files on content and presto, you have your mirror health in one go. at least until the problems are solved an apt-mirror on one mirror take s a few minutes, until an update is needed 60GB includes arm32 and arm64 brute force : file compare with ffs takes 1.5hr on a network share 80-100 M Byte p s can be done more intelligent
lanefu Posted December 12, 2021 Posted December 12, 2021 4 hours ago, Set3 said: So, well, yes, someone could every night (or more often as you wish), run apt-mirror on all mirrors (works incremental) , then run a compare files on content and presto, you have your mirror health in one go. Need to integrate https://github.com/armbian/mirror With https://github.com/armbian/dl-router 4 hours ago, Set3 said: Normally I mirror apt.armbian.com Rsync.armbian.com Or mirrors.dotsrc.org Dotsrc is fast
lanefu Posted December 12, 2021 Posted December 12, 2021 On 12/7/2021 at 11:24 AM, Nitrax said: Ok thanks! But Ill also try to set a static one, for some reason i often get stuff that looks chinese even though Im in sweden. It does it's best via geoip but data can be old. setting a mirror via armbian-config is an option
lanefu Posted December 12, 2021 Posted December 12, 2021 4 hours ago, Set3 said: That is a problem in at least this mirror : https://stpete-mirror.armbian.com/apt/dists/bionic/ That site has an InRelease file dated 28-11 but inside the file it says Date: Mon, 15 Nov 2021 14:22:59 UTC Thanks for catching that
Igor Posted December 13, 2021 Posted December 13, 2021 23 hours ago, Set3 said: That is a problem in at least this mirror Thank you for your research. Here is how we do it: # upload to NAS rsync -arH --exclude _h5ai/ --exclude "dists" "${HOME_BASE}"/output/repository/. igorp@${NAS}:/tank/armbian/repository # we sync index only when old enough or at once if we have beta repository if [[ `stat --format=%Y "${HOME_BASE}"/output/repository/db/LOG` -le $(( `date +%s` - ${WAITINSEC} )) || $1 == "beta" ]] then rsync -arH --exclude _h5ai/ --exclude "stats/" --delete "${HOME_BASE}"/output/repository/. igorp@${NAS}:/tank/armbian/repository fi After repository is recreated, we synch everything but index. Index got synched only if 24h nothing is changed. Which also needs around 1 day to get around mirrors. Now if this principle can be made better, I am all ears. Beta repository is currently hosted on one machine, which is why we don't need this.
Set3 Posted December 16, 2021 Posted December 16, 2021 Hi Igor, As said, "can be done more intelligent", but sync is not the main issue here I now realize. My problem is solved for now I switched to the .de mirror, and changed all apt files, in my Orange Pis, via my DIY bash based distribution system and all works. But we should use the apt-armbian.com as much as possible right ? so until solved. But lets repair this for all. Problem So if today ( file is still in error ) someone does apt update bionic armhf or arm64 and get this stpete mirror, they cant update/upgrade/install Somewhere in the creation/sync process on stpete mirror, something went terribly wrong Same for the InRelease and Release file I guess that this specific error might be solved when a new InRelease /Release is created, fingers crossed ? It is a different file : differences: 1- the date in the content of the file does not match the file datetime in file Date: Mon, 15 Nov 2021 14:22:59 UTC vs file date 2021-11-28 18:57 3. Complete different file sizes inside the InRelease / Release file, look like 2 - 3 times larger ? e.g. file Contents-arm64 : 559420517 vs 204947365 That is what apt update is complaining about : wrong file size how did this happen, Is InRelease / Release created locally on stpete ? Are they using a different source to copy / run a different program to create this file ? --------------------------------------------------------------------------------------------------------------------------------------- on .de mirror -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Origin: Armbian Label: Armbian Suite: bionic Codename: bionic Date: Sun, 28 Nov 2021 18:57:36 UTC Architectures: arm64 armhf Components: bionic-desktop bionic-utils main Description: Generated by aptly MD5Sum: 713f40b776be3e429bf4d490078fcebb 204947365 Contents-arm64 62aa28ca7533d9b851cee738e05aeb0c 11483603 Contents-arm64.gz 12b178bf3f7b725bd794dab5a91ac53f 192630392 Contents-armhf 420271a676b17787f5e066bca4b802d5 11234465 Contents-armhf.gz ... --------------------------------------------------------------------------------------------------------------------------------------- on stpete -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Origin: Armbian Label: Armbian Suite: bionic Codename: bionic Date: Mon, 15 Nov 2021 14:22:59 UTC Architectures: arm64 armhf Components: bionic-desktop bionic-utils main Description: Generated by aptly MD5Sum: c352cdecd45eaea57e1487fcff099476 559420517 Contents-arm64 870b03f3bcba70dfaba9c91ba1e4ea78 31939133 Contents-arm64.gz 8da79c257b6cf1d57b25d8126d2a773c 543748921 Contents-armhf aa6b322f000304e98f0d4f3b218988f0 32387711 Contents-armhf.gz ... --------------------------------------------------------------------------------------------------------------------------------------- So who can solve that ? Hope this helps.
lanefu Posted December 16, 2021 Posted December 16, 2021 3 hours ago, Set3 said: So who can solve that ? Hope this helps. I've pulled armbian.de mirror out of rotation for now... can you try another mirror? http://apt.armbian.com/mirrors
Set3 Posted December 16, 2021 Posted December 16, 2021 Hmm, Not sure I understand you, but I think you pulled the wrong one. Just to be clear : I think mirror.armbian.de is ok stpete-mirror.armbian.com has the wrong InRelease file Few days ago I made a wget script to pull all InRelease files from all mirrors in the mirrors list you referred to, all match, except the one from stpete Now don't get confused below : ( please read twice :-) I don't mirror stpete directly (yet), I currently I mirror apt.armbian.com mirror.armbian.de Just ran a compare again, and there is a mismatch only in the InRelease file I guess that I got that file from stpete via apt.armbian.com allocating me to stpete :-)
lanefu Posted December 17, 2021 Posted December 17, 2021 18 hours ago, Set3 said: apt.armbian.com are you using rsync or a web spider? apt.armbian.com is a redirect service.
lanefu Posted December 17, 2021 Posted December 17, 2021 On 12/16/2021 at 9:18 AM, Set3 said: how did this happen, Is InRelease / Release created locally on stpete ? Are they using a different source to copy / run a different program to create this file ? St Pete was syncing from a secondary source. Im not sure what happened but we were in the process of converting it to our primary source so must have been something amiss. It looks okay looking at the file dates now. Still having issues?
Set3 Posted December 23, 2021 Posted December 23, 2021 Let me try again : Did you say you pulled .de out or the mirror rotation ? In my humble opinion : I think that is a mixup stpete should be taken out of the mirror rotation .de mirror is ok stpete stil has the wrong file So users still can get the error message mentioned. For now I stay on the .de mirror, that is working fine
Igor Posted December 25, 2021 Posted December 25, 2021 On 12/24/2021 at 12:44 AM, Set3 said: stpete should be taken out of the mirror rotation .de mirror is ok .de is synching from stpete ... It is normal that index is different since ... if we would sync all at once, more critical problems would happen. For example: Main rsync and mirror X are having latest and greatest index, but not others. Files are also still synching and this can take up to 24h. You will run apt update and index will tell you that there are new files. OK, lets upgrade them. Upgrade process will request a file and redirector (apt.armbian.com) will select some mirror (at each file request can be different) which has no files yet and you will run into fatal error "Fetch error ... " Index is different (for some time) by purpose. It's a feature, not a bug.
Matthijs Kooijman Posted March 8, 2022 Posted March 8, 2022 I've also been seeing the size mismatch errors during builds and investigated a bit further. I believe the problem is caused by the combination of mirrors being out of sync (which is unavoidable, I do not think it is feasible to sync them all at the same time), and apt.armbian.com redirecting to different mirrors between requests (as Igor also pointed out). I created a github issue with some more details and a suggested fix here: https://github.com/armbian/dl-router/issues/29 I believe the fix for this would be to make apt.armbian.com do consistent redirects (i.e. redirect the same IP to the same mirror all the time, or at least for longer periods of time). @Igor, you said above that "It's a feature, not a bug", are you referring to the unavoidability of mirrors being out of sync, or also the inconsistent redirects? IOW, are you suggesting this is not fixable (or should not be fixed)?
Recommended Posts