Mirrors are unreliable, causing frequent failures of installs / upgrades

[dirkhh]

I started seeing this happen "now and then" a couple of months ago, but over the last month it has gone to "multiple times a day". It's not always the same mirror. Frequently it's the airframes.io mirrors, this morning it was lv.auroradev.org

 

The pattern is usually like this:

 

Err:1 https://armbian.lv.auroradev.org/apt bookworm/main arm64 linux-libc-dev arm64 23.02.2
  404  Not Found [IP: 23.186.113.5 443]
E: Failed to fetch https://armbian.lv.auroradev.org/apt/pool/main/l/linux-6.1.11-sunxi64/linux-libc-dev_23.02.2_arm64.deb  404  Not Found [IP: 23.186.113.5 443]

 

Is there a place to report these issues to get those mirrors fixed?

[Igor]

59 minutes ago, dirkhh said:

Is there a place to report these issues to get those mirrors fixed?

 

We don't have a dedicated operator of mirrors but it seems we should appoint someone as maintaining is getting complicated despite of all automation which should prevent this to happen. This is a corner case, as it shows that mirror is synced ... but not all files are there.

 

I have removed this mirror as its probably running out of space. Airframes maintainer responded yesterday, so I assume its back in sync.

[dirkhh]

Thanks for the quick response, @Igor I know that others are running into this - I am wondering if there is a way to get engaged and help.

 

But one particular question that I have is about the mirror redirector. Because when I get a failure response and retry, it would be amazing if the redirector could recognize that I just asked for that same file and that maybe it shouldn't direct me to that same mirror again? A simple redis cache with a short timeout for originating IP plus requested URL and if you see it a second time, given them a different answer? Just a thought...

[Igor]

1 minute ago, dirkhh said:

I am wondering if there is a way to get engaged and help.

 

You could study the code and improve it. Also we haven't merge PR with lots of improvements as nobody had time to test it:
https://github.com/armbian/armbian-router

 

Mirrors are stored in database and here we are pulling them out:
https://github.com/armbian/os/blob/main/.github/workflows/redirector-update.yml

From there a config is generated that is passed to armbian-router AKA re-director.

 

4 minutes ago, dirkhh said:

Because when I get a failure response and retry, it would be amazing if the redirector could recognize that I just asked for that same file and that maybe it shouldn't direct me to that same mirror again?

Perhaps new version knows that. You will need to try the one from PR and open a ticket there.

[dirkhh]

thanks for the pointers. I'll look at the code and see if I can contribute some improvements.

really appreciate the responsiveness!

[dirkhh]

Hi @Igor -- is it possible that the broken mirror is back? I just got this in a couple of builds:

2023-10-01T18:48:04.0451876Z Ign:2 http://armbian.lv.auroradev.org/apt bookworm/main arm64 armbian-firmware all 23.8.3
2023-10-01T18:48:04.0461377Z Err:2 http://apt.armbian.com bookworm/main arm64 armbian-firmware all 23.8.3
2023-10-01T18:48:04.0462970Z   404  Not Found [IP: 23.186.113.5 80]
2023-10-01T18:48:04.0677050Z Get:1 http://mirrors.jevincanders.net/armbian/apt bookworm/main arm64 base-files arm64 23.8.3-bookworm [53.0 kB]
2023-10-01T18:48:04.2046316Z Fetched 53.0 kB in 1s (76.1 kB/s)
2023-10-01T18:48:04.2050986Z E: Failed to fetch http://apt.armbian.com/pool/main/a/armbian-firmware/armbian-firmware_23.8.3_all__1-SA01f9-B981c-R448a.deb  404  Not Found [IP: 23.186.113.5 80]
2023-10-01T18:48:04.2051716Z E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

 

[Igor]

16 minutes ago, dirkhh said:

is it possible that the broken mirror is back?

Yes. I disabled it in the databases ... didn't test it as I assumed it should just work  But pulling were ignoring property.
https://github.com/armbian/os/commit/a423fcbccaeb95cb9569cef582309cfe9748256a

 

Try now.