Bash alternative to Reflector for Ranking Mirrors
So if you don't already know, I am a long-time user and supporter of Arch Linux. Arch uses a package management tool called pacman that works similarly to yum or apt, but much better IMHO. It uses a list of mirrors to perform the actual downloading of the package files, so you want the fastest mirrors to be in the mirror list. The old way is to use reflector to rank the speed of the mirrors, which is a python script. My way is pure bash using curl, sed, awk, xargs, and sort. Very simple and IMHO more effective than reflector.
Creating /etc/pacman.d/mirrorlist
Just run the script and redirect the output to /etc/pacman.d/mirrorlist like this:
$ ./reflector.sh | sudo tee /etc/pacman.d/mirrorlist
How it works
Well it's simple, essentially it performs these steps:
- Fetch the current list of (only current 100%) mirrors from the official site.
- Use curl to request a small 257 byte file from each of those mirrors (about 200-300), 40 at a time and save the fastest 50. This also gets the dns cached for the next step
- Use curl to request a 100 kb file from each of those mirrors, measuring the total time of the request, and the speed of the download, 10 at a time.
- Finally, merge the results of both of those tests into a list of 50 mirrors in the format for outputting directly to
/etc/pacman.d/mirrorlist
Why pure bash over reflector?
Well because I like my systems extra crazy lean, I often don't want to install python right after an initial install of Arch. Also, this is much faster and easier on the system resources, and I believe it is also more accurate. I'd like to encourage others to turn to pure shell scripting to do simple tasks like this, often that is a better long-term solution than building a new piece of software. But I'm not against reflector, it's a pretty awesome bit of python with many features.
reflector.sh Source
Download reflector.sh
#!/bin/bash # Updated: Tues May 07 21:04:12 2013 by webmaster@askapache # @ https://www.askapache.com/shellscript/reflector-ranking-mirrors/ # Copyright (C) 2013 Free Software Foundation, Inc. # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. # if mirrors exists, cat it, otherwise create it function get_mirrors () #{{{1 { if [[ -s $MIRRORS ]]; then cat $MIRRORS; else curl -LksS -o - 'https://www.archlinux.org/mirrors/status/json/' | \ sed 's,{,\n{,g' | sed -n '/rsync/d; /pct": 1.0/p' | sed 's,^.*"url": "\([^"]\+\)".*,\1,g' > $MIRRORS cat $MIRRORS; fi } function get_core_urls () #{{{1 { get_mirrors | sed "s,$,core/os/${ARCH}/core.db.tar.gz,g" } function get_gcc_urls () #{{{1 { get_mirrors | sed "s,$,core/os/${ARCH}/${GCC_URL},g" } # rm tmp file on exit trap "exitcode=\$?; (rm -f \$MIRRORS 2>/dev/null;) && exit \$exitcode" 0; trap "exit 1" 1 2 13 15; # file containing mirror urls MIRRORS=`(mktemp -t reflector-mirrorsXXXX) 2>/dev/null` && test -w "$MIRRORS" || MIRRORS=~/reflector.mirrorsXXX # arch ARCH=`(uname -m) 2>/dev/null` || ARCH=x86_64 # the gcc file GCC_URL=$( curl -LksSH --url ftp://ftp.archlinux.org/core/os/${ARCH}/ 2>/dev/null | sed -n 's/^.*\ \(gcc-[0-9]\+.*.tar.xz.sig\)\ -.*$/\1/gp' ); { # faster as primarily used to pre-resolve dns for 2nd core test get_gcc_urls | xargs -I'{}' -P40 curl -Lks -o /dev/null -m 3 --retry 0 --no-keepalive -w '%{time_total}@%{speed_download}@%{url_effective}\n' --url '{}' |\ sort -t@ -k2 -nr | head -n 50 | cut -d'@' -f3 | sed 's,core/os/'"${ARCH}/${GCC_URL}"',$repo/os/$arch,g' get_core_urls | xargs -I'{}' -P10 curl -Lks -o /dev/null -m 5 --retry 0 --no-keepalive -w '%{time_total}@%{speed_download}@%{url_effective}\n' --url '{}' |\ sort -t@ -k2 -nr | head -n 50 | cut -d'@' -f3 | sed 's,core/os/'"${ARCH}"'/core.db.tar.gz,$repo/os/$arch,g' } | sed 's,^,Server = ,g' | awk '{ if (!h[$0]) { print $0; h[$0]=1 } }' exit $?;
Xargs running curl in parallel
Just shows the output of htop while running the script.
« Htaccess Rewrite for Redirecting Uppercase to LowercaseAlienware M18xR2 Review of Dells fastest Laptop »
Comments