FREE THOUGHT · FREE SOFTWARE · FREE WORLD

Home  »  Shell Scripting  »  Bash alternative to Reflector for Ranking Mirrors

by comment

So if you don't already know, I am a long-time user and supporter of Arch Linux. Arch uses a package management tool called pacman that works similarly to yum or apt, but much better IMHO. It uses a list of mirrors to perform the actual downloading of the package files, so you want the fastest mirrors to be in the mirror list. The old way is to use reflector to rank the speed of the mirrors, which is a python script. My way is pure bash using curl, sed, awk, xargs, and sort. Very simple and IMHO more effective than reflector.


Creating /etc/pacman.d/mirrorlist

Just run the script and redirect the output to /etc/pacman.d/mirrorlist like this:

$ ./reflector.sh | sudo tee /etc/pacman.d/mirrorlist

Bash alternative to Reflector for Ranking Mirrors

How it works

Well it's simple, essentially it performs these steps:

  1. Fetch the current list of (only current 100%) mirrors from the official site.
  2. Use curl to request a small 257 byte file from each of those mirrors (about 200-300), 40 at a time and save the fastest 50. This also gets the dns cached for the next step
  3. Use curl to request a 100 kb file from each of those mirrors, measuring the total time of the request, and the speed of the download, 10 at a time.
  4. Finally, merge the results of both of those tests into a list of 50 mirrors in the format for outputting directly to /etc/pacman.d/mirrorlist

Why pure bash over reflector?

Well because I like my systems extra crazy lean, I often don't want to install python right after an initial install of Arch. Also, this is much faster and easier on the system resources, and I believe it is also more accurate. I'd like to encourage others to turn to pure shell scripting to do simple tasks like this, often that is a better long-term solution than building a new piece of software. But I'm not against reflector, it's a pretty awesome bit of python with many features.

reflector.sh Source

Download reflector.sh

#!/bin/bash
# Updated: Tues May 07 21:04:12 2013 by webmaster@askapache
# @ http://www.askapache.com/shellscript/reflector-ranking-mirrors.html
# Copyright (C) 2013 Free Software Foundation, Inc.
#
#   This program is free software: you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation, either version 3 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
# if mirrors exists, cat it, otherwise create it
function get_mirrors () #{{{1
{
   if [[ -s $MIRRORS ]]; then
          cat $MIRRORS;
   else
          curl -LksS -o - 'https://www.archlinux.org/mirrors/status/json/' | \
          sed 's,{,\n{,g' | sed -n '/rsync/d; /pct": 1.0/p' | sed 's,^.*"url": "\([^"]\+\)".*,\1,g' > $MIRRORS
          cat $MIRRORS;
   fi
}
 
function get_core_urls () #{{{1
{
   get_mirrors | sed "s,$,core/os/${ARCH}/core.db.tar.gz,g"
}
 
function get_gcc_urls () #{{{1
{
   get_mirrors | sed "s,$,core/os/${ARCH}/${GCC_URL},g"
}
 
# rm tmp file on exit
trap "exitcode=\$?; (rm -f \$MIRRORS 2>/dev/null;) && exit \$exitcode" 0;
trap "exit 1" 1 2 13 15;
 
# file containing mirror urls
MIRRORS=`(mktemp -t reflector-mirrorsXXXX) 2>/dev/null` && test -w "$MIRRORS" || MIRRORS=~/reflector.mirrorsXXX
 
# arch
ARCH=`(uname -m) 2>/dev/null` || ARCH=x86_64
 
# the gcc file
GCC_URL=$( curl -LksSH --url ftp://ftp.archlinux.org/core/os/${ARCH}/ 2>/dev/null | sed -n 's/^.*\ \(gcc-[0-9]\+.*.tar.xz.sig\)\ -.*$/\1/gp' );
 
{
   # faster as primarily used to pre-resolve dns for 2nd core test
   get_gcc_urls | xargs -I'{}' -P40 curl -Lks -o /dev/null -m 3 --retry 0 --no-keepalive -w '%{time_total}@%{speed_download}@%{url_effective}\n' --url '{}' |\
   sort -t@ -k2 -nr | head -n 50 | cut -d'@' -f3 | sed 's,core/os/'"${ARCH}/${GCC_URL}"',$repo/os/$arch,g'
 
   get_core_urls | xargs -I'{}' -P10 curl -Lks -o /dev/null -m 5 --retry 0 --no-keepalive -w '%{time_total}@%{speed_download}@%{url_effective}\n' --url '{}' |\
   sort -t@ -k2 -nr | head -n 50 | cut -d'@' -f3 | sed 's,core/os/'"${ARCH}"'/core.db.tar.gz,$repo/os/$arch,g'
} | sed 's,^,Server = ,g' | awk '{ if (!h[$0]) { print $0; h[$0]=1 } }'
 
exit $?;

Xargs running curl in parallel

Just shows the output of htop while running the script.

Bash alternative to Reflector for Ranking Mirrors

Tags

May 7th, 2013

Comments Welcome

Popular Articles
My Online Tools

Related Articles
Newest Posts
Twitter


  • @askapache · Jul 21
    Magic spells for sending thoughts across time? Books
  • @askapache · Jul 20
    TV is just a relic of the previous generation. We just don't know it yet.
  • @askapache · Jul 20
    I will never go back on my ideals, no matter the cost. I'll never let the economic vultures steal my dreams. I'd rather give up the ghost
  • @askapache · Jul 18
    I don't want a better seat, I want control of the engine
  • @askapache · Jul 14
    No matter how good u r, there will always be someone 2x, 5x, 100x better. This is true for me and everyone. No direction but forward.
  • @askapache · Jul 12
    Heads up, I'll DDoS the f out of askapache next week, to see how resilient it really is :) - will try to overflow disk, net, and ip stack
  • RUN GCC! This is a typical shirt I wear, from the  t.co/46LYbFr4k2  shop. A clerk at the LQ recognized it!  t.co/jjmT0dkCPu 
  • Merlin the Magician  t.co/iMmRbanUi4 
  • ROGUE CODE - Latest novel from @markrussinovich  t.co/apkn0LoPIt 
  • RTFM - surprisingly very helpful and way more comprehensive than it looks! @redteamfieldman #pwnAllTheThings  t.co/xiaJ5g0aC9 
  • Dear Hacker - Letters to the Editor of 2600, from Emmanuel Goldstein  t.co/JCfLab7FAJ 
  • The Mythical Man-Month - Essays on Software Engineering, by Frederick P. Brooks, Jr.  t.co/ilWN5GHElr 
  • "where wizards stay up late" - The Origins of the Internet. Favorite book detailing the birth of the net and IMPs  t.co/gY9VTGJgZz 
  • ZERO DAY - read before Trojan horse  t.co/pPMLGDJv8P 
  • Trojan Horse, a novel!  t.co/Hf8EtYaZVa 
  • The Hacker Playbook - very nice high level overview of attacks  t.co/lHwNVWi61u 
  • Clean Code - A Handbook of Agile Software Craftsmanship  t.co/hnJX0x1qIc 
  • Secrets of the JavaScript Ninja - By my absolute favorite JS hacker John Resig!  t.co/tZ42ljmcCl 
  • Hacking Exposed 7: Network Security Secrets & SolutionsMy all time favorite, basic but thorough and accurate.  t.co/jycW0RDVtZ 
  • Empty words will be no surrogate for cold resolve. Pain is nothing.  t.co/qXjpRxbjCw 

Hacking and Hackers

The use of "hacker" to mean "security breaker" is a confusion on the part of the mass media. We hackers refuse to recognize that meaning, and continue using the word to mean someone who loves to program, someone who enjoys playful cleverness, or the combination of the two. See my article, On Hacking.
-- Richard M. Stallman






[hide]

It's very simple - you read the protocol and write the code. -Bill Joy

Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution 3.0 License, just credit with a link.
This site is not supported or endorsed by The Apache Software Foundation (ASF). All software and documentation produced by The ASF is licensed. "Apache" is a trademark of The ASF. NCSA HTTPd.
UNIX ® is a registered Trademark of The Open Group. POSIX ® is a registered Trademark of The IEEE.

| Google+ | askapache

Site Map | Contact Webmaster | License and Disclaimer | Terms of Service

↑ TOPMain