FREE THOUGHT · FREE SOFTWARE · FREE WORLD

Home  »  Optimization  »  DNS Round Robin Configuration using Rsync over SSH

by 8 comments

Round Robin DNS

My traffic is growing, alot, and I need to plan how I'm going to maintain scalability, high availability, and redundancy. Scalability is an application's ability to support a growing number of users. High availability can be defined as redundancy or speed. I decided to setup Round Robin DNS for static.askapache.com, which is the "static" subdomain of AskApache that serves all the static assets like images, javascript, css, etc.. (BTW, the z stands for ZAP).. All I needed to attempt setting this up was another hosting account on a separate server. I have hosting accounts with around 10 different companies from working with various clients over the years, like Powweb and I don't use them because they suck in terms of the unix environment. Many of these web hosts are actually very fast bandwidth-wise..


Round Robin Concept

A few months ago I was given a free hosting account on HostGator to evaluate and tempt me away from DreamHost to become a Gator. I get a lot of these types of offers from time to time from the smaller Web companies who read AskApache.. but when I noticed the SSH access was jailed and saw how restrictive the shell was I felt like I was on a windows box.. I want a shell, cpanel sucks. I compile and run everything from the shell so thats was a deal-breaker and I sorta forgot all about it.

The goal is to add the HostGator server to be an exact mirror of the static.askapache.com domain, then to add that server as a 2nd A record to my DNS zone. That way half the visitors to the size will be taking up resources and bandwidth on the HostGator server instead of mine.

Round Robin A records in DNS are intended to evenly distribute queries between each host of the same name. Using some tricks straight out of a hackers toolbox we can verify if the distribution is taking place. (It is.)

DNS - Domain Name System

The Domain Name System is often analogized as a "phone book" for the Internet because it translates human-friendly computer hostnames into IP addresses. For example, www.askapache.com translates to 208.113.134.190. Every request for a human-friendly hostname first needs to be translated to the IP before the server can be queried. Normally each hostname corresponds to exactly 1 IP address. But in a Round Robin setup, the idea is to have the hostname correspond to multiple IP addresses, which are different servers that house the exact same content, resulting in some hosts requesting files from one server, and another host requesting files from the other server, resulting in less CPU resources and bandwidth.

Here is an the zone for static.askapache.com Round Robin records:

QUERYTTLCLASSTYPEANSWER
static.askapache.com13084INA64.111.114.111
static.askapache.com13084INA69.56.174.114

Round Robin DNS

Round robin DNS is a technique of load distribution, load balancing, or fault-tolerance provisioning multiple, redundant Internet Protocol service hosts, e.g., Web servers, FTP servers, by managing the Domain Name System's (DNS) responses to address requests from client computers according to an appropriate statistical model.

In its simplest implementation Round-robin DNS works by responding to DNS requests not only with a single IP address, but a list of IP addresses of several servers that host identical services. The order in which IP addresses from the list are returned is the basis for the term round robin. With each DNS response, the IP address sequence in the list is permuted. Usually, basic IP clients attempt connections with the first address returned from a DNS query so that on different connection attempts clients would receive service from different providers, thus distributing the overall load among servers.

Round robin DNS is often used for balancing the load of geographically-distributed Web servers. For example, a company has one domain name and three identical web sites residing on three servers with three different IP addresses. When one user accesses the home page it will be sent to the first IP address. The second user who accesses the home page will be sent to the next IP address, and the third user will be sent to the third IP address. In each case, once the IP address is given out, it goes to the end of the list. The fourth user, therefore, will be sent to the first IP address, and so forth.

Although easy to implement, round robin DNS has problematic drawbacks, such as those arising from record caching in the DNS hierarchy itself, as well as client-side address caching and reuse, the combination of which can be difficult to manage. Round robin DNS should not solely be relied upon for service availability. If a service at one of the addresses in the list fails, the DNS will continue to hand out that address and clients will still attempt to reach the inoperable service.

Does Round Robin Work

Definately. I can look at the access logs for both servers and see that they are splitting the requests nicely. It is definately not an exact split however, look at these statistics and you'll see what I mean.

$ dig @ns1.dreamhost.com +authority +all static.askapache.com

;; ANSWER SECTION:
static.askapache.com.        14400   IN      A       69.56.174.114
static.askapache.com.        14400   IN      A       64.111.114.111


$ dig @ns1.dreamhost.com +authority +all static.askapache.com

;; ANSWER SECTION:
static.askapache.com.        14400   IN      A       64.111.114.111
static.askapache.com.        14400   IN      A       69.56.174.114

The effects of caching will distort the effectiveness of any IP address allocation algorithm unless a 0 TTL is used which has the effect of significantly increasing the load on the DNS (and is not always implemented consistently). In this case the cure may be worse than the disease Good news we have good load balancing on our web servers. Bad news we need 17 more DNS servers!. Intuitively, and without running any experiments to verify, we would suggest that given a normal TTL (12 hours or more) and ANY IP allocation algorithm other than a single static list, loads should be reasonably balanced .


Full root server query

Tracing to static.askapache.com[a] via A.ROOT-SERVERS.NET, maximum of 3 retries
A.ROOT-SERVERS.NET [.] (198.41.0.4)
 |___ L.GTLD-SERVERS.NET [com] (192.41.162.30)
 |     |___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      ___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |___ D.GTLD-SERVERS.NET [com] (192.31.80.30)
 |     |___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      ___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |___ J.GTLD-SERVERS.NET [com] (192.48.79.30)
 |     |___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      ___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |___ H.GTLD-SERVERS.NET [com] (192.54.112.30)
 |     |___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      ___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |___ E.GTLD-SERVERS.NET [com] (192.12.94.30)
 |     |___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      ___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |___ F.GTLD-SERVERS.NET [com] (192.35.51.30)
 |     |___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      ___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |___ I.GTLD-SERVERS.NET [com] (192.43.172.30)
 |     |___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      ___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |___ G.GTLD-SERVERS.NET [com] (192.42.93.30)
 |     |___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      ___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |___ B.GTLD-SERVERS.NET [com] (2001:0503:231d:0000:0000:0000:0002:0030) Not queried
 |___ B.GTLD-SERVERS.NET [com] (192.33.14.30)
 |     |___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      ___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |___ A.GTLD-SERVERS.NET [com] (2001:0503:a83e:0000:0000:0000:0002:0030) Not queried
 |___ A.GTLD-SERVERS.NET [com] (192.5.6.30)
 |     |___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      ___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |___ C.GTLD-SERVERS.NET [com] (192.26.92.30)
 |     |___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      ___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |___ M.GTLD-SERVERS.NET [com] (192.55.83.30)
 |     |___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      ___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
  ___ K.GTLD-SERVERS.NET [com] (192.52.178.30)
       |___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
       |___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
        ___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer

  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114

Live Online DNS Tools

More Reading

RFC's related to DNS

  • RFC 920: Specified original TLDs: .arpa, .com, .edu, .org, .gov, .mil and two-character country codes
  • RFC 1032: Domain administrators guide
  • RFC 1033: Domain administrators operations guide
  • RFC 1034: Domain Names - Concepts and Facilities.
  • RFC 1035: Domain Names - Implementation and Specification
  • RFC 1101: DNS Encodings of Network Names and Other Types
  • RFC 1123: Requirements for Internet Hosts -- Application and Support
  • RFC 1912: Common DNS Operational and Configuration Errors
  • RFC 1995: Incremental Zone Transfer in DNS
  • RFC 1996: A Mechanism for Prompt Notification of Zone Changes (DNS NOTIFY)
  • RFC 2136: Dynamic Updates in the domain name system (DNS UPDATE)
  • RFC 2181: Clarifications to the DNS Specification
  • RFC 2182: Selection and Operation of Secondary DNS Servers
  • RFC 2308: Negative Caching of DNS Queries (DNS NCACHE)
  • RFC 2317: Classless IN-ADDR.ARPA delegation
  • RFC 2671: Extension Mechanisms for DNS (EDNS0)
  • RFC 3597: Handling of Unknown DNS Resource Record (RR) Types
  • RFC 3696: Application Techniques for Checking and Transformation of Names
  • RFC 4343: Domain Name System (DNS) Case Insensitivity Clarification
  • RFC 4592: The Role of Wildcards in the Domain Name System
  • RFC 4892: Requirements for a Mechanism Identifying a Name Server Instance
  • RFC 5001: DNS Name Server Identifier Option (NSID)
  • RFC 5395: Domain Name System (DNS) IANA Considerations
[isc_wikicop title="Round_robin_DNS"]

Tags

April 14th, 2012

Comments Welcome

  • timmy

    Wow, great article, and the bonus of listening to Kasabian. Hey any chance you could post your rsync script you are using to keep these static asset directories dialed in?

  • http://www.askapache.com/ AskApache

    @timmy

    I just add this to my crontab:

    */30 * * * * /usr/bin/rsync -e 'ssh' -rt --delete /pub/static.askapache.com/ gator:'/pub/static.askapache.com/'

    I detailed exactly how to do it in the previous article, Mirroring with rsync over SSH.. really this was a single article but I had to split it since I like to include all the verbose details.

  • Tim E

    Seriously? Music? Who the heck still does that? I was going to use this site for reference but it's just far too annoying. I don't know what you were thinking.

  • niku

    http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/dns-round-robin-is-useless.html

    "Round-robin has no real use. It's not even a poor man's load balancer."
    ---

    nice site.

  • bobiter

    I've read scores of articles claiming that round robin does not work for load balancing and they are just plain wrong. Apparently all the articles were written by people who didn't actually implement round robin and try it.

    I've been using round robin for years and it works fine if configured correctly. Granted some DNS servers don't implement it correctly but that doesn't stop it from working for most users.

    I've seen goofy suggestions to instead use a single server that then passes the load around on backend servers. But DUH what if that single server goes down?

    I know round robin works because I travel the country a lot and bring up our company site from all sorts of different computers. Each page has an encoded, tiny signature showing where it originated and it clearly shows it working. We use a TTL of 300 and it works fine. DNS requests are not that big a load on a server compared to serving up a whole website. From a single computer, using all major browsers, the originating server changes every five minutes.

    Jonathan Deboynepollard has wrote some great stuff that is very useful but his article on round robin is just simply wrong.

    My advice to talking tech heads? Try stuff before you write a whole article based on theory.

  • http://www.askapache.com/ AskApache

    @bobiter

    Thanks for your well-written comment. It was great for me to hear about someone else with real-world success with this. Of course, if you dig for A records of google.com, or most any giant domain, you will see very similar setups to Round Round-Robin, though they are focused more on load-balancing.

    I actually disengaged the round-robin setup as it became too much of a hassle to keep the mirrors current as I can do an enormous amount of editing and modifications to the site files somewhat regularly.

    What I have now is running subdomains on different servers so all requests to that subdomain will be handled by a sep server.

    In the future I will write more about the DNS strategies I'm testing and learning about to guarantee uptime of services. So far I have load-balanced DNS, next is to guarantee HTTP.

    @bobiter, it's too bad everyone doesn't share the same philosophy of doing your own research.. hopefully people don't confuse my personal research results as being expert advice.

  • http://vincent.maule.name Vince

    Why not look into Cloudflare, implementing that with your site???


Related Articles


My Online Tools
Popular Articles


Hacking and Hackers

The use of "hacker" to mean "security breaker" is a confusion on the part of the mass media. We hackers refuse to recognize that meaning, and continue using the word to mean someone who loves to program, someone who enjoys playful cleverness, or the combination of the two. See my article, On Hacking.
-- Richard M. Stallman









[hide]

It's very simple - you read the protocol and write the code. -Bill Joy

Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution 3.0 License, just credit with a link.
This site is not supported or endorsed by The Apache Software Foundation (ASF). All software and documentation produced by The ASF is licensed. "Apache" is a trademark of The ASF. NCSA HTTPd.
UNIX ® is a registered Trademark of The Open Group. POSIX ® is a registered Trademark of The IEEE.

+Askapache | askapache

Site Map | Contact Webmaster | License and Disclaimer | Terms of Service

↑ TOPMain