Upgrading to DreamHost Private Servers

« Adding Akismet Anti-Spam Protection AnywhereFast and Easy Custom WordPress New User Registration »

Upgrading to DreamHost Private Servers

I’ve recently relocated to the DreamHost private server setup, not for any reason other than its such a cool offer that I had to take advantage of it to learn more about it.

DreamHost Private Server

DreamHost PS uses Linux-VServer to give you your own “virtual machine”, thereby protecting your CPU and RAM from all other users on your physical machine.

Linux-VServer

Linux-VServer provides virtualization for GNU/Linux systems. This is accomplished by kernel level isolation. It allows to run multiple virtual units at once. Those units are sufficiently isolated to guarantee the required security, but utilize available resources efficiently, as they run on the same kernel. This particular virtual server model is implemented through a combination of “security contexts”, segmented routing, chroot, extended quotas and some other standard tools.

What Bothers Me about DreamHostPS

1. They don’t have the skill/desire to let you run only certain sites/usernames on the PS, its all or nothing, which completely sucks!
2. It’s so darn expensive, I spent more after the first 24 hours than I usually do in a MONTH
3. Although they say you still have the backups of your site, the .snapshot folder is no longer accessible and you have to manually contact support to ask for a backup! Ahh!
4. The environment on the shell doesn’t have as much access to good stuff like using locate
5. Processes can get out of control and use up to much memory if you have a busy site like me, which results in the sending of 503’s to everyone else!
6. Still uses NFS, the slowest thing in the world and not worth it now that backups aren’t even accessible.. it can also cause some hard-to-detect issues with caching setups

What Thrills me about DreamHostPS

1. I have more access and control over my server, sites, memory, and files via SSH login
2. I can scale the CPU and Memory whenever I want, and just like the static IP’s you only pay for the time you use, making experimenting affordable!
3. I can reboot the whole server at anytime! Sometimes useful if a script I’m experimenting with locks the whole server..
4. Only my shell accounts can access the server! Security is much stronger!
5. You can configure other webservers like lighttpd, nginx, and fnord to serve content on sites that are mostly static content!
6. I’ve only had it for 3 days, so I’m sure I’ll find alot more awesome capabilities

Problems migrating to DreamHostPS

Here are some issues I experienced during migration and the solutions I’ve used.

During this process it is important to note how helpful the DreamHost Support Staff were in putting up with my sometimes overly technical and detailed support requests. Thanks John, Brian, and Robert R!

PHP and HTTPD processes hogging all memory

Unlike on shared hosting accounts, where DH technical people have set up a very robust system, it appears they are missing the expertise of a past employee or something because this new setup is not as robust.. YET!

For instance I started out my account CPU and memory at the MAX (2300 MB / 2300 Mhz) but my sites were all still taking forever to serve content, simply because instead of on the shared servers where user processes and HTTPD instances are more controlled, this account seems to not have very well-thought out limits on it. So if 100 people asked for a page on my site, this server loads up 100 HTTPD processes under dhapache user and loads up 100 processes for the custom-compiled php.cgi I am running. This sounds like a cool thing but in reality it takes up so much of my memory that my bash shell login under SSH runs out of memory and won’t even let me do a simple ps, and it just keeps serving 503’s to anyone else who requests something on my site. DH will have to fix this soon or someone will launch a DDOS attack that will cripple them, unless a googlebot does it first!

Solution
I contacted support and received a very friendly and prompt reply suggesting a bad script and offering to setup a process watcher and killer, which I accepted. Eventually I located the problem to be an ErrorDocument 500 directive in my .htaccess that was pointing to a php file instead of a static .html

No crontabs or cronjobs

None of my crontab files were moved to my new server and in fact I was receiving permission denied just to access my crontab.

Solution
I contacted support and they installed new crontabs for me and offered to copy my old ones.

Static IP Changed for site with non-DreamHost DNS

One site uses DNS from Network Solutions, so when my site was migrated and got a new static IP address, my site went down. It would have been nice and should be expected that in this situation DreamHost would alert you that the change is going to happen so you can update your DNS without your site going offline.

Solution
Logged into my Network Solutions account and updated the DNS for my site to point to the new Static IP.

SSH Hosts, Authorized Keys Broken

Some of my sites and user accounts use passwordless SSH to make some things work, and all of these were made useless when I moved to my new private server.

Solution

1. Logged into my user accounts with SSH
2. Deleted the old files in folder .ssh
3. Created new keys and added them to other accounts
4. Logged in to new accounts to add to host files

Old Server and Static IP References in Site Files

I have some pretty technical and complex cgi’s, .htaccess files, shell scripts run by cronjobs, php scripts, etc., on some sites and shell accounts, and many of my files contain code to the Static IP and/or dreamhost server, either for access control or for faster connects by connecting straight to an IP instead of having to perform a DNS lookup. So when both the Static IP’s and dreamhost server changed it broke all my files.

Solution
Basically I knew I had to search all of my files and replace the old IP with the new IP. I also had to search files relacing the old server with my new server. To make life simpler, I wrote a simple shell script that I run from my account while logged in using SSH that does this automatically with the added feature of asking me if I would like to make the replacement for each file it finds, which is nice because I don’t want to replace this for old log files and misc stuff.

dreamhostps migration shell script

#!/bin/bash
# Version 1.0 by AskApache 5/29/2008

shopt -s extglob
renice 19 $$

OLDSERVER=208.113.183.103
NEWSERVER=208.113.134.190

FIXFILES=$(grep -R -l -i $OLDSERVER $HOME/!(Maildir|logs|backups|source|tmp|doit|php5|php526|ip_abuse) 2>/dev/null)

for thefile in ${FIXFILES[@]}; do
 if [ -f "$thefile" ]; then
  echo -e "\n\n\n\n"
  echo "___________________________________________________________________"
  echo "Name:  ${thefile}"
  echo "Type:  $(command file -b ${thefile})"
  echo "Size:  $(command du -hs ${thefile}|awk '{ print $1}')"
  echo "Matching Lines:"
  grep -i --color=auto $OLDSERVER $thefile
  echo -e "___________________________________________________________________\n"
  echo -en "Replace occurances of $OLDSERVER with $NEWSERVER? [y/N] " ; read -n 1 ans
  case "$ans" in
   n|N) echo -e "\nSKIPPING..."; ;;
   y|Y) echo -e "\nREPLACING..."
        cp $thefile $thefile.b1 &>/dev/null
        cat $thefile.b1 | sed "s/${OLDSERVER}/${NEWSERVER}/g" 1>$thefile
        rm $thefile.b1 &>/dev/null
        echo "DONE"; ;;
  esac
 fi
done
exit 0

Other Webservers Allowed on DreamHostPS

Lightweight web servers are Web servers which have been designed to run with very small resource overhead because of hardware, environment, or simply for the challenge of it.

Many of these systems have been created as a mental exercise to determine if a modern webserver could be written to run on limited resources such as those provided in a graphing calculator, an ancient Commodore 64 machine, or in 64 kB (64 KiB) total of memory. Others have been written as commercial endeavors to create webservers with low overhead for embedded systems (network router configuration pages) or low memory environments.

Apache webserver on DH is configured in process-based mode. This means that each process serves one simultaneous connection and uses significant amount of memory for that, thus limiting concurrent user count. You can download and install some lightweight web-servers like (lighttpd) or (nginx). These servers use async i/o and can handle large count of concurrent connections without consuming much RAM, especially when serving static files. But default port 80 is already occupied by apache, and you cannot change that. So, you can use another port for new lightweight server, but visitors will see ugly port number in address and such solution will not work for some corporate firewalls.

1. lighttpd – small memory footprint compared to other web-servers, effective management of the cpu-load, and advanced feature set
   • FastCGI
   • SCGI
   • Auth
   • Output-Compression
   • URL-Rewriting
   • many more
2. nginx – lightweight web server/reverse proxy and e-mail (IMAP/POP3) proxy
   • Handling of static files, index files and auto-indexing
   • Reverse proxy without caching, load balancing, and fault tolerance
   • SSL support
   • FastCGI support
   • Name- and IP-based virtual servers
   • FLV streaming
3. fnord – lightweight webserver written by Felix von Leitner. It aims to be a small, but extremely fast and secure webserver.
   • Small! (13k static Linux-x86 binary without CGI, 18k with CGI)
   • Fast! Uses mmap (and on Linux, sendfile)
   • connection keep-alive
   • el-cheapo virtual domains (similar to thttpd)
   • IPv6 support (through tcpserver)
   • CGI (through pipes, not temp files like Apache)
   • Content-Range (not the full specs, just a-b or a- byte ranges)
   • transparent content negotiation
   • Directory index generation

DreamHost PS Troubleshooting

1. Symptoms of an Overloaded DreamHost PS
2. Checking your Daily & Monthly Resource Usage
3. Changing your CPU & Memory Allocation
4. Rebooting your DreamHost PS
5. Overloaded DreamHost PS
6. Idle DreamHost PS
7. Changing web server
8. See also
9. External link

Symptoms of an Overloaded DreamHost PS

If you experience any of the following symptoms, most likely you will need to increase the resource allocation (CPU & memory) to your Private Server:

1. Out of memory
2. Internal server errors ("500" errors)
3. Killed scripts
4. Inability to log in ("ssh_exchange_identification: Connection closed by remote host")

Checking your Daily & Monthly Resource Usage

Check the daily & monthly usage graph in the Control Panel under (PRIVATE SERVERS > CPU/MEMORY) to see if your usage is going beyond your currently guaranteed CPU and memory allocation.

Changing your CPU & Memory Allocation

You can increase or decrease your resource allocation by moving the green slider in the Control Panel under (PRIVATE SERVERS > CPU/MEMORY). As you move it the amount of CPU and memory will update automatically, along with the rate you’ll be charged for that setting. Once you’re happy with the setting click on the "Change {servername}’s CPU / Memory Now!" button to push the change into place. It will take a short period of time for the setting change to be reflected. Typically no reboot is necessary.

Rebooting your DreamHost PS

If you have a problem you can try rebooting your Private Server yourself in the Control Panel under (PRIVATE SERVERS > REBOOT SERVER). No need to contact support for that! Isn’t that cool?! It may temporarily fix things, but if it doesn’t provide a long-term solution, you will need to increase the resources allocated to your Private Server by using the green slider as mentioned in the previous paragraph.

Overloaded DreamHost PS

The image above is what your daily usage graph might look if you’ve got your settings too low. Notice that the actual usage routinely exceeds the guaranteed amount of 150MHz/MB. This is bound to cause problems for your sites. To rectify the situation it’s recommended that you increase your resource allocation. Then keep an eye on the graph for several hours, and test our sites to see if you’ve allocated enough resources for things to run smoothly. It is recommended that you start out by doubling your current resource allocation to see if it’s enough. Once you’ve verified that things are running properly you can reduce your resource allocation to the point where your peaks just barely exceed what you have allocated. Of course you’ll want to routinely monitor your usage and increase the resource allocation as your needs increase. It’s best to over allocate then under allocate! You don’t want to find out that you’ve under allocated by your visitors/customers complaining about your sites not working properly.

Note that you’re only charged for the period of time that you have the slider in a particular position. So it’s safe to experiment. In fact, we recommend it. You can increase or decrease your resource allocation at any time.

You will typically see Apache processes running on your server and appearing to consume all of the memory. This is generally not the case because Apache processes share a significant amount of memory between one another. Additionally, we automatically configure Apache to work well within the memory allocation of your PS server. It is still possible for a busy website to overwhelm a DreamHost PS server, but it is not generally the fault of the apache webserver itself.

Of course you may also want to try to reduce your load on the server as well so you can reduce the resource allocation and save some money.

Idle DreamHost PS

The image above is what your daily usage graph might look if you’ve got nothing running under your DreamHost PS. This just shows the overhead resource usage. You could host a lightly loaded web page with this resource allocation (provided it uses static content), but probably not much more. If you’ve got your resource allocation set to 150MHz/MB it is recommendended that you monitor the usage very often.

Some Content on this page included from this article by Author History from the DreamHost Wiki and is licensed under the GNU FDL.

« Adding Akismet Anti-Spam Protection Anywhere
Fast and Easy Custom WordPress New User Registration »

MP3 Caper