≡ Menu

Geolocation detection on the web can be handy. You can locate where your visitors are coming from, and direct them to local content, or run a promotion in a specific country, or city. One of the best, and simplest solutions out there is the MaxMind geoIP database (www.maxmind.com). This article covers installing MaxMind on Centos5 (or RHEL), however most of these instructions apply to other Linux distributions as well.

1) Before we start, we need to make sure that both Apache (httpd) and the Apache Development packages are installed:

yum install httpd
yum install httpd-devel

Be aware, that on a 64bit system, you may need to add .x86_64 to the end of these packages to get the 64bit install.

2) Now we can being by installing the GeoIP API Libraries, which have already been packaged up for us by the CentOS community.

yum install GeoIP*

3) Now, we need to donwload and install the apache module from maxmind:

cd /tmp
wget http://www.maxmind.com/download/geoip/api/mod_geoip2/mod_geoip2_1.2.5.tar.gz
tar xvzf mod_geoip2_1.2.5.tar.gz
cd mod_geoip2_1.2.5
apxs -i -a -L/usr/lib -I/usr/include -lGeoIP -c mod_geoip.c

This will install the Apache 2 module. If for some reason you’re running Apache 1.3, any time you see “mod_geoip2”, simple change this to “mod_geoip” in the above commands. What we’ve done here is download, and compile the apache module for the GeoIP lookups.

4) The installation above will get you all setup with a country level database. If you don’t need any more than that, you can skip this step. If you’d like to have a city level lookup, then do the following:

cd /tmp
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
gunzip GeoLiteCity.dat.gz
cp GeoLiteCity.dat /var/lib/GeoIP/GeoLiteCity.dat

5) Next, lets remove the module from the main httpd.conf file and create it’s own config file. In your httpd.conf file (which should be located in: /etc/httpd/conf/httpd.conf), find and either remove or comment out the line that starts with “Load Module geooip_module“. Then create the following file: /etc/httpd/conf.d/geoip.conf containing the following:

LoadModule geoip_module      /usr/lib64/httpd/modules/mod_geoip.so
<IfModule mod_geoip.c>
GeoIPEnable On
GeoIPDBFile /var/lib/GeoIP/GeoIP.dat MemoryCache
GeoIPDBFile /var/lib/GeoIP/GeoLiteCity.dat MemoryCache
GeoIPOutput Notes
#  GeoIPOutput Env
#  GeoIPOutput All
</IfModule>

You’ll notice that both the GeoIP and GeoLiteCity databases are loaded. Comment out the GeoLiteCity database if you don’t need it. Below this, you’ll see three lines for GeoIPOutput. You have the choices of Apache notes, Environment variables, or both (all). Comment out the two lines you don’t need and enable the appropriate one for your use. In this case we have turned on Apache notes. You’ll also notice that beside the database lines, we’ve marked the file as “Memory Cache”. This does use additional RAM to cache the database, but speeds up access. For a small Database like the GeoIP Country database (which is 1MB), memory caching should be used. If you’re tight on memory in your server and can suffer the performance hit, you can change “Memory Cache” to “Standard” to turn caching off.

6) Now lets reload apache and pickup our changes:

apachectl -k graceful

7) Lastly, lets give it a test! Create a file in a web accessable directory, with the following code (eg: I created geotest.php):

<?php
/*
Uses mod-geoip to query the MaxMind GeoLite City binary database
and returns geographic information based on the client's IP address
*/
$country_code = apache_note("GEOIP_COUNTRY_CODE");
$country_name = apache_note("GEOIP_COUNTRY_NAME");
$city_name = apache_note("GEOIP_CITY");
$region = apache_note("GEOIP_REGION");
$metro_code = apache_note("GEOIP_DMA_CODE");
$area_code = apache_note("GEOIP_AREA_CODE");
$latitude = apache_note("GEOIP_LATITUDE");
$longitude = apache_note("GEOIP_LONGITUDE");
$postal_code = apache_note("GEOIP_POSTAL_CODE");

echo 'Country code: '.$country_code.'<br>';
echo 'Country name: '.$country_name.'<br>';
echo 'City name: '.$city_name.'<br>';
echo 'Region: '.$region.'<br>';
echo 'Metro code: '.$metro_code.'<br>';
echo 'Area code: '.$area_code.'<br>';
echo 'Latitude: '.$latitude.'<br>';
echo 'Longitude: '.$longitude.'<br>';
echo 'Postal code: '.$postal_code.'<br>';

print_r($_SERVER);

?>

Now, simply go to your webpage and you can check out the results! Use the php code in the above example to call this information in your web application.

{ 0 comments }

Working with remote servers and scripting can be a bit of a pain, since every time you connect to a remote machine, it’s going to prompt you for your password! But, there’s a simple and quick way around this.. SSH Keys! This is a quick little trick to allow one machine to login to another, without needing to input a password.

1) Generate the SSH Keypair
Your first step is to generate a SSH key on the machine, and as the user you want to SSH From. To do this, you need to run the following command:

ssh-keygen -t rsa

Accept all the defaults of this command when prompted. The resulting key files need to end up in the .ssh directory inside the users home directory to work properly. DO NOT enter a passphrase! Simply press enter twice when prompted to leave it empty.

2) Upload your SSH Key to the remote server
Now that we’ve created a key, we need to tell the other host what it is, so that it will accept our connection without a password. To do this, run the following command:

scp ~/.ssh/id_rsa.pub <user>@<host>:.ssh/authorized_keys

Replace <user> with your username, and <host> with the remote server name in the command above. Be careful with the authorized_keys file. If you’ve done this before, that file will already exist, and you’ll need to manually add the key to the end of that file. Once you’ve uploaded this file, DO NOT change the permissions on it, or the .ssh directory, as that may break things for you and make SSH connections as that user difficult.

3) Lets test it out!
This part is simple… SSH to the remote server as the user, with the following command:

ssh <user>@<host> ls

This should produce a directory listing, but not prompt you for your password to do so. Replace <user> with your username, and <host> with the remote server name again of course. And that’s that! You can now script away to your hearts content, accessing your remote server, and not be prompted for a password!

Note:
If you already have SSH keys in your .ssh/authorized_keys file on the remote machine, you may wish to use the following command while uploading your SSH key instead of the one shown in step 2.

cat ~/.ssh/id_rsa.pub | ssh <user>@<host> 'dd of=.ssh/authorizedkeys oflag=append conv=notrunc'

This will prevent you from accidentally overwriting an existing key in the file.

{ 0 comments }

In this post, I’m going to cover the basics behind optimizing an Apache/LAMP web server. There are many many great resources out there for getting into the nitty gritty of each of these topics, and I will be posting more about some of them on this blog over time, but simply googling a topic should get you a wealth of information about them.

Possibly one of the best books I’ve found as a linux resource is Linux Server Hacks (Link) Which also now has a volume 2 (also found on amazon). This book is a great resource for any Linux admin or hobbyist, and includes quick how-to summaries for common problems. Overall, a great reference book for those times where you need to quickly get an answer on how to do something without reading into a ton of detail about it.

When Optimizing LAMP/Apache, there are five key pieces to look at: 1) Hardware, 2) Linux, 3) Apache, 4) MySQL, 5) PHP. I’m going to cover the basics of each one, one at a time.

1 – Optimizing your hardware for web serving.
One of the first places to start, is to make sure you’re running on good hardware. Special consideration should be paid to what web apps your running. If you’re serving static HTML and Images, you can get away with minimal hardware. As you move into more dynamic sites, with lots of PHP and Database access, you’re going to want to look at adding more RAM and faster Hard Drives. In a web server, especially with the performance level of modern CPU’s, there are three places you’re likely to run into bottlenecks. 1) Network Bandwidth. The only way to solve this is to ensure you’ve selected a good hosting partner, and that they have a robust, low latency network. 2) Disk I/O. This is far less of an issue on small sites, but as your site grows, at least some of your traffic is going to be spent pulling data from the hard drive. There are several options to mitigate this, the most common being to move to SSD Hard drives, or to use RAID arrays (I recommend RAID-1 mirrored pairs for read heavy servers and limited budgets, and RAID-5 arrays of at least 4 drives for anything else). 3) Memory. The more RAM you have in your server, the less often the system needs to flush data out of it and go back to the hard drives to get it again. More RAM is also helpful in preventing swapping when server access gets high. RAM is cheap these days, so load up as much as you can!

2 – Linux
Now that you have your hardware selected, lets move on to the OS. Any linux distribution is fine, and you’re best to pick the one you’re most comfortable with, as it will make your administration easier. I personally like to use RedHat based distributions (CentOS, Feodra, or RHEL) as that’s what I’m more familiar with. FreeBSD or OpenBSD also make excellent web servers, and allot of the tips in this article hold true there as well. The first thing to do on the OS side is to strip away and disable anything you don’t need for your server. This not only adds security risks, but also consumes memory that can be used for other tasks. Things like Samba, and CUPS should all be disabled, or simply never installed. Next, patch all the various packages on the system, to ensure you have the most up to date versions for security and performance. Once that’s done, you should ensure that you configure IPTables if the server is directly connected to the internet. If you’re behind a firewall, this isin’t as important, but you may still wish to look into it.

3 – Apache
Current versions of Apache have a huge number of features and configuration options, so you really need to be careful not to take any one internet “how to” too literally, as some of the directions in an article may cause issues with the site you’re deploying. You should first start by disabling unused modules in the Apache httpd.conf file (typically found in /etc/httpd/conf/httpd.conf). Each unused module you load takes up memory space that you can better use for other things. A second place to look, which is essential in preventing crashing due to memory use, is the “MaxRequestsPerChild” variable in the worker and prefork modules. This limits how many requests an apache process will serve before it shuts down and re-spawns. There’s a fine balance here, between keeping memory use low, and site performance high, and you’ll need to tune this specifically to your needs. The larger the files/scripts on your site are, the larger the memory allocation of each Apache process will get over time, as it caches these files. A few other places to look when tuning the server include: Trying to avoid multiple .htaccess files in a directory tree, as each one needs to be loaded and checked before any request is served. Disable hostname lookups in your Apache configuration, as this adds a performance hit to resolve host names, that you likely don’t care about. This information can always be parsed in batch later by something like webalizer. If you’re really going for speed, you can disable apache logging, this does make troubleshooting harder, but saves you the disk I/O of writing logs. Lastly, the server-status and server-info pages (enabled by flags of the same name in the apache conf file) add a very small performance hit and should be disabled unless they’re being used specifically (eg: for Munin monitoring).

4 – MySQL
Tuning MySQL is somewhat akin to voodoo. That said, the goal with MySQL performance is to use the MySQL cache’s as often as possible, and avoid reading from Disk if you can help it. One of the best tools for perfomance tuning MySQL is the MySQLtuner pearl script (which can be downloaded on your system by using wget or curl, from here: http://mysqltuner.pl/mysqltuner.pl ). This script will analyze your MySQL install and offer tuning suggestions. Weigh these suggestions against the amount of RAM in your system (you may need to limit some of the increases due to memory constraints) and make the changes where appropriate. Once you’ve made and implemented the changes, come back the next day, and check again, and again. MySQL Optimization will take several days, as you need server accesses to properly gauge where the load is going. Another great tool for this is Munin, as the graphs it generates for MySQL can provide valuable insight over the long term to see what is happening. These two tools combined should provide you with more than enough details to make your MySQL install substantially faster than the out of the box version your OS was setup with.

5 – PHP
PHP is an interesting beast. Every time a script is executed, it has to be compiled first, and then run via an Apache module to produce results that are sent to the client. The best place to optimize PHP is on the script level. If you have control over writing the PHP code, start combing through it to see if there are routines that can be optimized, or database calls that can be trimmed down, as this will make the biggest difference in PHP Performance. If you don’t have that option, then the next biggest performance gain you’ll get is from an opcode cache. What this does is cache the compiled PHP scripts so that they can be re-used, and skip the compilation step each time they’re requested. Two of the most popular opcode caches are APC and eAccelerator. Be careful to monitor the opcode cache to ensure it’s working properly, as they can cause issues if not properly configured. Lastly, check your php.ini file, and ensure you’re memory limits are set to reasonable numbers. You want to provide PHP with just enough memory to run the largest script you have, but not such a large amount that it’s a waste.

Hopefully this provides a basic starting point of where to look for optimizations on your web server. I will be posting more articles on specific items here in the coming months. Stay Tuned!

{ 0 comments }

Copying files over SSH using tar

Sometimes IT is life and death… Perhaps you’re being chased by an army of zombies, while quickly trying to update a file on a remote server. Or maybe you’re hacking together a quick little backup script, before running off to a Halloween party in July (after one more slice of peach pie first of course). This is a quick little trick that I use quite often to move files around when all you have is 4 acorns, half a pack of gum, $0.13 in pennies and SSH access on two servers. Yes there are other ways to do this (e.g.: rsync), but this is nice and simple, and quick and easy for those times when you just need to get it done!

First, let me show you an example, and then we’ll break down how it works:

ssh [email protected] "cd /usr/folder/stuff; tar czf - ." | tar xvzf -

What we’re doing here is copying a file (or files) from a remote host back to the local host, and putting it in our current working directory on the local host. The command starts from the current machine, connects to a remote host (that’s the SSH part at the beginning). Once on the remote host, we go into the /usr/folder/stuff directory and tar up everything there. That’s the middle portion inside the Quotes. The last little part, after the | is the commands that are issued on the local host, to un-tar the file again.

“tar xzf – .” tells tar to archive (c), use gzip compression (z), and put the output into a file (f). The – tells tar to pipe the output to the ssh connection instead of an actual file, and the . tells it to pickup everything in the current directory. That second tar command is run using the xvzf flags for: extract (x), verbose (v), de-compress (z), and file (f), and define the file as -, so that it reads the incoming data stream.

To go the opposite direction, and push files to a remote system, instead of pulling them to you. You simply reverse the syntax:

cd /usr/folder/stuff
tar czf - . | "ssh [email protected]; tar xvzf -"

This will take the files in the /use/folder/stuff directory on the local machine, and place it in the users home directory on the remote machine. To do the same thing, but place the files in another directory, simply add a cd command to the remote side of the mix:

cd /usr/folder/stuff
tar czf - . | "ssh [email protected]; cd /remote/dir; tar xvzf -"

And off you go!

The gzip flag for tar (z) is most useful on slow network connections (eg: the internet). if you’re using this on a fast local network (ie: gigabit ethernet), you should probably skip that flag, as the CPU overhead to do the compression will actually slow you down compared to just transferring the files uncompressed.

{ 0 comments }

Installing Munin on Centos 5

Have you ever wondered what your linux server was doing when you weren’t looking? Maybe it skipped off to eat a bagel, or stopped to play with a passing puppy… Perhaps you just want to see how much load you’re actually putting on the server, so that you can load it up even more, like some kind of digital pack mule… This is where Munin comes in. Munin is an open source system monitoring and graphing tool. Below, we are going to briefly cover the install of Munin on a CentOS 5 system (Munin is available on most Linux flavours, FreeBSD, Windows, OS X and other OS’s as well, making it perfect for monitoring multi platform environments).

Lets get rolling shall we?

1) First off, lets add the DAG repo for yum, so that we actually have an up to date version of Munin to install:
• using your favorite text editor, create the file: /etc/yum.repos.d/dag.repo
• inside this file, add the following:

[dag]
name=Dag RPM Repository for Red Hat Enterprise Linux
baseurl=http://apt.sw.be/redhat/el$releasever/en/$basearch/dag
gpgcheck=1
gpgkey=http://dag.wieers.com/rpm/packages/RPM-GPG-KEY.dag.txt
enabled=1

2) Now that we have a repository containing munin, the install is simple. You can use yum to install it via the following command:

yum install munin munin-node

We need to install two parts for Munin to work properly. Munin it’s self handles the parsing of system info and generation of the pretty graphs. Munin-Node does the actual system monitoring. If you plan to install Munin on multiple systems and want to centralize all the graphing, you can skip the munin install and just put munin-node on the remote systems.

3) Now that it’s installed, you may wish to add/remove plugins to gather different information. Munin by default includes a directory of plugins in /usr/share/munin/plugins/.  To use any of these they are simply sym-linked to the munin plugin directory (which by default has a large number of these enabled.. Make sure you check out /etc/munin/plugins first to see if what you want is already there). For example, to add the apache monitoring plugins, you’d simply run the following command.

ln -s /usr/share/munin/plugins/apache_* /etc/munin/plugins

Note that some plugins may require special setup. For example the MySQL plugin requires access to the MySQL server, and the Apache plugins require system-status to be turned on in apache. There is plenty of good information on how to enable these properly by searching in google.

4) Now that you’ve installed Munin and your desired plugins, you need to start the munin-node to begin collecting data:

/etc/init.d/munin-node start

and you need to restart apache, to pickup the changes made there to display the graphs:

/etc/init.d/httpd restart

5) And you’re done! You’re stats should be visible at http://www.yoursite.com/munin/ keeping in mind that munin only adds data points to the graph every 5 minutes, so you may want to go get a coffee and come back before looking at your graphs. There are numerous configuration options available for munin, which again can be found on Google if needed.

{ 0 comments }