Stop Guessing: Deep Server Log Analysis with AWStats on Linux

The Myth of "JavaScript Analytics"

If you are relying solely on that little snippet of JavaScript from Mountain View to tell you what is happening on your server, you are flying blind. I learned this the hard way last week. A client's e-commerce site on a generic shared host slowed to a crawl. The analytics dashboard showed normal traffic. The server load average? It was spiking past 20.0.

JavaScript tags don't load when a script blocker is active. More importantly, they don't trigger for the thousands of bots, scrapers, and image hotlinkers hammering your Apache processes. To see the truth, you have to go to the source: /var/log/httpd/access_log.

But staring at raw text logs via tail -f is a recipe for madness. You need AWStats. And you need a server architecture that can handle parsing gigabytes of text without choking.

Prerequisites: The 2010 Stack

We are going to set this up on a standard LAMP stack. I’m assuming you are running CentOS 5.5 or Ubuntu 10.04 LTS (Lucid Lynx). If you are still on RHEL 4, upgrade immediately.

• Root Access: You cannot do this properly on shared hosting.
• Perl: Installed by default on most distros.
• Apache 2.2: With the combined log format enabled.

Step 1: Installation

Don't compile from source unless you enjoy dependency hell. Use the repositories.

CentOS / RHEL:
yum install awstats

Debian / Ubuntu:
apt-get install awstats

Step 2: Configuration for High Accuracy

The default configuration is usually garbage. It misses the nuances of modern browsers and bots. Open your config file, typically located at /etc/awstats/awstats.www.yourdomain.conf.

Make sure your LogFile path matches your Apache configuration exactly:

Pro Tip: If you serve a global audience but host locally here in Norway, enable the GeoIP plugin. It requires the Geo::IP Perl module and the MaxMind database. It’s the only way to prove to your boss that the "traffic spike" is actually a botnet from overseas and not potential customers.

Step 3: The I/O Bottleneck (and Why Hardware Matters)

Here is the ugly truth no one talks about. AWStats is a Perl script. It reads every single line of your log file. If you have a 2GB log file and you run the update process, your disk I/O is going to redline.

On a cheap VPS with oversold resources, this process will cause "iowait" to skyrocket. Your website will time out while the stats update. This is unacceptable.

The CoolVDS Difference: We don't oversubscribe our storage subsystems. When you run a disk-intensive task like log parsing on our platform, you get the dedicated throughput of our RAID-10 SAS arrays. We isolate I/O performance so your neighbors' bad code doesn't crash your database. Stability isn't a feature; it's a requirement.

Step 4: Automation and Cron

Don't run updates manually. Set up a cron job to run every hour. But be careful—if you run it too often on a slow server, you'll overlap processes.

Edit your crontab (crontab -e):

This keeps your stats fresh without manual intervention.

Privacy and the "Datatilsynet" Factor

Operating in Norway comes with responsibilities. The Data Protection Directive (95/46/EC) and the Norwegian Personal Data Act (Personopplysningsloven) are strict about storing Personally Identifiable Information (PII). IP addresses can be considered PII.

If you are storing raw logs for years, you might be non-compliant. Configure log rotation in /etc/logrotate.d/apache2 to compress and delete old logs after a set period (e.g., 90 days), and ensure your CoolVDS instance is secured behind a proper firewall (iptables) to prevent unauthorized access to these user data logs.

The Verdict

Google Analytics is for marketing. AWStats is for engineers. You need both.

Understanding your server logs allows you to block hotlinkers with .htaccess, identify 404 errors that bleed SEO value, and tune your MaxClients settings in Apache.

But remember: analysis requires compute power. Don't let your monitoring tools be the reason your site goes down. If your current host trembles when you run a Perl script, it’s time to move.

Need a sandbox to test your configs? Deploy a CoolVDS instance in Oslo. Low latency, high stability, and zero noisy neighbors. Start your trial today.