So here we go, we have Xymon, Ntop and Plone already installed.
We’re left with Splunk, syslog-ng and a small surprise afterwords.

Splunk is not really an open source software, but it’s definitely the best there is so far. It is free though, I mean, if you have less than 500MB of indexed data a day. Some will say it’s way more than they need, but then again some will say they have 10 times more than that on a least busiest day.
I do know that if you want to get a license - it will cost a lot of money. I think they even removed the prices from their website - not to scare people off:) To download the software you’ll have to register - no big deal.

Download it from here. Note, this line when you choose your version: 2.4+ kernel Linux distributions with NPTL / x86 2.6+ kernel Linux distributions / x86.
It’s not very clear, but if you need x86 - this is the one you want, don’t be fooled by the 2.4 kernel at the beginning of the line. I was - downloaded the 64bit version instead and set for 15 minutes trying to figure out why it’s not working.

Anyway, let me explain what it’s all about. You have servers. Servers generate logs. Apache, IIS, Squids, whatever. All kinds of logs. You need them in one place and you want to analyze them. Here’s what you do.
You need servers to send the logs to a centralized location and later have them analyzed and brought to a human readable condition.

So server-wise we’ll install syslog-ng and configure it. Its function will be to accept all the logs the clients(your servers) send and to categorize them. Usually it means making directories for each server, creating a year/month/day hierarchy and placing the logs there.

Client-wise it’s easier with Linux, since it already comes with syslog and we only need to alter it a bit. And a bit more complicated with Windows, because it needs a third-party software.
I use Snare - Epilog agent (open source), which you can download from here and setup on your Windows server. The important parts are port (514), directory to monitor(your log directory) and IP of the syslog-ng server(don’t forget port forwarding if you need).
On Linux server edit /etc/syslog.conf and add

*.*     @IPofTheSyslogServer

This will send the syslog to the syslog-ng server. If you have squids you should have it write the logs to the syslog file. Like this:

access_log syslog:user.debug squid

OK, now pay attention: do NOT install syslog-ng from Ubuntu Hardy repositories, at least at this stage. There is a nasty bug in a version 2.09. It will not create the directories and sort the log the way we need it to. So enable “Universe” karmic repositories and install the 2.0.9-4.1 version. Also it started crying about the mysql.pipe file not being available. This is done by creating the pipe:

sudo mkfifo /var/log/mysql.pipe

Another thing. Basically, most of the guides out there are for combining syslog-ng and phpsyslog, which is an open source alternative to Splunk (not the same league though). So, while going by those guides, I ended up creating a new mysql database and activating the pipe, but I don’t think it’s needed at all for our purposes, since we’re not gonna use this feature. So you should try just creating the pipe and see what happens, and only if it won’t work - go on with creating the database and continue with one of the guides on the net: there are hundreds, just look up syslog-ng and phpsyslog.

Now we actually need to edit the syslog-ng.conf:

sudo nano /etc/syslog-ng/syslog-ng.conf

Here I have a bit of a problem with telling you exactly what to alter in this file, since I just copied my old one and the new one is from a newer version, so I’m not sure which part I actually added back then and which were there by default. So I’ll just paste whatever differences I see between them:

Find

#####
#destinations

and above, before

};

udp(
                ip(0.0.0.0)
                port(514)
        );
        tcp(
                ip(0.0.0.0)
                port(514)
        );

Find

######
# filters

right above enter:

destination d_mysql {
pipe("/var/log/mysql.pipe"
template("INSERT INTO logs
(host, facility, priority, level, tag, datetime, program, msg)
VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC',
'$PROGRAM', '$MSG' );\n") template-escape(yes));
};
# Log Server destination
destination logs {
# Location of the log files using syslog-ng internal variables
file("/var/log/$HOST/$YEAR/$MONTH/$FACILITY.$YEAR-$MONTH-$DAY"
# Log files owned by root, group is adm and permissions of 665
owner(nobody) group(nobody) perm(775)
# Create the directories if they don't exist with 775 perms
create_dirs(yes) dir_perm(0775));
};


and right under:

# Anything that's from the program 'squid'
# and the 'user' log facility
filter f_squid { program("squid") and facility(user); };
# This is our squid destination log file
destination d_squid {
# The squid log file with dates
file("/var/log/$HOST/$YEAR/$MONTH/squid.$YEAR-$MONTH-$DAY"
owner(nobody) group(nobody) perm(775)
create_dirs(yes) dir_perm(0775));
};
# This is the actual Squid logging
log {
source(s_all);
filter(f_squid);
destination(d_squid);
};
# Remove the 'squid' log entries from 'user' log facility
filter f_remove { not program("squid"); };
# Log everything else less the categories removed
# by the f_remove period
log {
source(s_all);
filter(f_remove);
destination(logs);
};


by the way - those are for squids logs.

and finally, at the end of the file add:
log {
source(s_all);
destination(d_mysql);
};

That’s it. Restart syslog-ng, of course.

sudo /etc/init.d/syslog-ng restart

After that all you have to do is give some inputs to Splunk, i.e. directories to index and search/analyze your logs. Read their knowledge base if you need some help to understand what’s going on.
By the way here’s a script to put in /etc/init.d/ to start/stop/restart as you’re used to: splunk-init.
Don’t forget to do this afterwords:

sudo chmod +x /etc/init.d/splunk-init
sudo update-rc.d splunk defaults

UPD2: You don’t actually need to do it with version 4 (I have 4.07 installed now), you can run this command:

sudo /opt/splunk/bin/splunk enable boot-start -user YOURUSERNAME

This will create the script and update everything as it should.