I just encountered a small tool Microsoft released in 2005 called Log Parser 2.2. This is a tool that allows you to take literally any data format anywhere and turn it into anything, very handy - particularly when processing IIS logfiles.
The site that used to exist for it has vanished, but there's still a few good resources out there related to it:-
Alexis Eller's talk on using log parser (I couldn't find a blog for her unfortunately)
Professor Windows - May 2005 article which has a lot of good examples.
Update - I am trying to build a replacement for tools like Webalizer which aren't maintained anymore (or are insanely complex to install and configure and maintain strange files as 'summaries'). I am currently using the following command to create and maintain a multiple GB database containing an indexed set of logfiles and getting back graphs of the data instantly for display in Open Flash Chart.
The command I'm using is:
"C:\Program Files\Log Parser 2.2\logparser" "select 1, * into WebLog from <MyIISSiteNumber>" -i:IISW3C -o:SQL -server:(local) -database:NullifyDBLog -driver:"SQL Server" -createTable:ON -transactionRowCount 100 -iCheckpoint:CheckpointFile.lpc
Which is working really well, it took about 30 minutes to process 4 or 5 years of logs and now updates the database near instantly!
Watch out for the fact that LogParser isn't too bright when making tables, to do this with my log format you need to create extra indexes, I opted for indexing obvious things like date, time, the uri, the referrer, the bytes transferred and the user agent.Permalink