Computing

I see exam results are back in the news again this week as GCSE results are out and people are complaining once again that exams are too easy.

Does it really matter how hard the exams are or how many kids sit, surely it's the level to which they are educated that matters?

When I was in school it was my opinion that 90%+ of my time was wasted, looking back I'm fairly confident the real figure is higher, and as for kids today ...

Over the last 15 years I reckon I've interviewed over 1000 graduates and people of graduate age, these are people who've come through school, A-levels and some University or collages of higher education.

I spent much of this time being embarrassed for the candidate!

The people who "make it" out of the educational system with a good grounding, and there are some, tend to be people who've given up on the establishment (consciously or otherwise) and made an effort to educate themselves to their own chosen level.

Having been beaten down time and again by people who thought the educational system was wonderful, a couple of years ago I decided to see if I could find out what was going on, so I sighed up for a teacher training course. It's a fairly serious course, 9 months full-time, partly in a University and partly in a school.

To be a teacher -"Did you know"

• That to be a GCSE teacher in a particular subject, the only qualification you need in that subject is an 'A' level.
• You do not need what I would call a "real" degree to become a teacher, Open University degrees and what used to be "higher education" certificates from collages of HE (which are now also called "Degrees") can also be used.
• One other student teacher on the course, when asked "why" quoted;
  because I'm going in late (mid 40's) I'll get pension enhancement, so when I retire I'll get nearly a full pension. not only that, once you're in it's almost impossible to get rid of you so it's a job for life, no redundancies!
• The people who teach people to be teachers in a given subject, don't necessarily have a qualification in the subject you are studying to teach, so there's no way they can assess your technical knowledge or ability. Mine found it hard enough to operate an electronic white board, pretty poor considering we were studying to teach ICT!

So overall, we're not really helping!

Too easy?

So why is it so few kids seem to know what they're doing when they leave school, and so many foreign kids go off to become doctors, and skilled professions ... to the extent we have shortages in this country and have to import them by the boatload?

Maybe kids these days have it too easy?

We used to have to fight for local part time jobs when I was a kid, paper rounds were very sought after. Today, we can't actually get a paper delivered, let alone milk.

Dog walking, car washing .. forget it.

We had no money, so had to get off our rear ends and earn it. It does seem that today money is not in short supply (generally) in which case maybe the incentive to learn something and get to be good at it, really isn't there anymore.

(I'm not counting video games here folks, I know we have many young champions in our midst)

If this is true, the signs are that as a country, within the next few generations we will be taken over by people from nations we consider to be 'poorer', who have are driven to succeed and better themselves - while we simply wallow in our own success and vanish into obscurity.

Hi Hooooooo!

Well, there are other contributing factors, other people can see all the problems we're having and are sick to the back teeth of it all.

On the news yesterday, over the last year around 200,000 skilled/educated Britians emigrated, never to return to these shores.

Apparently not a problem as we had about 600,000 unskilled / semi-skilled immigrants to replace them.

So, intelligent people looking for a future can see the system is broken and are buggering off in ever increasing numbers.

I wonder if there will actually be anyone left with an IQ to effect a revolution when the time comes?!

I've always been a little wary of offers of Internet access via mobile phone, performance always seems to be a little dubious and the bill is something you'd rather see on TV as a component of our national debt.

However, after seeing Vodafone's recent "3G Broadband" offer the temptation was just too much, so off went the order. It did take a little while to arrive (2 weeks!) and it did come with a scary manual that pretty much said "don't even think about plugging it in until you've installed the Windows software". The bit about how to get the service running with a real Operating System seemed to be missing .. strange but I guess predictable.

So, manuals and CD straight into the bin, plug in the card, what's next ...

There are lots of nice "howto's" on the PHARScape.org website so I won't duplicate them here, needless to say it took a bit of fiddling, but I now have a fairly reliable broadband service running over a Vodafone wireless link using Ubuntu 7.04 Linux.

I wonder how many people are looking at the Vodafone product, which is essentially £25 per month for unlimited usage and in theory works at up to 1.4Mbps, and *not* buying it because it  doesn't look like it would run on their [linux] laptop ?! I know it's not just me because I've been asked many times for a solution in the past and have had to say "not unless you own a small country" .. can't help thinking that Vadafone are missing a trick by not paying a few part-time tekkies to answer support queries on a forum .. hell Ubuntu alone reckon they have 18 million users out there. There must be a reasonable percentage with laptops saying "if only I could be sure I could get it to work".

Installation and Performance

I had to tweak a few things in order for the machine to see the card (these were relatively trivial) and after that you need to set up a PPP session in /etc/ppp/peers and an associated chat script in /etc/chatscripts. If you pick up the examples from the PHARScape.org site, these are a pretty good starting point.

Once you've got a connection you'll find that Vodafone kindly omit to send out information about DNS servers that you can use. As a result you'll have a working PPP connection, but no dns resolution.

There is a fairly easy solution to this (albeit not a 'proper' solution) which is to use opendns, so right click on your network icon in the tool tray and manually set the dns servers to;

208.67.222.222
208.67.220.220

Then make sure your PPP config file in /etc/ppp/peers does NOT have usepeerdns in it. This solution seems to work well for me and flies quite happily via whichever network interface I seem to have connected at any given point in time.

In terms of speed it looks like I'm right on the edge of the network, indeed I don't get a great Orange signal and the strength indicator on the card reads 9,99 , which I'm guessing is only one bar from a possible 10 .. so a very weak signal.

Running speedtests from Speedtest.Net, I can get mixed results depending on which host I measure against. Typically I'm getting 350/360 kbps, which I guess is a reasonable GPRS connection. I've yet to see the fames 1.4Mbps, but then I guess I'll have to wait until I can use the laptop somewhere with better coverage.

For what it's worth, NX is fairly usable over GPRS.

I've noticed that occasionally my connection light goes red and everything stops, I don't know if this is the card completely losing signal or dopping back to from GRPS to 64k. I guess I'll get a better idea once I've traveled around with it a little.

So, come on Vodafone, what about a little support, it might even make you some more money! (and if anyone knows the answer to the missing DNS servers mystery ... )

We were recently asked to provide a CCTV facility for an existing customer who needed something setting up in a hurry. So after seeing many cameras above and previously fiddling with an Axis Camers, we had a crack an Azis PTZ running with "Zoneminder", which looked like a really cool open source project.

After (finally) making it do something moderately useful, it became apparent that although possibly useful under certain circumstances it really wasn't going to make a great CCTV system. So being a bit of a sucker, I set about writing one from scratch.

Guess what ...

It works!

We're running 1 daemon for each camera, this streams information from the camera and stores it in a local MySQL database at a given frequency, for example we're currently using 5 frames a second. This is using about 2% of the CPU on the local Linux server.

Then we run another optional daemon for each camera which logs begin/end events from the camera according to it's motion sensor, which is probably a little redundant in the daytime, but you never know - the functionality is there.

Then we get the fun part, a GTK application running on your local workstation allows you to configure and connect to a number of remote databases (which are being logged to by the camera daemons) and watch play, revese play etc etc stored pictures from the camera.

After a little experimentation I've opted to embed camera thumbnails (320x240) on the desktop and provide a full-size snapshot or recording on demand. This method means you can run as many cameras as you like and hide / unhide them from the gadget in the tooltray.

The tool-tray also shows you the current streaming bandwidth in use so you've half a chance of not saturating cameras located on remote sites. Overall, enough to get a fairly substantial CCTV configuration set up, and as soon as GNOME/X11 allow applications to migrate between virtual screens we should be able to run a panel of camera views across multiple screens .. although I might add this functionality sooner myself as they've been dragging their heels on this for years.

The Technology ...

Sounds like it might have been difficult, however end-to-end it was probably only a week's coding in between other stuff. The key to this is a system called GAMBAS. At first sight the web page can be a little off-putting as it doesn't look "that" professional, however once you get into the application itself, you have a Visual basic variant for Linux which is pretty damn solid and certainly something which has the potential to rival Borland Delphi.

When it comes to it, application development in GAMBAS is generally faster than in Delphi (or Kylix) , it's just missing some of the finer points with regards to component extensions.

Anyway, with any luck there should be some GAMBAS .deb's flying around in the not-too-distant future, if anyone's interested that is. If not, it should make a good demo ..

A few weeks ago we noticed our traffic spike by about x100 as a BOTNET focussed it's attention on our main server array. This was enough to cause serious discomfort to our server's ability to render web pages, and to our ability to pay for our transit bill (!)

So, how does one defeat a BOTNET ?

Well I guess the bottom line is that it depends on the BOTNET size, but we've currently collected around 25,000 offending IP's so our method seems to work on a relatively small scale.

The first thing you need is a nice interface to the system's IPTABLES ...

I've written a bit of python code called 'blacklist' (see below) which provides a command line interface to iptables and allows you to easily block or unblock one or more IP addresses. The beauty of the system is that it sticks the results in a MySQL database, hence you can use the results on any machine on your network.

# blacklist info - show stats about collected addressed
# blacklist status - dump the contents of the blacklist
# blacklist block <cidr> <duration> - block an address range for a period
# blacklist unblock <cidr> - remove the address range from the blacklist
# blacklist start - call on system boot, load the list to the kernel
# blacklist stop - call to remove all entries from iptables

Making the list

The next trick is to actually generate the list. For this we have sql_scanner.py (below) which will try to read through all apache log files in /var/log/apache2/* and identify each hit on the server. Setting this up (and maintaining it) can be a little laborious to get going, but it's well worth it if it stops abusers from bringing you down.

The code tries to identify each entry based on client IP, URL etc and if possible, mark the entry as 'dealt with', either updating the white or black list in the process. If it finds an entry it can't automatically cope with, it will ask you. You then have the chance to black or white list the entry either by referrer, ip address or URI.

This information is all stored in the MySQL database and fed down through your IPTABLES as necessary. Once you've run through logs on one machine, the information is then available to other machines on your network, so long as they're all set up with a common SQL database.

Subsequently running sql_scanner.py on another machine will run through it's log files and automatically add any matching IP's it's the IPTABLES, or alternatively stopping and starting blackhole will reload the global ip blacklist from the database.

Anyway, it seems to be pretty effective, here's what the implementation did for us;

Database Structure

/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
CREATE DATABASE /*!32312 IF NOT EXISTS*/ `blackpages` /*!40100 DEFAULT CHARACTER SET latin1 */;
USE `blackpages`;
DROP TABLE IF EXISTS `ips`;
CREATE TABLE `ips` (
  `ip` char(20) NOT NULL,
  `server` char(20) character set utf8 NOT NULL,
  `scope` enum('local','global') NOT NULL default 'global',
  `colour` enum('black','white') NOT NULL default 'black',
  `start` bigint(20) unsigned zerofill NOT NULL,
  `duration` bigint(20) unsigned zerofill NOT NULL,
  PRIMARY KEY  (`ip`),
  KEY `ips_index01` (`server`),
  KEY `ips_index02` (`colour`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
DROP TABLE IF EXISTS `referrers`;
CREATE TABLE `referrers` (
  `referrer` char(128) character set utf8 NOT NULL,
  `server` char(20) character set utf8 NOT NULL,
  `scope` enum('local','global') NOT NULL default 'global',
  `colour` enum('black','white') NOT NULL default 'black',
  `start` bigint(11) default NULL,
  `duration` bigint(20) default NULL,
  PRIMARY KEY  (`referrer`),
  KEY `referrers_index01` (`server`),
  KEY `referrers_index02` (`colour`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
DROP TABLE IF EXISTS `uris`;
CREATE TABLE `uris` (
  `uri` char(255) NOT NULL,
  `server` char(20) character set utf8 NOT NULL,
  `scope` enum('local','global') NOT NULL default 'global',
  `colour` enum('black','white') NOT NULL default 'black',
  `start` bigint(20) default NULL,
  `duration` bigint(20) default NULL,
  PRIMARY KEY  (`uri`),
  KEY `uris_index01` (`server`),
  KEY `uris_index02` (`colour`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;
/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;

Blacklist

This is the source code for the blacklist module, store it in /sbin. Don't forget to change the database settings to match your environment!

#!/usr/bin/python

import sys
import time
import os
import MySQLdb
import MySQLdb.cursors
#
##############################################################################
#
#       connect to the database
#
server = MySQLdb.connect(host="mysql",user="blacklist",passwd="blackpass1234",db="blackpages",cursorclass=MySQLdb.cursors.DictCursor)
cursor = server.cursor()
#
##############################################################################
#
#       read the hostname (better python method?)
#
wc=os.popen("hostname")
hostname=wc.readline().split("\n")[0]
wc.close()
#
#
##############################################################################
#
data={}

def do_status():
        """ print a list of blocked ips """
        now = time.time()
        print "%-20s\t%-43s\t%8s\t%8s" % ("ip", "blocking since", "duration", "remains")

        cursor.execute("SELECT ip,server,scope,colour,start,duration FROM ips ORDER BY ip")
        rows = cursor.fetchall()

        for row in rows:
                start_time = time.strftime("%a %b %d %H:%M:%S %Y",time.localtime(row['start']))
                time_left = row['start']+row['duration']-now
                print "%-20s\t%-43s\t%8d\t%8d" % (row['ip'],start_time,row['duration'],time_left)

def do_info():
        """ print out sql info """
        cursor.execute("SELECT colour,COUNT(ip) as count FROM ips GROUP BY colour")
        rows = cursor.fetchall()

        for row in rows:
                print "IP       Database [%s] count is %s" % (row['colour'],row['count'])

        cursor.execute("SELECT colour,COUNT(referrer) as count FROM referrers GROUP BY colour")
        rows = cursor.fetchall()

        for row in rows:
                print "Referrer Database [%s] count is %s" % (row['colour'],row['count'])

        cursor.execute("SELECT colour,COUNT(uri) as count FROM uris GROUP BY colour")
        rows = cursor.fetchall()

        for row in rows:
                print "URI      Database [%s] count is %s" % (row['colour'],row['count'])

def do_block(cidr,duration,quiet):
        """ block off an ip or ip range """
        cursor.execute("SELECT ip FROM ips WHERE ip = '%s'" % cidr)
        rows = cursor.fetchall()
        if len(rows):
                print "IP address (%s) is already blocked" % cidr
                return

        print "Block (%s) for (%s)" % (cidr,duration)
        stdin,stdout = os.popen4("/sbin/iptables -I BLACKLIST -s %s -j DROP" % cidr)
        line = stdout.readline()
        stdin.close()
        stdout.close()

        if len(line):
                print "Bad IP address: (%s)" % line
                return

        try:
                duration = long(duration)
        except:
                print "Bad duration (%s)" % duration
                return

        cursor.execute("BEGIN")
        cursor.fetchall()
        cursor.execute("INSERT INTO ips (ip,server,scope,colour,start,duration) VALUES ('%s','%s','global','black','%s','%s')" % (cidr,hostname,time.time(),duration))
        cursor.fetchall()
        cursor.execute("COMMIT")
        cursor.fetchall()
        if quiet: sys.exit(0)

def do_unblock(cidr):
        """ unblock an ip or ip range """
        cursor.execute("SELECT ip FROM ips WHERE ip = '%s'" % cidr)
        rows = cursor.fetchall()
        if not len(rows):
                print "IP address (%s) is *NOT* blocked" % cidr
                return

        print "Unblock (%s)" % (cidr)
        stdin,stdout = os.popen4("/sbin/iptables -D BLACKLIST -s %s -j DROP" % cidr)
        line = stdout.readline()
        stdin.close()
        stdout.close()

        if len(line):
                print "Bad IP address: (%s)" % line
                return

        cursor.execute("BEGIN")
        cursor.fetchall()
        cursor.execute("DELETE FROM ips WHERE ip = '%s'" % cidr)
        cursor.fetchall()
        cursor.execute("COMMIT")
        cursor.fetchall()

def do_clear():
        """ clear the table """
        os.system("/sbin/iptables -F BLACKLIST")
        os.system("/sbin/iptables -A BLACKLIST -j RETURN")

def blockip(ip):
        """ block an ip address """
        stdin,stdout = os.popen4("/sbin/iptables -I BLACKLIST -s %s -j DROP" % ip)
        line = stdout.readline()
        stdin.close()
        stdout.close()
        return line

def do_start():
        """ load the table from the database """
        do_clear()
        cursor.execute("SELECT ip FROM ips WHERE colour = 'black'")
        rows = cursor.fetchall()
        count=1
        max = len(rows)
        for row in rows:
                print ("Loading %d/%d\r" % (count,max)),
                sys.stdout.flush()
                count += 1
                line = blockip(row['ip'])
                if len(line):
                        time.sleep(1)
                        line = blockip(row['ip'])

                if len(line): print "Bad table entry (%s): %s" % (row['ip'],line)

        print "\nFirewall reloaded (%s) entries." % len(rows)


if len(sys.argv)<2:
        print "Usage: blacklist status | info | start | stop | block <ip> <duration> | unblock <ip>"
        sys.exit(1)

def heading():
        """ print the heading """
        print ""
        print "Looking up BlackPages on host ** %s **" % hostname
        print ""

if sys.argv[1] == "status":
        heading()
        do_status()
elif sys.argv[1] == "info":
        heading()
        do_info()
elif sys.argv[1] == "start":
        heading()
        do_start()
elif sys.argv[1] == "stop":
        heading()
        do_clear()
elif sys.argv[1] == "unblock":
        if len(sys.argv)<3:
                print "Usage: blacklist unblock <cidr>"
        else:
                heading()
                do_unblock(sys.argv[2])

elif sys.argv[1].lower() == "block":
        if len(sys.argv)<4:
                print "Usage: blacklist block <cidr> <duration>"
        else:
                if sys.argv[1] == "block": 
                        heading()
                        quiet = False
                else:
                        quiet = True
                do_block(sys.argv[2],sys.argv[3],quiet)
else:
        print "No such command (%s)" % sys.argv[1]

print ""
sys.exit(0)

sql_scanner.py

Finally here's the code to scan through your apache logs, bear in mind that you mush use the combined log format otherwise the URL parser won't work. Also note that you may need to install additional python modules, see the first few lines of code for import statements.

#!/usr/bin/python 
import string
import sys
import os
import time
import pyparsing
import MySQLdb
import MySQLdb.cursors
#
print ""
##############################################################################
#
#       Global Variables
#
progress        = 0
last_printed    = 0
exceptions      = []
#
##############################################################################
#
#       read the hostname (better python method?)
#
wc=os.popen("hostname")
hostname=wc.readline().split("\n")[0]
wc.close()
#
##############################################################################
#
#       connect to the database
#
server = MySQLdb.connect(host="mysql",user="blacklist",passwd="blackpass1234",db="blackpages",cursorclass=MySQLdb.cursors.DictCursor)
cursor = server.cursor()
#
##############################################################################
#
#       handle the optional [--batch] command line argument
#
batch=False
if len(sys.argv)>1 and (sys.argv[1] == "--batch"): batch = True
#
##############################################################################
#
#       getCmdFields - used to split a request to components
#
def getCmdFields( s, l, t ):
        splits = t[0].strip('"').split()
        if len(splits)>0: t["method"] = splits[0] 
        else: t["method"]="GET"
        if len(splits)>1: t["requestURI"] = splits[1] 
        else: t["requestURI"] = "unknown.com"
        if len(splits)>2: t["protocolVersion"] = splits[2] 
        else: t["protocolVersion"] = "HTTP/1.0"
#
##############################################################################
#
#       set up the pyparser
#
integer = pyparsing.Word( pyparsing.nums )
ipAddress = pyparsing.delimitedList( integer, ".", combine=True )
timeZoneOffset = pyparsing.Word("+-",pyparsing.nums)
month = pyparsing.Word(string.uppercase, string.lowercase, exact=3)
serverDateTime = pyparsing.Group( pyparsing.Suppress("[") + 
                        pyparsing.Combine( integer + "/" + month + "/" + integer + ":" + integer + ":" + integer + ":" + integer ) + 
                        timeZoneOffset + pyparsing.Suppress("]") ) 

logLineBNF =    ( ipAddress.setResultsName("ipAddr") + pyparsing.Suppress("-") + 
                ("-" | pyparsing.Word( pyparsing.alphas+pyparsing.nums+"@._" )).setResultsName("auth") +
                serverDateTime.setResultsName("timestamp") +
                pyparsing.dblQuotedString.setResultsName("cmd").setParseAction(getCmdFields) +
                (integer | "-").setResultsName("statusCode") +
                (integer | "-").setResultsName("numBytesSent")  +
                pyparsing.dblQuotedString.setResultsName("referrer").setParseAction(pyparsing.removeQuotes) +
                pyparsing.dblQuotedString.setResultsName("clientSfw").setParseAction(pyparsing.removeQuotes) )
#
##############################################################################
#
#       load - read a batch 
#
def load(sql,key):
        """load a list from a table"""
        list=[]
        cursor.execute(sql)
        rows = cursor.fetchall()
        for row in rows:
                list.append(row[key])
        return list

#
##############################################################################
#
#       save - write a batch 
#
def save(table,field,colour,list):
        """save a list back to a table"""
        if not len(list): return
        print "Updating table (%s) with (%s) entries of colour (%s)" % (table,len(list),colour)

        if (table=="ips") and (colour=="black"):
                for x in list:
                        os.system("/sbin/blacklist BLOCK %s 999999999" % x)
        else:
                cursor.execute("BEGIN")
                cursor.fetchall()
                for x in list:
                        sql=("INSERT INTO %s (%s,server,scope,colour,start,duration) VALUES ('%s','"+hostname+"','global','%s','%s','999999999')") % (table,field,x,colour,time.time())
                        cursor.execute(sql)
                        cursor.fetchall()
                cursor.execute("COMMIT")
                cursor.fetchall()
#
##############################################################################
#
#       preload information from the database
#
ip_blacklist    = load("SELECT ip FROM ips WHERE colour = 'black'","ip")
ip_whitelist    = load("SELECT ip FROM ips WHERE colour = 'white'","ip")
ref_blacklist   = load("SELECT referrer FROM referrers WHERE colour = 'black'","referrer")
ref_whitelist   = load("SELECT referrer FROM referrers WHERE colour = 'white'","referrer")
uri_blacklist   = load("SELECT uri FROM uris WHERE colour = 'black'","uri")
uri_whitelist   = load("SELECT uri FROM uris WHERE colour = 'white'","uri")

ip_blacklist_local    = []
ip_whitelist_local    = []
ref_blacklist_local   = []
ref_whitelist_local   = []
uri_blacklist_local   = []
uri_whitelist_local   = []
#
##############################################################################
#
#       merge the blacklist data
#
#io = open("/etc/blacklist.dat")
#while True:
#       line = io.readline()
#       if not line: break
#       line = line.split("\n")[0].split(" ")[0]
#       if not line in ip_blacklist: ip_blacklist_local.append(line)
#
#io.close()
#print "[Merging %d entries from /etc/blacklist.dat]" % len(ip_blacklist_local)
#
##############################################################################
#
#       print_progress - track our process through the log file
#
def print_progress(p,l):
        global last_printed
        global batch

        if batch: return

        if progress>(last_printed+99):
                print "[%d of %d]\r" % (p,l),
                last_printed = progress
                stdout.flush()
        return
#
##############################################################################
#
#       scan - main routine to itterate through a log file
#
def scan(filename):
        """scan a file for log entries"""
        filename="/var/log/apache2/%s" % filename
        wc=os.popen('wc -l %s' % filename,"r")
        lines=int(wc.readline().split(" ")[0])
        wc.close()

        print "Scanning '%s' for %d lines" % (filename,lines)
        logfile = open(filename,"r")

        progress=0
        while True:
                line = logfile.readline().split("\n")[0]
                progress = progress + 1
                print_progress(progress,lines)
                if not line: break

                if line[:3] == "::1": continue

                try:
                        fields = logLineBNF.parseString(line)
                except:
                        print "** Exception: "+line

                ip = fields['ipAddr']
                if (ip in ip_whitelist) or (ip in ip_whitelist_local): continue
                if (ip in ip_blacklist) or (ip in ip_blacklist_local): continue

                referrer = ""
                if 'referrer' in fields: referrer = fields['referrer'][:255]

                #print "Full:",referrer
                ref=referrer
                if referrer[:7] == "http://": ref=ref[7:]
                ref=ref.split("/")[0]
                #print "Short:",ref

                #if len(referrer):
                #       if (referrer in ref_whitelist) or (referrer in ref_whitelist_local): continue
                #       if (referrer in ref_blacklist) or (referrer in ref_blacklist_local): continue

                uri = fields['requestURI']

                if (uri[:7]=="http://") or (uri[:8]=="https://") or (fields['method']=="CONNECT"):
                        ip_blacklist_local.append(ip)
                        continue

                if (uri in uri_whitelist) or (uri in uri_whitelist_local):
                        ip_whitelist_local.append(ip)
                        continue

                if (uri in uri_blacklist) or (uri in uri_blacklist_local):
                        ip_blacklist_local.append(ip)
                        continue

                if (ref in ref_whitelist) or (ref in ref_whitelist_local):
                        ip_whitelist_local.append(ip)
                        continue

                if (ref in ref_blacklist) or (ref in ref_blacklist_local):
                        ip_blacklist_local.append(ip)
                        continue

                if batch: continue

                #if len(referrer)>1:
                #       default="Blacklist Referrer"
                #elif len(ip)>1:
                #       default="Blacklist URI"
                #else:
                #       default="Blacklist IP"

                print "+---------------------------------------------------------------------+"
                print "| * * * * Manual clarification required for unknown log entry * * * * |"
                print "|                                                                     |"
                if len(referrer): 
                        print "| Referrer ..... %-52s |" % referrer
                        print "| Ref Domain ... %-52s |" % ref
                print "| Address ...... %-52s |" % ip
                print "| URI .......... %-52s |" % uri[:55]
                print "+---------------------------------------------------------------------+"

                while True:
                        print "[q]=Quit  [b]=BlackList [w]=WhiteList [return]=WhitelistIP (q,b,w,return) > ",
                        choice = sys.stdin.readline().split("\n")[0].lower()

                        if choice == 'q': break
                        if choice == '': 
                                ip_whitelist_local.append(ip)
                                break

                        #       if default == "Blacklist Referrer":     ref_blacklist_local.append(referrer)
                        #       if default == "Blacklist URI":          uri_blacklist_local.append(uri)

                        if (choice <> 'w') and (choice <> 'b'): continue

                        while True:
                                print "[r]=By Referrer  [i]=By IP  [u]=By URI > ",
                                action = sys.stdin.readline().split("\n")[0].lower()

                                if action == "r":
                                        if choice == "w":
                                                ref_whitelist_local.append(ref)
                                        else:
                                                ref_blacklist_local.append(ref)
                                        break

                                if action == "i":
                                        if choice == "w":
                                                ip_whitelist_local.append(ip)
                                        else:
                                                ip_blacklist_local.append(ip)
                                        break

                                if action == "u":
                                        if choice == "w":
                                                uri_whitelist_local.append(uri)
                                        else:
                                                uri_blacklist_local.append(uri)
                                        break

                        break

        logfile.close()
#
##############################################################################
#
#       main routine
#
print ("Scanning Log files on host ** %s **" % hostname),
if batch:
        print " (batch mode)"
else:   print ""

for f in os.listdir("/var/log/apache2/"):
        if f[-4:]==".log":
                if f[-9:]=="error.log": continue
                scan(f)
#
print ""
#
#       save all our changes
#
save("ips"      ,"ip"           ,"black",ip_blacklist_local)
save("ips"      ,"ip"           ,"white",ip_whitelist_local)
save("referrers","referrer"     ,"black",ref_blacklist_local)
save("referrers","referrer"     ,"white",ref_whitelist_local)
save("uris"     ,"uri"          ,"black",uri_blacklist_local)
save("uris"     ,"uri"          ,"white",uri_whitelist_local)
#
#       Quit here!
#
print ""

Have Fun!

Ok, got to get (apparently) a plug in for our hosting provider. Encryptec Limited host these Blogs and they're pushing their plone hosting product.

As hosting goes, it seems to work Ok - not had any problems to date .. in terms of speed / reliability, this is the absolute bottom of the line / bargain basement product, and I've not had problems.

Anyway, if you want a cracking dynamic website for a bargain price, support the little guy!
plone hosting!

Also, a couple of up and coming sites to watch;

• A Week in Tuscany
• Tuscan Villa Rental
  

Ok, this is a rant but I think the content is perfectly valid. I moved to Ubuntu from Gentoo generally because I was leaning towards binary distro's after having problems with Gentoo's political system.

Ubuntu is great, but unfortunately after making itself "so" good in certain areas, when I find it's falling over badly in other areas, expectations tend to be dashed.

In particular I'm talking about the fringes that Gentoo supports so well, and Ubuntu screws up so badly ...

Let's talk about Ubuntu server and name services, BIND is and always has been the center of any hosting setup. DLZ has for many years been a standard add-on package that has now finally been included in the standard BIND source tree.

Indeed, if you install the latest Ubuntu package and run it against your BIND/DLZ/MySQL setup, it generates NO ERRORS, however when you use it, IT DOESN'T WORK!

It doesn't log any failures, configuration errors, nothing ... just sits there silently with no clue as to where to start looking. Just to reiterate, they thought DLZ was so important, it's now part of BIND itself, and Ubuntu seem to have half implemented it such that I've just spent the last 3 hours trying to make it work - and failed.

My Solution;

a. Download the latest source
b. Unpack
c. ./configure--prefix=/usr --with-dlz-mysql=/usr --sysconfdir=/etc/bind \
                            --localstatedir=/var/run/bind
d. make install
e. /etc/init.d/bind restart

Hey presto, problem solved.

Exactly how did Ubuntu compile this and screw it up???

Incidentally, this is on Gutsy / AMD64.