The aim of the Fast Logging Project is to decouple the output to a database
from the real snort process.

snort has to process the network packets to find alert signatures.
If an alert is found the result has to be stored somewhere to process
or simply store it.

The problem with the normal database output plugin is that snort
is blocked until all data is written to the database. If there are
any problems with the database then snort is hunging and probably
some network packets get lost.

The FLoP project decouples this output and processing from snort.
Furtheron all alerts are spooled to a central server where they
are feed in a database via unix sockets which is much faster than
TCP/IP via any physical network.

Since snort will write to an unix domain socket there is no blocking
possible. (Of course you can loose some alerts if no process reads 
these alerts. But this is another story...)

Addition: There is still blocking possible if the reading
process blocks. But this will never happen with sockserv
since one thread is still waiting for input.

The program sockserv generates such an unix socket and reads all
incoming alerts. The program consists of two threads, one reads
the alert and buffers them in memory. The second take the alerts
and spools them to the central server. 

Here the program servsock reads these alerts. This program forks
off a child process for each sockserv connection. These child
processes consists of two threads. One thread receives the
alerts and the second one feeds them to the database. Again a
buffering takes place to overcome some slow downs due to too
many inserts in the database.

Finally to avoid a buffer overrun in form of a SIGSEGV there
is a 'drop' feature implemented. This works with a high
and low water mark, if more then high water alerts are stored
in memory they are dropped to an unix socket until the low water 
mark is reached.

The drop program is able to receive these dropped alerts
and will send an email consisting of a short message which
alerts were dropped. (The only real thing missing is the
payload.)

Additionally there is an alert feature. This is intended for
sending email if alerts with a high priority arrive. This 
should inform an admin earlier then the periodic check of
the database.

And last but not least there is a false positive generator
to build network packest which are able to raise alerts
within snort. (Of course there exist some similar programs
but I needed one which is able to generate alerts with a
high rate to check if all pograms are able to deal with it
or will break.

With a slight extension of the database it is possible to
store the full network packet in the database. A program
called getpacket is able to rebuild a pcap file from
this data. This pcap file can then be analyzed with tools
like ethereal

Further helpful programs are in the contrib directory:

+ classification.pl: Adds the classification.config to
                     the database

+ some cgi scripts to analyze alerts via a web server

+ stats.pl: A script to gather statistics from snort

+ rules.pl: A perl script to add the signatures and
            references of the rule files to the database

README files:

README: ok, you read this now so you should know about it

README.debug: This describes how to activate the debugging
              functions of sockerv, servsock, alert, drop

README.endian: Some comments on the endianess of the sensors
               and the central server

README.payload: This describes how the full payload (more than
                only the alert level payload, e.g. TCP or UDP)
                can be stored in the database

README.snort: This explains the changes made to snort introduced 
              by the patch. Additionally there are some comments 
              on why only one snort process per sensor is allowed.

README.barnyard: This files shows some differences between
                 barnyard (or mupdit) and FLoP

contrib/README: This is a survey of the useful files in the
                contrib directory

contrib/README.rules.pl: How to add all signatures with references
                         to the database. This should speed up the
                         insert process since most rules are now
                         part of the database and need not to be
                         inserted together with the alert


If you find this software useful then send a picture postcard to:

Dirk Geschke
Karlsbader Str. 12
85435 Erding

GERMANY

(Maybe I will get at least one picture postcard?)

For any other feedback send me an email: <Dirk@geschke-online.de>
