pftop parsing

[danlynch]

Hello,
I've added a unix script to that identifies all NATed traffic across my BSD firewall (pf):

/usr/local/sbin/pftop -ab -v long -w 140 

The output is in this format:

PR    DIR SRC  DEST   GW  STATE  AGE       EXP  PKTS BYTES   AVG RU

in props.conf I have:

[pftop]
SHOULD_LINEMERGE = false
LINE_BREAKER = ^()$
TRUNCATE = 1000000
DATETIME_CONFIG = CURRENT
REPORT-pftop_fields = pftop_fields

in transforms.conf:

[pftop_fields]
REGEX = (tcp|udp) (in|out) (\d+\.\d+\.\d+\.\d+)\:?(\d*) (\d+\.\d+\.\d+\.\d+)\:?(\d*) (\d+\.\d+\.\d+\.\d+)\:?(\d*)   (\w+) (\d*) (\d*) (\d*) (\d*) (\d*) (\d*)
FORMAT = proto::$1 direction::$2 src_ip::$3 src_port::$4 dest_ip::$5 dest_port::$6 gw_ip::$7 gw_port::$8 state::$9 age::$10 expires::$11 packets::$12 bytes::$13 ave::$14 rule_number::$15
CLEAN_KEYS = 1
MV_ADD = 0

My problem is that splunk is parsing the output so each time the script runs it is seen as one event. Not only would I like to be able to see individual connects but also search on some of the key variables. netstat is experiencing the same difficulties on my system. Any help would be appreciated.

Update 2 Mar 13:
I reformated the the stdout, but I'm not sure how you would add a timestamp to each line. Below is the code for the shell script:

. `dirname $0`/common.sh
HEADER='PR     DIR    SRC                     DEST                   GW                   STATE                   AGE         EXP          PK
HEADERIZE="BEGIN {print \"$HEADER\"}"
PRINTF='{printf "%-5s  %-5s  %-21s  %-21s  %-21s %-23s %-10s  %-10s  %6s  %6s  %5s  %3s\n", $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12}'
CMD='eval /usr/local/sbin/pftop -ab -v long -w 140 | grep "tcp\|udp\|icmp"'
FORMAT='{gsub("[46]","",$1); if(!$12) { for (j=13; j>5; j--) $j=$(j-1); $5="N/A"} }'
assertHaveCommand $CMD
$CMD | tee $TEE_DEST | $AWK "$HEADERIZE $FIGURE_SECTION $FILTER $FORMAT $FILL_BLANKS $PRINTF"  header="$HEADER"
echo "Cmd = [$CMD];  | $AWK '$HEADERIZE $FIGURE_SECTION $FILTER $FORMAT $FILL_BLANKS $PRINTF' header=\"$HEADER\"" >> $TEE_DEST

Sample output:

PR  DIR SRC             DEST           GW  STATE                   AGE        EXP      PKTS BYTES AVG RU
tcp Out 127.0.0.1:14801 127.0.0.1:7736 N/A ESTABLISHED:ESTABLISHED 503:29:07  00:05:58 42K    200M  115 1

[Ayn]

Turn off line merging.

In props.conf:

SHOULD_LINEMERGE = false

[danlynch]

It is set to false but it doesn't seem to help

[Kate_Lawrence-G]

I've found the easiest away around this to reformat the stdout of the info to create an "event" with strict field assignments and a time data stamp so that when Splunk reads it it will be indexed by the time format I set in the props.conf.