ncftpd log extractions

[jspears]

I'm trying to do field extractions for ncftpd xfer logs. These are generally csv but the fields differ depending on what operation is being logged ( http://ncftpd.com/ncftpd/doc/xferlog.html if you want the details.)

Is there a clean way to extract the first few generic fields and do the other field extractions depending on the values extracted? Let's call the first field 'operation'. Can I say something like

search operation="S" OR operation="R" | do field extractions specific to these
search operation="T" | do field extractions specific to this

[hexx · Accepted solution]

You can't automate a conditional delimiter-based field extraction, but you can define one unique field extraction per type of xferlog event and select which one to apply using the extract command.

First, define an automatic field extraction for the "operation" field, the value of which should determine which delimiter-based extraction we will apply.

• props.conf:

[xferlog]
EXTRACT-operation = ^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}[^|]*\|\s+(?\w),

Now in transforms.conf, define one delimiter-based field extraction for each type of event. As an example, based on the xferlog reference page here's one definition for the S and R log entry types and one for the T (directory listing) log entry type.

• transforms.conf:

[fields_store_retrieve]
DELIMS = ","
FIELDS = "Head","Pathname","Size","Duration","Rate","User","Email","Host","Suffix","Completion","Transfer_Type","Transfer_notes","Start_of_transfer","Session_ID","Starting_size","Starting_offset"

[fields_dirlist]
DELIMS = ","
FIELDS = "Head","Pathname","Completion","Pattern","Recursion","User","Email","Host","Session ID"

Now when you search, apply the appropriate field extraction depending on the value of the "operation" field of the events you are querying :

sourcetype=xferlog operation=R OR operation=S | extract fields_store_retrieve

or:

sourcetype=xferlog operation=T | extract fields_dirlist

It's too bad that one cannot define automatic field extractions based on event types because this would have been an ideal use-case for that.