Splunk Search

ncftpd log extractions

jspears
Communicator

I'm trying to do field extractions for ncftpd xfer logs. These are generally csv but the fields differ depending on what operation is being logged ( http://ncftpd.com/ncftpd/doc/xferlog.html if you want the details.)

Is there a clean way to extract the first few generic fields and do the other field extractions depending on the values extracted? Let's call the first field 'operation'. Can I say something like

search operation="S" OR operation="R" | do field extractions specific to these
search operation="T" | do field extractions specific to this

Tags (1)
1 Solution

hexx
Splunk Employee
Splunk Employee

You can't automate a conditional delimiter-based field extraction, but you can define one unique field extraction per type of xferlog event and select which one to apply using the extract command.

First, define an automatic field extraction for the "operation" field, the value of which should determine which delimiter-based extraction we will apply.

  • props.conf:

[xferlog]
EXTRACT-operation = ^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}[^|]*\|\s+(?\w),

Now in transforms.conf, define one delimiter-based field extraction for each type of event. As an example, based on the xferlog reference page here's one definition for the S and R log entry types and one for the T (directory listing) log entry type.

  • transforms.conf:

[fields_store_retrieve]
DELIMS = ","
FIELDS = "Head","Pathname","Size","Duration","Rate","User","Email","Host","Suffix","Completion","Transfer_Type","Transfer_notes","Start_of_transfer","Session_ID","Starting_size","Starting_offset"

[fields_dirlist]
DELIMS = ","
FIELDS = "Head","Pathname","Completion","Pattern","Recursion","User","Email","Host","Session ID"

Now when you search, apply the appropriate field extraction depending on the value of the "operation" field of the events you are querying :

sourcetype=xferlog operation=R OR operation=S | extract fields_store_retrieve

or:

sourcetype=xferlog operation=T | extract fields_dirlist

It's too bad that one cannot define automatic field extractions based on event types because this would have been an ideal use-case for that.

View solution in original post

hexx
Splunk Employee
Splunk Employee

You can't automate a conditional delimiter-based field extraction, but you can define one unique field extraction per type of xferlog event and select which one to apply using the extract command.

First, define an automatic field extraction for the "operation" field, the value of which should determine which delimiter-based extraction we will apply.

  • props.conf:

[xferlog]
EXTRACT-operation = ^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}[^|]*\|\s+(?\w),

Now in transforms.conf, define one delimiter-based field extraction for each type of event. As an example, based on the xferlog reference page here's one definition for the S and R log entry types and one for the T (directory listing) log entry type.

  • transforms.conf:

[fields_store_retrieve]
DELIMS = ","
FIELDS = "Head","Pathname","Size","Duration","Rate","User","Email","Host","Suffix","Completion","Transfer_Type","Transfer_notes","Start_of_transfer","Session_ID","Starting_size","Starting_offset"

[fields_dirlist]
DELIMS = ","
FIELDS = "Head","Pathname","Completion","Pattern","Recursion","User","Email","Host","Session ID"

Now when you search, apply the appropriate field extraction depending on the value of the "operation" field of the events you are querying :

sourcetype=xferlog operation=R OR operation=S | extract fields_store_retrieve

or:

sourcetype=xferlog operation=T | extract fields_dirlist

It's too bad that one cannot define automatic field extractions based on event types because this would have been an ideal use-case for that.

Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...