Splunk Search

ncftpd log extractions

jspears
Communicator

I'm trying to do field extractions for ncftpd xfer logs. These are generally csv but the fields differ depending on what operation is being logged ( http://ncftpd.com/ncftpd/doc/xferlog.html if you want the details.)

Is there a clean way to extract the first few generic fields and do the other field extractions depending on the values extracted? Let's call the first field 'operation'. Can I say something like

search operation="S" OR operation="R" | do field extractions specific to these
search operation="T" | do field extractions specific to this

Tags (1)
1 Solution

hexx
Splunk Employee
Splunk Employee

You can't automate a conditional delimiter-based field extraction, but you can define one unique field extraction per type of xferlog event and select which one to apply using the extract command.

First, define an automatic field extraction for the "operation" field, the value of which should determine which delimiter-based extraction we will apply.

  • props.conf:

[xferlog]
EXTRACT-operation = ^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}[^|]*\|\s+(?\w),

Now in transforms.conf, define one delimiter-based field extraction for each type of event. As an example, based on the xferlog reference page here's one definition for the S and R log entry types and one for the T (directory listing) log entry type.

  • transforms.conf:

[fields_store_retrieve]
DELIMS = ","
FIELDS = "Head","Pathname","Size","Duration","Rate","User","Email","Host","Suffix","Completion","Transfer_Type","Transfer_notes","Start_of_transfer","Session_ID","Starting_size","Starting_offset"

[fields_dirlist]
DELIMS = ","
FIELDS = "Head","Pathname","Completion","Pattern","Recursion","User","Email","Host","Session ID"

Now when you search, apply the appropriate field extraction depending on the value of the "operation" field of the events you are querying :

sourcetype=xferlog operation=R OR operation=S | extract fields_store_retrieve

or:

sourcetype=xferlog operation=T | extract fields_dirlist

It's too bad that one cannot define automatic field extractions based on event types because this would have been an ideal use-case for that.

View solution in original post

hexx
Splunk Employee
Splunk Employee

You can't automate a conditional delimiter-based field extraction, but you can define one unique field extraction per type of xferlog event and select which one to apply using the extract command.

First, define an automatic field extraction for the "operation" field, the value of which should determine which delimiter-based extraction we will apply.

  • props.conf:

[xferlog]
EXTRACT-operation = ^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}[^|]*\|\s+(?\w),

Now in transforms.conf, define one delimiter-based field extraction for each type of event. As an example, based on the xferlog reference page here's one definition for the S and R log entry types and one for the T (directory listing) log entry type.

  • transforms.conf:

[fields_store_retrieve]
DELIMS = ","
FIELDS = "Head","Pathname","Size","Duration","Rate","User","Email","Host","Suffix","Completion","Transfer_Type","Transfer_notes","Start_of_transfer","Session_ID","Starting_size","Starting_offset"

[fields_dirlist]
DELIMS = ","
FIELDS = "Head","Pathname","Completion","Pattern","Recursion","User","Email","Host","Session ID"

Now when you search, apply the appropriate field extraction depending on the value of the "operation" field of the events you are querying :

sourcetype=xferlog operation=R OR operation=S | extract fields_store_retrieve

or:

sourcetype=xferlog operation=T | extract fields_dirlist

It's too bad that one cannot define automatic field extractions based on event types because this would have been an ideal use-case for that.

Get Updates on the Splunk Community!

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...

Adoption of Infrastructure Monitoring at Splunk

  Splunk's Growth Engineering team showcases one of their first Splunk product adoption-Splunk Infrastructure ...