Splunk Search

How to configure props.conf for automatic field extractions from a CSV file without specifying all field names?

nk-1
Path Finder

Using Splunk v6.2.0

The default field-extraction ( sourcetype=csv ) from a CSV logfile worked fine, but it incorrectly used the seventh field as the event's timestamp.

So I created a new sourcetype, specifying DATETIME_CONFIG = CURRENT (in props.conf) for the event's timestamp, but now it lost all the default CSV field extractions.

How can I get it to do the default field extractions from CSV without having to specify all the field names in some .conf file?
(fieldnames are in the first line of every .csv logfile)

Stanza from my props.conf:

[ctrlup]
DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = csv
KV_MODE = none
CHECK_FOR_HEADER = true
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format, but using current time as event timestamp.
disabled = false
pulldown_type = true
0 Karma
1 Solution

nk-1
Path Finder

I found this in the Splunk Docs under Create custom fields at index time :

Note: If you are employing heavy forwarders in front of your search peers, the props and transforms processing takes place on the forwarders, not the search peers. Therefore, you must deploy the props and transforms changes to the forwarders, not the search peers.

splunkd.log also indicates that The CHECK_FOR_HEADER setting is deprecated.

Deployed this updated props.conf on the forwarder as well, and everything works as expected now.

[ctrlup]
DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = csv
KV_MODE = auto
HEADER_FIELD_LINE_NUMBER = 2
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format, but using current time as event timestamp.
disabled = false
pulldown_type = true

Thanks.

View solution in original post

nk-1
Path Finder

I found this in the Splunk Docs under Create custom fields at index time :

Note: If you are employing heavy forwarders in front of your search peers, the props and transforms processing takes place on the forwarders, not the search peers. Therefore, you must deploy the props and transforms changes to the forwarders, not the search peers.

splunkd.log also indicates that The CHECK_FOR_HEADER setting is deprecated.

Deployed this updated props.conf on the forwarder as well, and everything works as expected now.

[ctrlup]
DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = csv
KV_MODE = auto
HEADER_FIELD_LINE_NUMBER = 2
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format, but using current time as event timestamp.
disabled = false
pulldown_type = true

Thanks.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

Can you show us some of the data?

---
If this reply helps you, an upvote would be appreciated.
0 Karma

nk-1
Path Finder
Table: Sessions
Stress Level,ID,Session Name,User,Citrix Receiver Version,State,Connect Time,Disconnect Time,Idle Time,Logon Time,Processes,ICA Latency - Last Recorded,Client IP,Client Name,Computer,Domain DNS,Initial Program,CPU,Page Faults /sec,I/O Read Operations/sec,I/O Write Operations/sec,Memory (Private Bytes),Memory (Working Set),View Client Connection Server URL,View Client Domain,View Client Protocol,View Client Type,View Client Tunnel,User Logon Server,Group Policy Load Time,Profile Load Time,Logon Duration - Other,Desktop Load Time,Logon Duration
"4.00","0","Services","","","Disconnected","","","","","61","0","0.0.0.0","","XXXX-AAAA","YYYY.com","","0.125","119.235","56.11817","0","1.040945E+09","1.223873E+09","","","","","","","0","0","0","0","0"
"3.00","119","BBBB","ZZZZ","11.3.0.41560","Active","9/1/2015 2:39:59 PM","","00:00:00","9/1/2015 2:40:47 PM","26","47","10.0.0.1","CCCC","XXXX-DDDD","YYYY.com","My Desktop","0.25","118.1605","1.851977","1.474779","6.997975E+08","9.383158E+08","","","","","","XXXX-EEE","6","31","6","1","44"
0 Karma

nk-1
Path Finder

I am forwarding the .csv files from a different host using a Splunk Forwarder, if that makes a difference.

[monitor://C:\MyLogs]
crcSalt = <SOURCE>
index = ctrlup
sourcetype = ctrlup
disabled = false
0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!