Getting Data In

csv with headers

a212830
Champion

Hi,

I have a csv file with headers that needs processing. I want to 1) filter out the header and 2) have the fields recognized in the indexer. I tried following the steps listed in this link: http://splunk-base.splunk.com/answers/41551/how-do-i-get-auto-field-detection-on-forwarded-csv ,but it's not working.

My csv file:

"Time","IOPS","Latency(ms)","BW (MBps)"
"2013-04-03 22:06:00","9715","3.0","353.0"
"2013-04-03 22:07:00","8308","2.0","179.0"
"2013-04-03 22:08:00","6436","3.0","244.0"
"2013-04-03 22:09:00","4894","4.0","223.0"
"2013-04-03 22:10:00","4730","4.0","246.0"

input.conf:
[monitor:///fisc/dasd/xiv/perfstats/*.csv]
index=perfstats
sourcetype=xiv:perf:arrayStats
followTail = 0

props.conf:

[xiv:perf:arrayStats]
CHECK_FOR_HEADER = true
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = 1
TIME_FORMAT = %Y-%m-%d %H:%M:$S
SHOULD_LINEMERGE = false
TIME_PREFIX = ^"
TZ = US/Eastern

TRANSFORMS-xiv:perf:arrayStats = NoHeader
TRANSFORMS-xiv:perf:arrayStats = csv-fieldextraction

transforms.conf:
[NoHeader]
REGEX = "Time","IOPS","Latency(ms)","BW (MBps)"
DEST_KEY = queue
FORMAT = nullQueue

[csv-fieldextraction]
DELIMS=","
FIELDS="Time","IOPS","Latency","BW (MBps)"
props file:

Tags (2)
0 Karma

sbrant_splunk
Splunk Employee
Splunk Employee

Have you considered bringing this all in, as a single event and then just piping it to multikv at search time? This will utilize the header as the field name and eliminate the overhead of processing this pre-index.

0 Karma

a212830
Champion

Can't be a single event - they are different timestamps associated with performance stats.

0 Karma
Get Updates on the Splunk Community!

AppDynamics Summer Webinars

This summer, our mighty AppDynamics team is cooking up some delicious content on YouTube Live to satiate your ...

SOCin’ it to you at Splunk University

Splunk University is expanding its instructor-led learning portfolio with dedicated Security tracks at .conf25 ...

Credit Card Data Protection & PCI Compliance with Splunk Edge Processor

Organizations handling credit card transactions know that PCI DSS compliance is both critical and complex. The ...