Getting Data In

csv with headers

a212830
Champion

Hi,

I have a csv file with headers that needs processing. I want to 1) filter out the header and 2) have the fields recognized in the indexer. I tried following the steps listed in this link: http://splunk-base.splunk.com/answers/41551/how-do-i-get-auto-field-detection-on-forwarded-csv ,but it's not working.

My csv file:

"Time","IOPS","Latency(ms)","BW (MBps)"
"2013-04-03 22:06:00","9715","3.0","353.0"
"2013-04-03 22:07:00","8308","2.0","179.0"
"2013-04-03 22:08:00","6436","3.0","244.0"
"2013-04-03 22:09:00","4894","4.0","223.0"
"2013-04-03 22:10:00","4730","4.0","246.0"

input.conf:
[monitor:///fisc/dasd/xiv/perfstats/*.csv]
index=perfstats
sourcetype=xiv:perf:arrayStats
followTail = 0

props.conf:

[xiv:perf:arrayStats]
CHECK_FOR_HEADER = true
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = 1
TIME_FORMAT = %Y-%m-%d %H:%M:$S
SHOULD_LINEMERGE = false
TIME_PREFIX = ^"
TZ = US/Eastern

TRANSFORMS-xiv:perf:arrayStats = NoHeader
TRANSFORMS-xiv:perf:arrayStats = csv-fieldextraction

transforms.conf:
[NoHeader]
REGEX = "Time","IOPS","Latency(ms)","BW (MBps)"
DEST_KEY = queue
FORMAT = nullQueue

[csv-fieldextraction]
DELIMS=","
FIELDS="Time","IOPS","Latency","BW (MBps)"
props file:

Tags (2)
0 Karma

sbrant_splunk
Splunk Employee
Splunk Employee

Have you considered bringing this all in, as a single event and then just piping it to multikv at search time? This will utilize the header as the field name and eliminate the overhead of processing this pre-index.

0 Karma

a212830
Champion

Can't be a single event - they are different timestamps associated with performance stats.

0 Karma
Get Updates on the Splunk Community!

The Payment Operations Wake-Up Call: Why Financial Institutions Can't Afford ...

The same scenario plays out across financial institutions daily. A payment system fails at 11:30 AM on a busy ...

Make Your Case: A Ready-to-Send Letter for Getting Approval to Attend .conf25

Hello Splunkers, Want to attend .conf25 in Boston this year but not sure how to convince your manager? We've ...

Community Spotlight: A Splunk Expert's Journey

In the world of data analytics, some journeys leave a lasting impact not only on the individual but on the ...