Splunk IT Service Intelligence
Highlighted

ingest CSV file into metric index

Explorer

I have following data file which I want to ingest as metrics data. Is it possible?

NodeName, IfIndex, datetime, ifInOctets, ifOutOctets, ifInErrors, ifOutErrors, locIfCarTrans, locIfInCRC, locIfOutputQueueDrops, locIfInputQueueDrops, IfName, IfType, IfSpeed, IfAlias
server1.com,066,20190821 13:01:10,0,350782,0,0,0,0,0,0,Gi2/42,ethernetCsmacd,1 Gbps, eth6-1.1.1.1
server2.com,051,20190821 13:01:10,0,0,0,0,0,0,0,0,Gi2/3,ethernetCsmacd,1 Gbps, Gig1.6-Private-intf
server3.com,111,20190821 13:01:25,0,0,0,0,0,0,0,0,Gi3/39,ethernetCsmacd,1 Gbps, ETH0 decom
server4.com,003,20190821 13:01:30,15179,4690,0,0,0,0,0,0,Gi0/1,ethernetCsmacd,40 Mbps, Gi0/2.3412 : MetroE 40M

Notice, datatime format is yyyymmdd HH:MM:SS format and its in 3rd column. NOT in first column.
metric names are ifInOctets, ifOutOctets, ifInErrors, ifOutErrors, locIfCarTrans, locIfInCRC, locIfOutputQueueDrops, locIfInputQueueDrops
remaining other fields Node
Name, IfIndex, AND IfName, IfType, IfSpeed, IfAlias need to use as dimensions.

0 Karma
Highlighted

Re: ingest CSV file into metric index

SplunkTrust
SplunkTrust

Your CSV is not in the expected format. See https://docs.splunk.com/Documentation/Splunk/7.3.1/Metrics/GetMetricsInOther#Get_metrics_in_from_fil....

Another method is to define a log-to-metric sourcetype. Go to Settings->Source types and create a new type. Be sure to choose "Log to Metrics" as the Category. Then click on the Metrics tab and enter your measures and dimensions. Next, create props and transforms for your sourcetype to extract the fields.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: ingest CSV file into metric index

Explorer

Thanks Rich,

to simplify, I re-arranged columns as shown below with time stamp as first field, then metrics and then dimensions.
I created sourcetype logtomonitor and used ifOutOctets,ifInErrors,ifOutErrors,locIfCarTrans,locIfInCRC
,locIfOutputQueueDrops,locIfInputQueueDrops as metrics. I added monitor on splunk forwarder.
However, I am not seeing any data into metric index.
is this due to date_time format coming from log files?

date_time,ifInOctets,ifOutOctets,ifInErrors,ifOutErrors,locIfCarTrans,locIfInCRC
,locIfOutputQueueDrops,locIfInputQueueDrops,Node_Name,IfIndex,IfName,IfType,IfSpeed,IfAlias
20190821 17:53:10,300,348,0,0,0,0,0,0,server1.com,001,Gi0/0/0,ethernetCsmacd,40 Mbps,Link to zzz : Gi2/2/0.3421: MetroE 40M : Vlan3424 :
20190821 17:53:40,57611,23173,0,0,0,0,0,0,server2.com,001,Gi0/0/0,ethernetCsmacd,20 Mbps,Link to yyy : Vlan3596 : MetroE 20M : Vlan3596 : IPC
20190821 17:53:40,62129,36565,0,0,0,0,0,0,server3.net,001,Gi0/0/0,ethernetCsmacd,40 Mbps,BC : Link [w/Nano] to xxx : G0/0/10 : iLite

0 Karma
Highlighted

Re: ingest CSV file into metric index

SplunkTrust
SplunkTrust

If you have TIME_FORMAT = %Y%m%d %H:%M:%S in your props.conf file for that sourcetype then the date_time column should not be a factor. What are your props.conf and transforms.conf settings?

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: ingest CSV file into metric index

Explorer

Hi Rich,

props.conf on indexer server
[5secpollerinterfacemetrics]
DATETIME
CONFIG =
INDEXEDEXTRACTIONS = csv
LINE
BREAKER = ([\r\n]+)
METRIC-SCHEMA-TRANSFORMS = metric-schema:5secpollerinterfacemetrics1566415584485
NOBINARYCHECK = true
category = Log to Metrics
description = 5secpollerinterfacemetrics
pulldown
type = 1

transforms.conf
[metric-schema:5secpollerinterfacemetrics1566415584485]
METRIC-SCHEMA-MEASURES = ifInOctets,ifOutOctets,ifInErrors,ifOutErrors,locIfCarTrans,locIfInCRC,locIfOutputQueueDrops,locIfInputQueueDrops

0 Karma
Highlighted

Re: ingest CSV file into metric index

SplunkTrust
SplunkTrust

That looks like it would work. Verify the fields are extracted properly by commenting-out the METRIC-SCHEMA-TRANSFORMS line in props.conf and sending the data to an events index.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: ingest CSV file into metric index

Explorer

In that case which index gets the data?

0 Karma
Highlighted

Re: ingest CSV file into metric index

SplunkTrust
SplunkTrust

The index specified in inputs,conf, which must be a metrics index.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: ingest CSV file into metric index

Explorer

Also, Do I need to add TIME_FORMAT = %Y%m%d %H:%M:%S line in props.conf as you indicated earlier?

0 Karma
Highlighted

Re: ingest CSV file into metric index

SplunkTrust
SplunkTrust

It's a best practice to always include TIME_FORMAT along with TIME_PREFIX, MAX_TIMESTAMP_LOOKAHEAD, SHOULD_LINEMERGE, LINE_BREAKER, and TRUNCATE in all props.conf stanzas.

---
If this reply helps you, an upvote would be appreciated.
0 Karma