topic Re: Splunk extract data with comma deliminator in Splunk Search

Splunk extract data with comma deliminator

sweenj — Tue, 13 Jun 2017 12:43:55 GMT

I am attempting to have splunk forward a script of comma separated values. The values are coming into search as one large string, rather than separated by commas with their field label. Could anyone look this over and see what I am doing wrong?

transforms.conf

 [group_fields] 
DELIMS="," 
FIELDS = Record_Date,filesystem1,filesystem12,filesystem3,filesystem4,filesystem5,filesystem6,filesystem7

props.conf

[forecast]
SHOULD_LINEMERGE = False
pulldown_type = 1
REPORT-getfields = group_fields

inputs.conf

 [script://./bin/forecast.sh]
 interval = 83400
 source = forecast
 sourcetype = forecast

In the splunk search, it's showing up like this. It is not creating comma delimited fields, just one raw field of all the data

TIMESTAMP                     RAW
    6/13/17
8:04:08.000 AM  06-08-17,424,159,1067,606,7,1,1

The script outputs the data as below.

11/27/2016,289,159,866,1221,7,1,1
11/28/2016,289,159,866,1221,7,1,1
11/29/2016,289,159,813,1258,7,1,1
11/30/2016,289,159,812,1338,7,1,1
12/4/2016,304,159,828,1321,7,1,1
12/5/2016,304,159,828,1321,7,1,1
12/6/2016,295,159,830,1327,7,1,1

Re: Splunk extract data with comma deliminator

woodcock — Tue, 13 Jun 2017 13:24:19 GMT

Download the *nix app from apps.splunk.com and see how it does this and then do it the same way. For one thing, I see that your first event's date is different than your other events' dates. This will surely be a problem.

Re: Splunk extract data with comma deliminator

sweenj — Tue, 13 Jun 2017 13:30:26 GMT

Hey woodcock, I have more data in the file, that's just a sample. Why would that matter though as it's a range of dates? Wouldn't it just not have an entry for that particular date?

Thanks for taking a look.

Re: Splunk extract data with comma deliminator

newbie2tech — Tue, 13 Jun 2017 14:29:44 GMT

Hi Sweenj,

Try this and let us know how it goes

props.conf

[forecast]
INDEXED_EXTRACTIONS = csv
KV_MODE = none
category = Structured
description = Comma-separated value format.  
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
pulldown_type = true
disabled = false
REPORT-getfields = forecast_fields

transforms.conf

[forecast_fields] 
DELIMS="," 
FIELDS = "Record_Date","filesystem1","filesystem12","filesystem3","filesystem4","filesystem5","filesystem6","filesystem7"

Re: Splunk extract data with comma deliminator

sweenj — Wed, 14 Jun 2017 11:17:24 GMT

I made the changes and bounced splunk. It didn't seem to make a difference.

Can I use the extract fields process in the GUI to make this distinction?

Re: Splunk extract data with comma deliminator

newbie2tech — Wed, 14 Jun 2017 13:56:59 GMT

Hi Sweenj,

Are you not seeing key value pairs in the interesting fields in verbose mode? the event might appear as single string with commas but you should have the fields created and you should be able to use them in your search query.

Also can you share information on your architecture, all of this is on one single server or you have got search head, indexer and the server where your are trying to forward from?

Re: Splunk extract data with comma deliminator

sweenj — Wed, 14 Jun 2017 18:50:46 GMT

I'm not really sure how I turn on this verbose mode.

This is a server with splunk forwarder pushing to a separate indexer.

If I use

sourcetype="forecast" host="node" | fields + "filesystem1"

 fields + "filesystem1"

no changes are made, still just getting the raw event.

Re: Splunk extract data with comma deliminator

newbie2tech — Wed, 14 Jun 2017 21:02:27 GMT

Hi Sweenj,

I hope you have made the suggested changes to transforms.conf and props.conf on the indexer(and bounce it), if NOT go ahead and do them on indexer. Once done , on your search head run below command by selecting "Verbose Mode" the dropdown next to search icon which displays "Fast Mode" "Smart Mode" "Verbose Mode". Once you run below command with "Verbose Mode" and the search complete, look for interesting fields on the left hand side and you should see the fields which you listed in the transforms.conf.

sourcetype="forecast" host="node"

Check and let us know