Splunk Search

Splunk extract data with comma deliminator

Explorer

I am attempting to have splunk forward a script of comma separated values. The values are coming into search as one large string, rather than separated by commas with their field label. Could anyone look this over and see what I am doing wrong?

transforms.conf

 [group_fields] 
DELIMS="," 
FIELDS = Record_Date,filesystem1,filesystem12,filesystem3,filesystem4,filesystem5,filesystem6,filesystem7

props.conf

[forecast]
SHOULD_LINEMERGE = False
pulldown_type = 1
REPORT-getfields = group_fields

inputs.conf

 [script://./bin/forecast.sh]
 interval = 83400
 source = forecast
 sourcetype = forecast

In the splunk search, it's showing up like this. It is not creating comma delimited fields, just one raw field of all the data

TIMESTAMP                     RAW
    6/13/17
8:04:08.000 AM  06-08-17,424,159,1067,606,7,1,1

The script outputs the data as below.

11/27/2016,289,159,866,1221,7,1,1
11/28/2016,289,159,866,1221,7,1,1
11/29/2016,289,159,813,1258,7,1,1
11/30/2016,289,159,812,1338,7,1,1
12/4/2016,304,159,828,1321,7,1,1
12/5/2016,304,159,828,1321,7,1,1
12/6/2016,295,159,830,1327,7,1,1

Communicator

Hi Sweenj,

Try this and let us know how it goes

props.conf

[forecast]
INDEXED_EXTRACTIONS = csv
KV_MODE = none
category = Structured
description = Comma-separated value format.  
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
pulldown_type = true
disabled = false
REPORT-getfields = forecast_fields

transforms.conf

[forecast_fields] 
DELIMS="," 
FIELDS = "Record_Date","filesystem1","filesystem12","filesystem3","filesystem4","filesystem5","filesystem6","filesystem7"

Explorer

I made the changes and bounced splunk. It didn't seem to make a difference.

Can I use the extract fields process in the GUI to make this distinction?

0 Karma

Communicator

Hi Sweenj,

Are you not seeing key value pairs in the interesting fields in verbose mode? the event might appear as single string with commas but you should have the fields created and you should be able to use them in your search query.

Also can you share information on your architecture, all of this is on one single server or you have got search head, indexer and the server where your are trying to forward from?

0 Karma

Explorer

I'm not really sure how I turn on this verbose mode.

This is a server with splunk forwarder pushing to a separate indexer.

If I use

sourcetype="forecast" host="node" | fields + "filesystem1" 

or

 fields + "filesystem1"  

no changes are made, still just getting the raw event.

0 Karma

Communicator

Hi Sweenj,

I hope you have made the suggested changes to transforms.conf and props.conf on the indexer(and bounce it), if NOT go ahead and do them on indexer. Once done , on your search head run below command by selecting "Verbose Mode" the dropdown next to search icon which displays "Fast Mode" "Smart Mode" "Verbose Mode". Once you run below command with "Verbose Mode" and the search complete, look for interesting fields on the left hand side and you should see the fields which you listed in the transforms.conf.

sourcetype="forecast" host="node"

Check and let us know

0 Karma

Esteemed Legend

Download the *nix app from apps.splunk.com and see how it does this and then do it the same way. For one thing, I see that your first event's date is different than your other events' dates. This will surely be a problem.

0 Karma

Explorer

Hey woodcock, I have more data in the file, that's just a sample. Why would that matter though as it's a range of dates? Wouldn't it just not have an entry for that particular date?

Thanks for taking a look.

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!