I am attempting to have splunk forward a script of comma separated values. The values are coming into search as one large string, rather than separated by commas with their field label. Could anyone look this over and see what I am doing wrong?
transforms.conf
[group_fields]
DELIMS=","
FIELDS = Record_Date,filesystem1,filesystem12,filesystem3,filesystem4,filesystem5,filesystem6,filesystem7
props.conf
[forecast]
SHOULD_LINEMERGE = False
pulldown_type = 1
REPORT-getfields = group_fields
inputs.conf
[script://./bin/forecast.sh]
interval = 83400
source = forecast
sourcetype = forecast
In the splunk search, it's showing up like this. It is not creating comma delimited fields, just one raw field of all the data
TIMESTAMP RAW
6/13/17
8:04:08.000 AM 06-08-17,424,159,1067,606,7,1,1
The script outputs the data as below.
11/27/2016,289,159,866,1221,7,1,1
11/28/2016,289,159,866,1221,7,1,1
11/29/2016,289,159,813,1258,7,1,1
11/30/2016,289,159,812,1338,7,1,1
12/4/2016,304,159,828,1321,7,1,1
12/5/2016,304,159,828,1321,7,1,1
12/6/2016,295,159,830,1327,7,1,1
Hi Sweenj,
Try this and let us know how it goes
props.conf
[forecast]
INDEXED_EXTRACTIONS = csv
KV_MODE = none
category = Structured
description = Comma-separated value format.
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
pulldown_type = true
disabled = false
REPORT-getfields = forecast_fields
transforms.conf
[forecast_fields]
DELIMS=","
FIELDS = "Record_Date","filesystem1","filesystem12","filesystem3","filesystem4","filesystem5","filesystem6","filesystem7"
I made the changes and bounced splunk. It didn't seem to make a difference.
Can I use the extract fields process in the GUI to make this distinction?
Hi Sweenj,
Are you not seeing key value pairs in the interesting fields in verbose mode? the event might appear as single string with commas but you should have the fields created and you should be able to use them in your search query.
Also can you share information on your architecture, all of this is on one single server or you have got search head, indexer and the server where your are trying to forward from?
I'm not really sure how I turn on this verbose mode.
This is a server with splunk forwarder pushing to a separate indexer.
If I use
sourcetype="forecast" host="node" | fields + "filesystem1"
or
fields + "filesystem1"
no changes are made, still just getting the raw event.
Hi Sweenj,
I hope you have made the suggested changes to transforms.conf and props.conf on the indexer(and bounce it), if NOT go ahead and do them on indexer. Once done , on your search head run below command by selecting "Verbose Mode" the dropdown next to search icon which displays "Fast Mode" "Smart Mode" "Verbose Mode". Once you run below command with "Verbose Mode" and the search complete, look for interesting fields on the left hand side and you should see the fields which you listed in the transforms.conf.
sourcetype="forecast" host="node"
Check and let us know
Download the *nix app from apps.splunk.com and see how it does this and then do it the same way. For one thing, I see that your first event's date is different than your other events' dates. This will surely be a problem.
Hey woodcock, I have more data in the file, that's just a sample. Why would that matter though as it's a range of dates? Wouldn't it just not have an entry for that particular date?
Thanks for taking a look.