Splunk Search

Splunk extract data with comma deliminator

sweenj
Explorer

I am attempting to have splunk forward a script of comma separated values. The values are coming into search as one large string, rather than separated by commas with their field label. Could anyone look this over and see what I am doing wrong?

transforms.conf

 [group_fields] 
DELIMS="," 
FIELDS = Record_Date,filesystem1,filesystem12,filesystem3,filesystem4,filesystem5,filesystem6,filesystem7

props.conf

[forecast]
SHOULD_LINEMERGE = False
pulldown_type = 1
REPORT-getfields = group_fields

inputs.conf

 [script://./bin/forecast.sh]
 interval = 83400
 source = forecast
 sourcetype = forecast

In the splunk search, it's showing up like this. It is not creating comma delimited fields, just one raw field of all the data

TIMESTAMP                     RAW
    6/13/17
8:04:08.000 AM  06-08-17,424,159,1067,606,7,1,1

The script outputs the data as below.

11/27/2016,289,159,866,1221,7,1,1
11/28/2016,289,159,866,1221,7,1,1
11/29/2016,289,159,813,1258,7,1,1
11/30/2016,289,159,812,1338,7,1,1
12/4/2016,304,159,828,1321,7,1,1
12/5/2016,304,159,828,1321,7,1,1
12/6/2016,295,159,830,1327,7,1,1

newbie2tech
Communicator

Hi Sweenj,

Try this and let us know how it goes

props.conf

[forecast]
INDEXED_EXTRACTIONS = csv
KV_MODE = none
category = Structured
description = Comma-separated value format.  
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
pulldown_type = true
disabled = false
REPORT-getfields = forecast_fields

transforms.conf

[forecast_fields] 
DELIMS="," 
FIELDS = "Record_Date","filesystem1","filesystem12","filesystem3","filesystem4","filesystem5","filesystem6","filesystem7"

sweenj
Explorer

I made the changes and bounced splunk. It didn't seem to make a difference.

Can I use the extract fields process in the GUI to make this distinction?

0 Karma

newbie2tech
Communicator

Hi Sweenj,

Are you not seeing key value pairs in the interesting fields in verbose mode? the event might appear as single string with commas but you should have the fields created and you should be able to use them in your search query.

Also can you share information on your architecture, all of this is on one single server or you have got search head, indexer and the server where your are trying to forward from?

0 Karma

sweenj
Explorer

I'm not really sure how I turn on this verbose mode.

This is a server with splunk forwarder pushing to a separate indexer.

If I use

sourcetype="forecast" host="node" | fields + "filesystem1" 

or

 fields + "filesystem1"  

no changes are made, still just getting the raw event.

0 Karma

newbie2tech
Communicator

Hi Sweenj,

I hope you have made the suggested changes to transforms.conf and props.conf on the indexer(and bounce it), if NOT go ahead and do them on indexer. Once done , on your search head run below command by selecting "Verbose Mode" the dropdown next to search icon which displays "Fast Mode" "Smart Mode" "Verbose Mode". Once you run below command with "Verbose Mode" and the search complete, look for interesting fields on the left hand side and you should see the fields which you listed in the transforms.conf.

sourcetype="forecast" host="node"

Check and let us know

0 Karma

woodcock
Esteemed Legend

Download the *nix app from apps.splunk.com and see how it does this and then do it the same way. For one thing, I see that your first event's date is different than your other events' dates. This will surely be a problem.

0 Karma

sweenj
Explorer

Hey woodcock, I have more data in the file, that's just a sample. Why would that matter though as it's a range of dates? Wouldn't it just not have an entry for that particular date?

Thanks for taking a look.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...