topic Re: Field Extraction of CSV on forwarder in Splunk Search

Field Extraction of CSV on forwarder

lohit — Mon, 12 Aug 2013 10:26:52 GMT

Hi,

I have setup an universal forwarder to monitor a csv file and send the output to indexer(single instance acting as splunk web also). below are the config files on Universal forwarder.

input.conf [monitor://C:\logs_sql\logs.csv]

props.conf [source::C:\logssql\logs.csv] SHOULDLINEMERGE= false REPORT-logcsv=logcsv

tranforms.conf

[log_csv] DELIMS="," FIELDS="visitorid","uniqueid","country","Browser","Referrer","entrydate","IP"

Sample log file

visitorid,uniqueid,country,Browser,Referrer,entrydate,IP 10,26837:63373,United Kingdom,Mozilla/4.0,2009-10-02 14:59:37.437000000,16410458

After performing these changes nothing chnaged in output in splunk web.All the changes have been done on Universal forwarder only. Please let me know where i am going wrong?

Re: Field Extraction of CSV on forwarder

Ayn — Mon, 12 Aug 2013 11:17:28 GMT

The problem lies in what you've said yourself - all the changes have been made on the forwarder, not your indexer. Almost all field extraction takes place at search-time, not index-time, so fields are extracted when you run your query on the indexer. As such, all search-time settings should go on the indexer, not the forwarder.

Re: Field Extraction of CSV on forwarder

lohit — Mon, 12 Aug 2013 11:50:01 GMT

Hi Ayn..i have done the follwing changes

On forwarder

inputs.conf
[monitor://C:logs_sql/logs.csv]
sourcetype=dblogs1

ON indexer

props.conf
[dblogs1]
SHOULDLINEMERGE= false
REPORT-logcsv=logcsv

tranforms.conf
[log_csv]
DELIMS="," FIELDS="visitorid","uniqueid","country","Browser","Referrer","entrydate","IP"

Still files in splunk web is appearing in default way.

Re: Field Extraction of CSV on forwarder

kristian_kolb — Mon, 12 Aug 2013 11:55:23 GMT

SHOULDLINEMERGE != SHOULD_LINEMERGE

logcsv != log_csv

typos?

Re: Field Extraction of CSV on forwarder

linu1988 — Mon, 12 Aug 2013 11:59:26 GMT

Were the data re-indexed?

Re: Field Extraction of CSV on forwarder

lohit — Mon, 12 Aug 2013 12:04:21 GMT

@kristian

yeah its a typo. but taken care in conf files.

@linu

I have deleted the data from splunk web and reintsall splunk forwarder.What steps should i follow to get it done?

Re: Field Extraction of CSV on forwarder

kristian_kolb — Mon, 12 Aug 2013 12:05:38 GMT

Re-indexing not needed here. This is all search-time stuff, but with a REPORT directive in props.conf without a matching stanza in transforms.conf, the DELIMS/FIELDS will not be applied at all.

Thus, ensure that they match, like so (only relevant parts shown);

props.conf
[dblogs1]
REPORT-blah = log_csv

transforms.conf
[log_csv]

Re: Field Extraction of CSV on forwarder

lohit — Mon, 28 Sep 2020 14:33:03 GMT

@Kristian.

these are file on indexer
props.conf
[dblogs1]
SHOULD_LINEMERGE= false
REPORT-logcsv=log_csv

tranforms.conf
[log_csv]
DELIMS="," FIELDS="visitorid","uniqueid","country","Browser","Referrer","entrydate","IP"

I guess mapping is right but not getting desired output

Re: Field Extraction of CSV on forwarder

Ayn — Mon, 12 Aug 2013 12:28:53 GMT

The raw output itself will not change. What will happen when this works properly is that the fields you've specified will be available.

Your settings look OK, provided that they correspond to the CSV file's format, that sourcetype is correct etc.

Re: Field Extraction of CSV on forwarder

lohit — Mon, 12 Aug 2013 12:32:12 GMT

i have specified the sourcetype a custom one.Also in splunk web shouldnot it be like country=US etc.

Re: Field Extraction of CSV on forwarder

Ayn — Mon, 12 Aug 2013 12:35:10 GMT

Yes, if everything went OK you should be able to do something like sourcetype="dblogs1" | table country and get values for the country field.

Re: Field Extraction of CSV on forwarder

lohit — Mon, 12 Aug 2013 12:38:56 GMT

Correct but it is not happening. file is still showing in csv default format

Re: Field Extraction of CSV on forwarder

Ayn — Mon, 12 Aug 2013 12:40:29 GMT

Like I said, THAT will NOT change. Raw output will still be the same regardless of how many fields you extract from it.

Re: Field Extraction of CSV on forwarder

lohit — Mon, 12 Aug 2013 12:53:09 GMT

if i got u correctly u saying the indexed logs will still be shown in csv format. and i have to remember the fields i specified in tranform.conf and use that field in query itself.

Re: Field Extraction of CSV on forwarder

Ayn — Mon, 12 Aug 2013 12:56:20 GMT

Yes, that is correct.

Re: Field Extraction of CSV on forwarder

kristian_kolb — Mon, 12 Aug 2013 14:22:10 GMT

Also, DELIMS and FIELDS should be on separate lines in transforms.conf.

[log_csv]
DELIMS =","
FIELDS = blah, bleh, bluh, bloh ...

Re: Field Extraction of CSV on forwarder

lohit — Mon, 12 Aug 2013 15:53:23 GMT

Thanks a ton Ayn,Kristian for your help. am able to run searches over it. I would like to know one thing i have uninstall Splunk UV and trying to delete source,sourcetype,host from Splunk web using |delete(have assigned can_delete role) .Initially it was working fine. But i follow uninstall/install UV process 5-6 times. and after that i am not able to delete it. If i am trying to view the events from source/sourcetype it is showing No events but on splunkweb it is still listed among source/sourcetype. What should be the reason around this.