Splunk Search

Field Extraction of CSV on forwarder

lohit
Path Finder

Hi,

I have setup an universal forwarder to monitor a csv file and send the output to indexer(single instance acting as splunk web also). below are the config files on Universal forwarder.

input.conf [monitor://C:\logs_sql\logs.csv]

props.conf [source::C:\logssql\logs.csv] SHOULDLINEMERGE= false REPORT-logcsv=logcsv

tranforms.conf

[log_csv] DELIMS="," FIELDS="visitorid","uniqueid","country","Browser","Referrer","entrydate","IP"

Sample log file

visitorid,uniqueid,country,Browser,Referrer,entrydate,IP 10,26837:63373,United Kingdom,Mozilla/4.0,2009-10-02 14:59:37.437000000,16410458

After performing these changes nothing chnaged in output in splunk web.All the changes have been done on Universal forwarder only. Please let me know where i am going wrong?

Tags (1)
0 Karma

Ayn
Legend

The problem lies in what you've said yourself - all the changes have been made on the forwarder, not your indexer. Almost all field extraction takes place at search-time, not index-time, so fields are extracted when you run your query on the indexer. As such, all search-time settings should go on the indexer, not the forwarder.

lohit
Path Finder

Thanks a ton Ayn,Kristian for your help. am able to run searches over it. I would like to know one thing i have uninstall Splunk UV and trying to delete source,sourcetype,host from Splunk web using |delete(have assigned can_delete role) .Initially it was working fine. But i follow uninstall/install UV process 5-6 times. and after that i am not able to delete it. If i am trying to view the events from source/sourcetype it is showing No events but on splunkweb it is still listed among source/sourcetype. What should be the reason around this.

0 Karma

kristian_kolb
Ultra Champion

Also, DELIMS and FIELDS should be on separate lines in transforms.conf.

[log_csv]
DELIMS =","
FIELDS = blah, bleh, bluh, bloh ...

/K

0 Karma

Ayn
Legend

Yes, that is correct.

0 Karma

lohit
Path Finder

if i got u correctly u saying the indexed logs will still be shown in csv format. and i have to remember the fields i specified in tranform.conf and use that field in query itself.

0 Karma

Ayn
Legend

Like I said, THAT will NOT change. Raw output will still be the same regardless of how many fields you extract from it.

0 Karma

lohit
Path Finder

Correct but it is not happening. file is still showing in csv default format

0 Karma

Ayn
Legend

Yes, if everything went OK you should be able to do something like sourcetype="dblogs1" | table country and get values for the country field.

0 Karma

lohit
Path Finder

i have specified the sourcetype a custom one.Also in splunk web shouldnot it be like country=US etc.

0 Karma

Ayn
Legend

The raw output itself will not change. What will happen when this works properly is that the fields you've specified will be available.

Your settings look OK, provided that they correspond to the CSV file's format, that sourcetype is correct etc.

lohit
Path Finder

@Kristian.

these are file on indexer
props.conf
[dblogs1]
SHOULD_LINEMERGE= false
REPORT-logcsv=log_csv

tranforms.conf
[log_csv]
DELIMS="," FIELDS="visitorid","uniqueid","country","Browser","Referrer","entrydate","IP"

I guess mapping is right but not getting desired output

0 Karma

kristian_kolb
Ultra Champion

Re-indexing not needed here. This is all search-time stuff, but with a REPORT directive in props.conf without a matching stanza in transforms.conf, the DELIMS/FIELDS will not be applied at all.

Thus, ensure that they match, like so (only relevant parts shown);

props.conf
[dblogs1]
REPORT-blah = log_csv

transforms.conf
[log_csv]

/K

0 Karma

lohit
Path Finder

@kristian

yeah its a typo. but taken care in conf files.

@linu

I have deleted the data from splunk web and reintsall splunk forwarder.What steps should i follow to get it done?

0 Karma

linu1988
Champion

Were the data re-indexed?

0 Karma

kristian_kolb
Ultra Champion

SHOULDLINEMERGE != SHOULD_LINEMERGE

logcsv != log_csv

typos?

0 Karma

lohit
Path Finder

Hi Ayn..i have done the follwing changes

On forwarder

inputs.conf
[monitor://C:logs_sql/logs.csv]
sourcetype=dblogs1

ON indexer

props.conf
[dblogs1]
SHOULDLINEMERGE= false
REPORT-logcsv=logcsv

tranforms.conf
[log_csv]
DELIMS="," FIELDS="visitorid","uniqueid","country","Browser","Referrer","entrydate","IP"

Still files in splunk web is appearing in default way.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...