topic Re: Universal Forwarder and CSV (from Remote System) in Getting Data In

Universal Forwarder and CSV (from Remote System)

nikhilmehra79 — Thu, 06 Mar 2014 19:04:42 GMT

Hi,

I have a Universal Forwarder whose source file is reading all files in a specific directory , the dir has many files including HTM and CSV files, forwarder is reading all HTML files but not CSV by default. SO indexer is not seeing that data which in CSV file....what am i missing here.

From remote system
Here is my input.conf

[default]
host = ABCD

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0

[monitor://D:\web\System_Availablity_Analytics\*.csv]

Re: Universal Forwarder and CSV (from Remote System)

lukejadamec — Thu, 06 Mar 2014 19:18:53 GMT

Are you seeing any errors in the splunkd log?

Re: Universal Forwarder and CSV (from Remote System)

nikhilmehra79 — Mon, 28 Sep 2020 16:03:51 GMT

Yes i see them
03-06-2014 13:14:45.760 -0600 ERROR TailingProcessor - Ignoring path="D:\web\System_Availablity_Analytic\Report.csv" due to: Cannot checksum file due to unknown charset="AUTO".f
03-06-2014 13:14:50.635 -0600 ERROR ExecProcessor - message from ""C:\Program Files\SplunkForwarder\bin\splunk-MonitorNoHandle.exe"" wmain: Operating system major version 5, detected -- A minimum of 6 (VISTA/Server 2008) is required. Exitting.

Re: Universal Forwarder and CSV (from Remote System)

nikhilmehra79 — Thu, 06 Mar 2014 19:28:04 GMT

03-06-2014 13:14:55.651 -0600 ERROR ExecProcessor - message from ""C:\Program Files\SplunkForwarder\bin\splunk-netmon.exe"" splunk-netmon - Splunk network monitor is not available on this version of Windows.
03-06-2014 13:15:00.651 -0600 INFO ExecProcessor - message from ""C:\Program Files\SplunkForwarder\bin\splunk-regmon.exe"" splunk-regmon - SysmonMigrator::read - 'sysmon.conf' was not found, no migration is required.

Re: Universal Forwarder and CSV (from Remote System)

lukejadamec — Thu, 06 Mar 2014 19:35:41 GMT

Only the first one relates to your csv problem.

Does the file use a character set other than utf-8 or ascii?

Re: Universal Forwarder and CSV (from Remote System)

nikhilmehra79 — Thu, 06 Mar 2014 19:46:12 GMT

no it should be ASCII....

Here is CSV file

"","Availability"
"","%"
"","All"
"",""
"Element",""
"ABCD",100.00000000

"Auto Range: Previous Hour","Subject: REPORT","Created: 03/06/2014 12:55:03 PM"
"From: 03/06/2014 11:55 AM","","Time Zone: (GMT-06:00) Central Time"
"To: 03/06/2014 12:55 PM"

Re: Universal Forwarder and CSV (from Remote System)

lukejadamec — Thu, 06 Mar 2014 19:50:47 GMT

Is that the whole thing or one event?
I would copy one of the files to the indexer c:temp directory, and use the monitor files and directories GUI to create the inputs.conf and props.conf configs, and then transfer those configs to the forwarder.
If you do this, when you transfer the configs you will have to correct the monitor path when you move the configs to the forwarder.

Re: Universal Forwarder and CSV (from Remote System)

nikhilmehra79 — Thu, 06 Mar 2014 20:03:09 GMT

no its whole file...

Re: Universal Forwarder and CSV (from Remote System)

nikhilmehra79 — Thu, 06 Mar 2014 20:04:12 GMT

so if i create this through indexer ....next time this file is overwritten will it work?

Re: Universal Forwarder and CSV (from Remote System)

lukejadamec — Thu, 06 Mar 2014 20:25:54 GMT

The source will be different, so yes. Once you are done creating the configs and move them to the forwarder, then you can delete the input from the indexer. Don't forget to use a test index, and change the index in inputs.conf to 'main' when you move the configs to the forwarder.

Re: Universal Forwarder and CSV (from Remote System)

nikhilmehra79 — Thu, 06 Mar 2014 20:32:27 GMT

ok confused a little bit...
I put this in indexer and used GUI to map CSV file - i see it indexed the CSV file locally and it shows content on the Summary Tab

Now when i go to find indexer.conf and props.conf - i see many of them on system but not sure where it made change...which file will it write to and what i need to change before i move to unversal forwarder. Thanks for your help on this.

Re: Universal Forwarder and CSV (from Remote System)

lukejadamec — Thu, 06 Mar 2014 20:42:13 GMT

On the indexer, check etc/apps/?/local/inputs.conf for the input. The app might be will be what ever app you were in before you went to the manager.
The props.conf will be in the etc/apps/learned/local/props.conf
You will only need the stanzas for the input you created in inputs.conf, and the sourcetype that was created in props.conf. They should both be at the bottom.

Re: Universal Forwarder and CSV (from Remote System)

nikhilmehra79 — Mon, 28 Sep 2020 16:03:53 GMT

i dont see anything in input.conf which has changed today (search app is default), props.conf has this entry
[csv-2]
KV_MODE = none
REPORT-AutoHeader = AutoHeader-1
SHOULD_LINEMERGE = False
pulldown_type = true

So should go to same place in Universal Forwarder?

Re: Universal Forwarder and CSV (from Remote System)

lukejadamec — Thu, 06 Mar 2014 20:58:15 GMT

The inputs.conf is probably not necessary, because you already have one on the forwarder. What you will need to do is add the new sourcetype to the input stanza on the forwarder like this:

[monitor://D:\web\System_Availablity_Analytics\*.csv] sourcetype = csv-2

Re: Universal Forwarder and CSV (from Remote System)

nikhilmehra79 — Mon, 28 Sep 2020 16:03:58 GMT

and i see the data , here is what i did
Added stanza to etc/apps/learned/local/props.conf and modified input.conf to listen
[monitor://D:\web\System_Availablity_Analytics*.csv] sourcetype = csv-2

Thanks for all help on this, lastly - i see the file being read by indexer as breaking events not the way i like - in order for me to make it learn how it reads:

In past i was building props.conf of a particular application to write logic to split where i want to, is it best to put the Global Props.conf in Search App and any new App i build will leverage that?

Re: Universal Forwarder and CSV (from Remote System)

lukejadamec — Thu, 06 Mar 2014 21:40:13 GMT

Good news. If you can provide specifics then perhaps we can help.

Re: Universal Forwarder and CSV (from Remote System)

lukejadamec — Fri, 07 Mar 2014 15:53:29 GMT

The best place to put global props.conf stuff is on the indexer in etc/system/local.

Re: Universal Forwarder and CSV (from Remote System)

nikhilmehra79 — Wed, 12 Mar 2014 03:03:07 GMT

Also where is best place to put the same props.conf in Universal Forwarder