topic Re: Stop splunk processing ? in Getting Data In

Stop splunk processing ?

flo_cognosec — Wed, 19 Dec 2012 14:57:36 GMT

I might get things wrong, but for now I have the following problem / setup

forwarder with some files in some directories monitored by fschange defined in inputs.conf
some processing options in props.conf (I would like to stop processing here as all has been done)

stuff gets sent to the indexer

now the issue is that splunk does start to parse the events again on the indexes and does indeed se-set some of my options.

So is it a good idea to do some parsing on the forwarder or is it possible to tell splunk to stop processing some events and just take what is coming in from the forwarder ?

Should I do all the processing on the indexer and only keep an inputs.conf on the forwarder and drop the props.conf ?

Side-question: if I set the sourcetype in inputs.conf, which events does this effectively affect when using some fschange stanza ?

Re: Stop splunk processing ?

flo_cognosec — Wed, 09 Jan 2013 10:59:26 GMT

No ideas or was the question phrased wrong ?

Re: Stop splunk processing ?

Drainy — Wed, 09 Jan 2013 11:25:44 GMT

Are you sure any parsing is actually being done on the forwarder? If you are using a Universal or Light forwarder then they don't actually do any parsing, the parsing is all handled by the indexer so what you may be experiencing is actually the normal behaviour, even if it isn't what you expected 🙂

Define all your parsing requirements at the indexer. Just define your inputs.conf on the universal forwarder and let the indexer handle the rest.

Re: Stop splunk processing ?

flo_cognosec — Wed, 09 Jan 2013 11:28:51 GMT

Actually some parsing IS done on the UF, there was a document explaining it but I can't find it anymore or maybe the changed that 😞

Thanks anyway 🙂

Re: Stop splunk processing ?

Drainy — Mon, 28 Sep 2020 13:04:10 GMT

No, parsing isn't handled on the UF. I've got the internal queue doc here listing its processes. It does have a parsing queue for windows events but thats a special exception. The only props configurations it will handle are for CHARSET, NO_BINARY_CHECK,CHECK_METHOD and CHECK_FOR_HEADER (depreciated in v5).
EDIT: From the public docs, http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Introducingtheuniversalforwarder.
The universal forwarder does not parse data.

Re: Stop splunk processing ?

flo_cognosec — Mon, 28 Sep 2020 13:04:19 GMT

So to make this very clear for everybody 😉

Putting this into the props.conf on the UF is useless as the sourcetype will not be assigned ?
Will the BREAK_ONLY_BEFORE stuff be evaluated ?
It looks like it is for me (version 4.3.4 build 136012.) but I need to do the same on the indexer ?

[source::/boot/...]
BREAK_ONLY_BEFORE_DATE=false
BREAK_ONLY_BEFORE=goblygook
LEARN_MODEL = false
LEARN_SOURCETYPE = false
MAX_EVENTS=200000
sourcetype=os_files

Re: Stop splunk processing ?

Drainy — Wed, 09 Jan 2013 13:04:58 GMT

You would set the sourcetype in the inputs.conf and then reference the sourcetype in the props on the indexer to handle the linebreaking.

Re: Stop splunk processing ?

flo_cognosec — Mon, 28 Sep 2020 13:04:30 GMT

What should be noted somewhere is the fact that assigning a sourcetype in inputs.conf AND using fschange stanza only means you can assign a sourcetype to this "kind" of event:

Wed Jan 9 18:22:00 2013 action=add, path="/sbin/bla_false.txt", isdir=0, size=66359, gid=0, uid=0, modtime="Wed Jan 9 18:19:55 2013", mode="rw-r--r--", hash=Bm8/v+HakIJOvaUvaEbn7ofqDHBh3VUs673BHCxaU6f= host=10.0.0.1
sourcetype=tmp_files
source=fschangemonitor
path=/sbin/bla_false.txt
action=add

but NOT to the event containing the file content itself.

Re: Stop splunk processing ?

flo_cognosec — Wed, 23 Jan 2013 10:54:41 GMT

Somehow the problem still exists.

Just assigning the sourcetype in the fschange stanza in inputs.conf and then on the indexer try to do some parsing in props.conf does NOT work in a useful way as described above.
(short: the file change event will get the correct sourcetype, the file content event not)

Testing shows that I need to assign the sourcetype in a source:: stanza on the UF in props.conf as well as assiging this on the indexer in props.conf did NOT work 😕

Where is this exactly documented and why isn't this working as one might expect ?

Re: Stop splunk processing ?

flo_cognosec — Wed, 23 Jan 2013 12:35:52 GMT

It somehow contradicts this wiki page

http://wiki.splunk.com/Deploy:HowToSetupFschange

If this page is wrong, maybe you are able to update it based on the most recent splunk docs.

Re: Stop splunk processing ?

flo_cognosec — Fri, 17 May 2013 12:58:40 GMT

Interesting enough this wiki page has not been changed and it still seems I have to assign sourcetypes in both the inputs.conf and the props.conf on the UF to catch all the information I need (file content AND the file / change metainformation) (so the wiki page actually seems to be correct)