topic Heavy Forwarder Selective forwarding in Splunk Search

Heavy Forwarder Selective forwarding

Adevill — Mon, 12 Apr 2021 10:19:38 GMT

Hey all. I need help to selective forward (on a HF) from a log file that is being monitored by a UF. I only need to forward lines that contain the exact words "Read line". I've tried the below confs but the HF is still forwarding all lines that are written to the log.

props.conf

[dcs_event] TRANSFORMS-routing = dcs_allow,dcs_drop

transforms.conf

[dcs_allow] DEST_KEY = queue REGEX = (Read line) FORMAT = indexQueue [dcs_drop] DEST_KEY = queue REGEX = . FORMAT = nullQueue

Re: Heavy Forwarder Selective forwarding

gcusello — Mon, 12 Apr 2021 10:25:48 GMT

Hi @Adevill,

please try inverting the commands in props.conf

[dcs_event] TRANSFORMS-routing = dcs_drop, dcs_allow

You have to put before the one containing all the logs (dcs_drop) and then the one to take the selected logs (dcs_allow) as described at https://docs.splunk.com/Documentation/Splunk/8.1.3/Forwarding/Routeandfilterdatad#Keep_specific_events_and_discard_the_rest

the problem is only on props.conf, instead it isn't relevant in transforms.conf.

Ciao.

Giuseppe

Re: Heavy Forwarder Selective forwarding

Adevill — Mon, 12 Apr 2021 10:38:26 GMT

Ciao Giuseppe @gcusello

I've made the change you indicated but all the lines are still being forwarded. Any other advice?

Regards

Anton

Re: Heavy Forwarder Selective forwarding

gcusello — Mon, 12 Apr 2021 10:44:49 GMT

Hi @Adevill,

let me understand:

you inverted the positions in props.conf command,
you restarted HF,
you continue to receive all the lines from the HF;

is it correct?

One quick question: you have a multi line event or a single line event?

The above approach is correct in single lines events, if instead you have a multi line events and you want to take only a part of the event, it's different.

Could you share a sample of your logs (both lines to take and likes to discard)?

Ciao.

Giuseppe

Re: Heavy Forwarder Selective forwarding

Adevill — Mon, 12 Apr 2021 10:48:05 GMT

Ciao @gcusello

Yes you are correct with all 3 points. The log file is being written to continuously, so I'm not sure if that will be seen as a single or multline event. I've used the below link for testing:

https://regex101.com/r/3RmTnm/1

Re: Heavy Forwarder Selective forwarding

gcusello — Mon, 12 Apr 2021 11:15:24 GMT

Hi @Adevill,

they seems to be single line events, so the configuration seem to be correct!

It shouldn't be useful, but, please try this regex in transforms.conf:

REGEX = \s+Read line:

Another stupid question: are you sure that those logs pass through the HF?

Try to put the same props.conf and transforms.conf also on Indexers.

Ciao.

Giuseppe

Re: Heavy Forwarder Selective forwarding

Adevill — Mon, 12 Apr 2021 11:33:00 GMT

Ciao @gcusello

I've tried both your suggestions (change the regex and put the conf files at indexer) but all the lines are still being indexed. Any other suggestions I can try?

Re: Heavy Forwarder Selective forwarding

gcusello — Mon, 12 Apr 2021 11:59:45 GMT

Hi @Adevill,

at this point, try the more strange things: are you sure that the sourcetype of your logs is "dcs_event"?

Ciao.

Giuseppe

Re: Heavy Forwarder Selective forwarding

Adevill — Mon, 12 Apr 2021 13:28:57 GMT

Hi @gcusello

Yes, I'm sure, I've also double checked it and the spelling of everything.

Re: Heavy Forwarder Selective forwarding

gcusello — Mon, 12 Apr 2021 13:38:49 GMT

Hi @Adevill,

could you share the inputs.conf on the UF that takes those logs?

Ciao.

Giuseppe

Re: Heavy Forwarder Selective forwarding

Adevill — Mon, 12 Apr 2021 13:41:01 GMT

Hi @gcusello

As requested:

[monitor://C:\Program Files (x86)\ABB Symphony Plus\Operations\History\PlantConnect.SYS\Debug\PlaCoEventImporter\Administrator_PlaCoEventImporter.log] disabled = 0 source = dcs sourcetype = dcs_event

Re: Heavy Forwarder Selective forwarding

gcusello — Mon, 12 Apr 2021 14:02:48 GMT

Hi @Adevill,

it's really strange because:

the sourcetype on UF's inputs.conf is correct (dcs_event);
the props.conf on HF is correct

[dcs_event] TRANSFORMS-routing = dcs_drop, dcs_allow

you restarted HF after props.conf modification;
The regex in transforms.conf is correct (otherwise you discard all the events),
events pass through the HF so they should be parsed by HF.

The only hint I can give is check again the above chain: something could be different.

Ciao.

Giuseppe

Re: Heavy Forwarder Selective forwarding

Adevill — Mon, 12 Apr 2021 14:29:32 GMT

@gcusello ,thank you for all the assistance. I'll go back and check everything everything.

Re: Heavy Forwarder Selective forwarding

gcusello — Mon, 12 Apr 2021 14:33:13 GMT

Hi @Adevill,

I'm sorry to be not able to help you more: it's really strange I did many of these condifugrations without any problem.

Check again every step and surely there's a little particular the will solve the problem.

Ciao.

Giuseppe