Heavy Forwarder Selective forwarding

[Adevill]

Hey all. I need help to selective forward (on a HF) from a log file that is being monitored by a UF. I only need to forward lines that contain the exact words "Read line". I've tried the below confs but the HF is still forwarding all lines that are written to the log.

props.conf

[dcs_event]
TRANSFORMS-routing = dcs_allow,dcs_drop

transforms.conf

[dcs_allow]
DEST_KEY = queue
REGEX = (Read line)
FORMAT = indexQueue

[dcs_drop]
DEST_KEY = queue
REGEX = .
FORMAT = nullQueue

[gcusello]

Hi @Adevill,

please try inverting the commands in props.conf 

[dcs_event]
TRANSFORMS-routing = dcs_drop, dcs_allow

You have to put before the one containing all the logs (dcs_drop) and then the one to take the selected logs (dcs_allow) as described at https://docs.splunk.com/Documentation/Splunk/8.1.3/Forwarding/Routeandfilterdatad#Keep_specific_even...

the problem is only on props.conf, instead it isn't relevant in transforms.conf.

Ciao.

Giuseppe

[Adevill]

Ciao Giuseppe @gcusello 

I've made the change you indicated but all the lines are still being forwarded. Any other advice?

Regards

Anton

[gcusello]

Hi @Adevill,

let me understand:

• you inverted the positions in props.conf command,
• you restarted HF,
• you continue to receive all the lines from the HF;

is it correct?

One quick question: you have a multi line event or a single line event?

The above approach is correct in single lines events, if instead you have a multi line events and you want to take only a part of the event, it's different.

Could you share a sample of your logs (both lines to take and likes to discard)?

Ciao.

Giuseppe

[Adevill]

Ciao @gcusello 

Yes you are correct with all 3 points. The log file is being written to continuously, so I'm not sure if that will be seen as a single or multline event. I've used the below link for testing:

https://regex101.com/r/3RmTnm/1 

[gcusello]

Hi @Adevill,

they seems to be single line events, so the configuration seem to be correct!

It shouldn't be useful, but, please try this regex in transforms.conf:

REGEX = \s+Read line:

Another stupid question: are you sure that those logs pass through the HF?

Try to put the same props.conf and transforms.conf also on Indexers.

Ciao.

Giuseppe

[Adevill]

Ciao @gcusello 

I've tried both your suggestions (change the regex and put the conf files at indexer) but all the lines are still being indexed. Any other suggestions I can try?

[gcusello]

Hi @Adevill,

at this point, try the more strange things: are you sure that the sourcetype of your logs is "dcs_event"?

Ciao.

Giuseppe

[Adevill]

Hi @gcusello 

Yes, I'm sure, I've also double checked it and the spelling of everything.

[gcusello]

Hi @Adevill,

could you share the inputs.conf on the UF that takes those logs?

Ciao.

Giuseppe

[Adevill]

Hi @gcusello 

As requested:

[monitor://C:\Program Files (x86)\ABB Symphony Plus\Operations\History\PlantConnect.SYS\Debug\PlaCoEventImporter\Administrator_PlaCoEventImporter.log]
disabled = 0
source = dcs
sourcetype = dcs_event

[gcusello]

Hi @Adevill,

it's really strange because:

• the sourcetype on UF's inputs.conf is correct (dcs_event);
• the props.conf on HF is correct

[dcs_event]
TRANSFORMS-routing = dcs_drop, dcs_allow

• you restarted HF after props.conf modification;
• The regex in transforms.conf is correct (otherwise you discard all the events),
• events pass through the HF so they should be parsed by HF.

The only hint I can give is check again the above chain: something could be different.

Ciao.

Giuseppe

[Adevill]

@gcusello  ,thank you for all the assistance. I'll go back and check everything everything.

[gcusello]

Hi @Adevill,

I'm sorry to be not able to help you more: it's really strange I did many of these condifugrations without any problem.

Check again every step and surely there's a little particular the will solve the problem.

Ciao.

Giuseppe