Why am I unable to filter logs using props.conf & transforms.com?

[alexspunkshell]

I am using universal forwarder. Created app named - cisco-ios. Then inputs.conf , props.conf & transforms.conf inside the app.

In inputs.conf i am monitoring syslogs. I can able to receive syslog in console.

I have too much logs. I want to filter logs containg "Domain" keyword inside raw logs.

Props.conf

[source::.../syslog/cisco/ios/cisco-ios.log(.\d+)?]
TRANSFORMS-null= setnull

Transforms.conf

[setnull]
REGEX = \[^%.*:$\]
DEST_KEY = queue
FORMAT = nullQueue

But i am unable to filter logs containing keyword "Domain".

Please help me here.

[shivanshu1593]

Hello,

Props and transforms don't work on a Universal forwarder, since these features are only supported by a Splunk enterprise version (Heavy forwarder, Indexers, Search Heads). Hence you're not able to drop the data. You can do either of the following:

1. Use blacklist. Not exactly 100% sure if it'll work here, as I've only used it with files and event codes. In your inputs.conf, edit and paste the following.

blacklist = your_field_name_which_contains_domain = (.+Domain.+)

If you can give some sample data, maybe I can help with writing more accurate regex. Or you can play around here and build your own regex: https://regex101.com/r/8rBc6U/1 . Once you know that your regex will filter out all the fields containing the keyword "Domain", use:

blacklist = <your_custom_regex>

2. Move your props and transforms to the Indexers, and have them drop the data.

Let me know if it helps.

Thank you,

S

[to4kawa]

https://docs.splunk.com/Documentation/Splunk/8.0.5/Data/Monitorfilesanddirectorieswithinputs.conf

@shivanshu1593 
blacklist can make filter the name of log. not log value without Windows Event log.
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf#Windows_Event_Log_Monitor

@alexspunkshell 
where is your indexer? 
UF → IX → SH
props.conf works on IX.
transforms.conf works on SH.

check index=_internal host=<your UF host name>
https://qiita.com/odorusatoshi/items/5a703b9befc253ab7deb(japanese)

https://docs.splunk.com/Documentation/Splunk/8.0.5/Indexer/Howindexingworks
https://docs.splunk.com/Documentation/Splunk/8.0.5/Admin/Configurationparametersandthedatapipeline

I does not have proper education in this category, which certification should I have? 

[shivanshu1593]

@to4kawa, I agree. Hence I wrote I'm not sure if it'll work. It shouldn't, as parsing doesn't happen on a UF.

Just FYI: Transforms.conf doesn't necessarily work on SH. If you are doing search time extractions, then you put your transforms and props on search head. In this case, we're doing Index time extractions, as we don't want to ingest a certain set of logs.  Hence they'll go on either on a Heavy Forwarder, or on an Indexer. If there are multiple indexers, then via Cluster master (My second suggestion on the previous answer).

@alexspunkshell,

Please put the props and transforms on your Indexers, and set the regex accurately, as everything that'll match it, will not be indexed. 

Hope this helps,

[isoutamo]

@alexspunkshell as I said earlier put the both (props.conf and transform.conf) to the first full Splunk Enterprise instance (HF or indexer) on the lane from UF to indexers. 

[to4kawa · Accepted solution]

[setnull]
REGEX = Domain
DEST_KEY = queue
FORMAT = nullQueue

your regex isn't related "Domain"

\[^%.*:$\]

[alexspunkshell]

@to4kawa  Thanks for your reply

Changed transforms.conf 

[setnull]
REGEX = Domain
DEST_KEY = queue
FORMAT = nullQueue

Still unable to filter logs.

In raw logs, i want to filter logs contain "Domain" Please help me.

[isoutamo]

Hi

You are saying that you are using UF, so I suppose that you have installed that app to UF and try to do routing part on UF. Unfortunately this needs full splunk enterprise instance. So you must install that app to HF or Indexers and do that event dropping there not in UF.

• https://docs.splunk.com/Documentation/Splunk/8.0.5/Forwarding/Routeandfilterdatad
• https://community.splunk.com/t5/Getting-Data-In/Which-properties-are-available-for-a-Universal-Forwa...
• https://wiki.splunk.com/Community:HowIndexingWorks

r. Ismo

[alexspunkshell]

@isoutamo 

Thanks for your reply.

In UF i am trying to filter logs through props.conf and transforms.conf

Props.conf 

[source::.../syslog/cisco/ios/(.\d+)?]
TRANSFORMS-null= setnull

Transforms.conf

[setnull]
REGEX = Domain
DEST_KEY = queue
FORMAT = nullQueue

In raw logs, i am trying to filter "Domain". But I am unable to filter it.

Could you please help me.

[isoutamo]

Hi

as I pointed earlier you couldn't filter on UF. You must do it on indexer or in HF, if you have heavy forwarder(s) between UF and IDX.

So you need to install that app also to HF and/or Indexers which one is nearest UFs. Basically it could be wise to send events directly to this HF/IDX instead of UF.

r. Ismo