Drop lines from log file without modiying sourcetype

[shocko]

I'm using Splunk Enterprise 9.1 with Windows Universal Forwarders. I'm ingesting the Windows Domain Contoller netlogon.log file. The Splunk Add-on for Windows has all the parsing/extraction rules defined for me to parse netlogon.log via its sourcetype=MSAD:NT6:Netlogon definition. Now, my use case is that I only wish to retain certain lines from netlogon.log and discard all others. How can I acheive this? Is it a case of defining a new sourcetype and copying the props/transforms from this Splunk_TA_Windows or is there a way to keep using the sourcetype sourcetype=MSAD:NT6:Netlogon and discard the lines via some other mechanism that does not result in my modidying the Splunk_TA_Windows app? 

[deepakc]

You can continue to use the props/transforms from the Splunk Windows TA BUT you need to use the /local/props.conf and /local/transforms.conf

So, create a local folder within the TA and add the two files, if you change the default props and trans, they will get overwritten during upgrades in the future. 

So, configure as per below example, you will need to work out what events you want to discard, so some regex, and this is the better way, the rest will get logged into Splunk.

props.conf

[MSAD:NT6:Netlogon]
TRANSFORMS-send_to_null_events = send_null_netlogin_events

# transforms.conf

[send_null_netlogin_events]
REGEX = <YOUR REGEX FOR LINES YOU DONT WANT>
DEST_KEY = queue
FORMAT = nullQueue

The above code then needs to placed on the Indexers or Heavy forwarder - if the data is sent here first (Splunk Full Instances), so deploy the Windows TA that contains your new code. Note: The UF will not do it)

Another way is to create your own side car TA and have the code there and run it alongside the Windows TA.

[shocko]

Thanks @deepakc , good call on use of the /local folder for this. when you say:

Another way is to create your own side car TA and have the code there and run it alongside the Windows TA.

What do you mean by sidecar? Is this simply buidling a new app for the indexer?

[deepakc]

Hi @shocko  yes precisely that (the code is then in another separate custom add-on TA) that you control for custom code changes, it lives side by side with the Splunkbase TA and this is optional,  so example, App called my_windows_sidecar_ta and add local there and push out, but you need to know the app structure for this some people do it others don't,  main thing is local.

example of creating your own TA 

https://dev.splunk.com/enterprise/tutorials/quickstart_old/createyourfirstapp/        

[shocko]

Ah! I thing I get it now. So basically we leave Splunk_Windows_TA alone and then simply create a new TA with a /local/props.conf and transforms.conf that override that sourcetype. Is this understanding correct? 

[deepakc]

That's correct, the behaviour of Splunk is that any code(props/trans) in local will override the app default settings.  As long as you use the same sourcetype name that's in the Windows TA and use that in your custom TA, and they live side by side and the local will override. Note: The other Windows TA settings will still kick in but yours will override.    

[shocko]

Last question on this. The existing SplunkTA alreayd defines the info I'm interested in an event type defined in /default/eventtypes.conf. Is it possible to configure this so that only a specific event type is indexed and all others discarded?

[deepakc]

Eventtypes are for search specific events/data your interested in (quick way to get some results from data that has already been indexed. 

1. If you are only interested in some specific eventtypes, and want to discard the rest, you could copy each of the eventtypes stanzas names into the /local/eventtypes.conf and disable them, but not sure why you want to do that as many of these also use tags for future use case such as Splunk Data models etc. 

2. If you want to tune some of these, by adding your index name, then also do that into the local/eventtypes.conf

Example disable an eventype

/local/eventtypes.conf 

[windows_event_signature]
disabled = [1|0] (1 = disabled - 0 = enabled)

or tune an eventtype with my index example

/local/eventtypes.conf 

[windows_event_signature]
search = index=my_windows_index sourcetype=WinEventLog OR sourcetype=XmlWinEventLog OR sourcetype=WMI:WinEventLog:System OR sourcetype=WMI:WinEventLog:Security OR sourcetype=WMI:WinEventLog:Application OR sourcetype=wineventlog OR sourcetype=xmlwineventlog

More on eventtypes concepts 

https://docs.splunk.com/Documentation/Splunk/9.2.1/Knowledge/Abouteventtypes