Keep specific events and discard the rest -Heavy Forwarder Setup

[kevinbullock]

I am setting up a heavy forwarder to keep specific events and discard the rest. My heavy forwarder is forwarding all of the events and not discarding anything. I am guessing I am either editing the incorrect files or my modifications are incorrect. The only events I would like to keep are Fatal and Warning events

All of the documentation I have read says to update the transforms.conf and props.conf in /etc/system/local. I am on a windows machine so that directory structure does not exist. There are few different props.conf and transforms.conf. I am editing the ones in c:\Program Files\Splunk\etc\apps\search\default. Are these the correct ones? If so, then I must have a problem somewhere else.

I have updated the transforms.conf and props.conf in c:\Program Files\Splunk\etc\apps\search\default as follows:

props.conf:
[source::C:\ProgramData\Folder1\Folder2\*.sts]
TRANSFORMS-set= setnull,setparsing

transforms.conf:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \[(Fatal|Warning)\]
DEST_KEY = queue
FORMAT = indexQueue

My Sample Data looks like this:
2019/01/14 14:29:36.356 - (47) [Informational] : TEST File: SCHD_Scd.Cpp Line: 963
SCHD-S-Information (1860) Releasing Locking logs...
2019/01/14 14:29:36.231 - (47) [Informational] : TEST File: SCHD_Scd.Cpp Line: 963
SCHD-S-Information (1860) Cleaning up Locked logs...
2019/01/14 14:29:36.106 - (47) [Informational] : TEST File: SCHD_Scd.Cpp Line: 963
SCHD-S-Information (1860) 479 Loaded 225 Scheduled
2019/01/14 14:29:35.950 - (47) [Informational] : TEST File: SCHD_Scd.Cpp Line: 963
SCHD-S-Information (1860) Releasing Locking logs...
2019/01/14 14:29:35.263 - (126) [Fatal] : TEST: c:\test\1\4\s\r\h\DB_Tran.Inl Line: 83
DB-F-RoutineFail (1272) Failure occurred in Routine: [CCDbTransaction::Commit].
2019/01/14 14:29:35.263 - (126) [Fatal] : TEST: DB_Con.Cpp Line: 601
DB-F-RoutineFail (1272) Failure occurred in Routine: [CCDbConnect::Execute].
2019/01/14 14:29:35.263 - (126) [Fatal] : TEST: DB_Con.Cpp Line: 598
DB-F-GeneralFailure (1272) A General Failure Occurred In Routine [CCDbConnect::Execute]. COMMIT TRANSACTION;.

The Source files that are being monitored by the universal forwarder and sent to the heavy forwarder are like this
C:\ProgramData\Folder1\Folder2\Test1.sts
C:\ProgramData\Folder1\Folder2\Test2.sts

The universal forwarder inputs.conf has the following:
[monitor://C:\ProgramData\Folder1\Folder2\*.sts]
current_only = 1
disabled = 0
start_from = oldest
sourcetype = stslog
index = sts

Any help would be appreciated! Thank you

[kevinbullock]

Björn,
Thank you for the reply. This was all very useful information.
I tried populating C:\Program Files\Splunk\etc\apps\search\local with my settings for props.conf and transforms.conf, but that didn't work.
I ended up populating the settings in C:\Program Files\Splunk\etc\system\local for props.conf and transforms.conf. However, at first, this still didn't work.

The real problem was in Splunks documentation that I was using found here: Forwarding Data

Under the section, Keep specific events and discard the rest, I copied the this specific line for the profs.conf configuration:
TRANSFORMS-set= setnull,setparsing

The problem was that there needs to be a space inbetween the comma and "setparsing". This line should read like:
TRANSFORMS-set = setnull, setparsing

After adding the space, everything is working correctly now. I can't tell you how many hours I have spent trying to figure this out the last two days.

Again, thank you for your help! It is greatly appreciated.

[bjoernjensen]

Hi,
c:\Program Files\Splunk\etc\apps\search\default
you should "never" edit c:\Program Files\Splunk\etc\apps\search\default ... that is product release defaults, since it is a default app.

In your case you should create and edit files in c:\Program Files\Splunk\etc\apps\search\local. Splunk will "merge" the configs at runtime.
https://docs.splunk.com/Documentation/Splunk/7.2.1/Admin/Configurationfiledirectories

In order to debug your current runtime configuration it is very handy to use the btool:
https://docs.splunk.com/Documentation/Splunk/7.2.3/Troubleshooting/Usebtooltotroubleshootconfigurati...

Note: If you want to grep in Windows, use the PowerShell in the following way (example):
C:\Program Files\Splunk\bin> .\splunk.exe cmd btool outputs list --debug | Select-String -Pattern "<REGEX_PATTERN>

Configuration should be described as here:
https://docs.splunk.com/Documentation/SplunkCloud/7.1.3/Forwarding/Routeandfilterdatad#Discard_speci...

Hope that guides you a little.

Cheerz - Björn