Getting Data In

Why aren't transforms working when attempting to not index keepalive data from source or source type?

wgawhh5hbnht
Communicator

Source type is being set in inputs.conf via
/deployment-apps/Splunk_TA_microsoft-iis/local/inputs.conf
contents of inputs.conf:

    [monitor://C:\inetpub\logs\AdvancedLogs\*.log]
    index = winiislog 
    disabled = false
    #sourcetype = ms:iis:default
    sourcetype = iis

On the indexers, I have two different entries, 1 by source type & 1 by source, in props
contents of props.conf:

[iis]
TRANSFORMS-null = setnull
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3N

[source::C:\\inetpub\\...]
TRANSFORMS-null = setnull

contents of transforms.conf:

[ise_setnull]
REGEX = (^Compiled\s|^Copyright\s|^Technical\sSupport:)
DEST_KEY = queue
FORMAT = nullQueue

[setnull]
REGEX = keepalive
DEST_KEY = queue
FORMAT = nullQueue

I've pushed the cluster-bundle and did a rolling restart on the indexers, but I'm still getting keepalive data.

Any ideas?

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Assuming you've got a default-configured Windows Universal Forwarder it will already cook the data on the UF for sourcetype iis. Run splunk btool --debug props list iis on the UF to confirm, look for something like INDEXED_EXTRACTIONS = w3c. The indexers won't re-cook already cooked data, so your transforms aren't being touched.

Your best option is to copy the nullqueueing to the UFs. I'd keep a copy on the indexers in case you get uncooked data in the future for some reason.

See https://wiki.splunk.com/Community:HowIndexingWorks for some background on what happens where.

View solution in original post

gcusello
SplunkTrust
SplunkTrust

Hi wgawhh5hbnht,
some additional information:
I don't understand your regex, what do you want to exclude?
parenthesis and beginning of string should be used in different way, try with

^((Compiled\s)|(Copyright\s)|(Technical\sSupport:))

at the same time the source [source::C:\inetpub\...] should be used without double backslash

[source::C:\inetpub\...]

Bye.
Giuseppe

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Assuming you've got a default-configured Windows Universal Forwarder it will already cook the data on the UF for sourcetype iis. Run splunk btool --debug props list iis on the UF to confirm, look for something like INDEXED_EXTRACTIONS = w3c. The indexers won't re-cook already cooked data, so your transforms aren't being touched.

Your best option is to copy the nullqueueing to the UFs. I'd keep a copy on the indexers in case you get uncooked data in the future for some reason.

See https://wiki.splunk.com/Community:HowIndexingWorks for some background on what happens where.

wgawhh5hbnht
Communicator

Thank you martin_mueller, that worked!

For those that want more information as to why you would need the props & transforms on the UF instead of the indexers:
https://answers.splunk.com/answers/268335/why-is-the-sourcetype-specified-in-inputsconf-on-t.html

0 Karma

wgawhh5hbnht
Communicator

not shown, but I've tried both entries in the transforms.conf of [ise_setnull] & [setnull], neither work.

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...