topic Re: Data forwarded as syslog gets always indexed in Getting Data In

Data forwarded as syslog gets always indexed

FRoth — Mon, 28 Sep 2020 14:44:04 GMT

I filtered some of our Windows events before indexing. This worked perfectly.

We had the idea to send the filtered data to a remote system for archiving via syslog.

UPDATE 3

Clarification: This is what we try to accomplish:

Universal Forwarders -- in splunk format --> Filter 2 way

1. into index of Splunk Server (Indexer)

2. as syslog to --> Remote Syslog Server

Everything works find except that ALL the data that gets forwarded as syslog ALSO gets indexed and there seems to be no way to avoid this. There is no way to filter the incoming data stream an route a data stream into the indexer and another to a remote syslog server.

This is the content of the props.conf:

[source::WinEventLog...]
TRANSFORMS-filter=filterWinFw,filterEventIDs

This is the content of the transforms.conf - the old config that worked perfectly is commented out:

[filterWinFw]
REGEX = (169\.254\.|bin\\splunkd.exe)
# DEST_KEY = queue
# FORMAT = nullQueue
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslogarchive

[filterEventIDs]
REGEX=EventCode=(4656|5447|5154|5145|5156|4957|4634|4663|4658|4688|4768|4769|4689)
# DEST_KEY = queue
# FORMAT = nullQueue
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslogarchive

This is the outputs.conf

[syslog:syslogarchive]
indexAndForward = false
server = 10.1.1.12:1577

What happens as expected is the syslog forwarding. BUT - it still indexes these events which is not the desired result.
Any ideas?

UPDATE 1

I tried to change the transforms.conf to add another format that sends the events to the nullQueue - without effect:

[filterWinFw]
REGEX = (169\.254\.|bin\\splunkd.exe)
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslogarchive,send_to_null

[filterEventIDs]
REGEX=EventCode=(4656|5447|5154|5145|5156|4957|4634|4663|4658|4688|4768|4769|4689)
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslogarchive,send_to_null

[send_to_null]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

UPDATE 2

I even tried this in outputs.conf without success and tried to apply "selectiveIndexing":

[syslog]
index = false
indexAndForward = false

[tcpout]
index = false
indexAndForward = false

[syslog:syslogarchive]
index = false
indexAndForward = false
server = 10.1.1.12:1577

UPDATE 4

I even tried this:

Section in props.conf:

[source::WinEventLog...]
TRANSFORMS-sendsyslog=syslogWinFw,syslogEventIDs
TRANSFORMS-filter=filterWinFw,filterEventIDs
TRANSFORMS-index=winnativeindex

Section in transforms.conf:

[winnativeindex]
REGEX = .
FORMAT = win_native
DEST_KEY = _MetaData:Index

[filterWinFw]
REGEX = (169\.254\.|bin\\splunkd.exe)
DEST_KEY = queue
FORMAT = nullQueue

[filterEventIDs]
REGEX=EventCode=(4656|4658|4769|5156|5158|5447)
DEST_KEY = queue
FORMAT = nullQueue

[syslogWinFw]
REGEX = (169\.254\.|bin\\splunkd.exe)
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslogarchive

[syslogEventIDs]
REGEX=EventCode=(4656|4658|4769|5156|5158|5447)
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslogarchive

Result:
1. Filtered log gets indexed
2. No Syslog sending

UPDATE 5

I also tried these 3 versions - Result: Filtering = Yes, Syslog Forwarding = No.

It seems that the filters always apply before the _SYSLOG_ROUTING is done no matter what order I use.
(I checked the forwarding by removing the "filter.." statements and yes, everything gets forwarded via syslog as expected)

TRANSFORMS-send_and_filter=syslogWinFw,filterWinFw,syslogEventIDs,filterEventIDs
TRANSFORMS-index=winnativeindex

TRANSFORMS-send_and_filter=filterWinFw,syslogWinFw,filterEventIDs,syslogEventIDs
TRANSFORMS-index=winnativeindex

TRANSFORMS-send=syslogWinFw,syslogEventIDs
TRANSFORMS-xfilter=filterWinFw,filterEventIDs
TRANSFORMS-index=winnativeindex

Re: Data forwarded as syslog gets always indexed

FRoth — Mon, 28 Sep 2020 14:44:06 GMT

I found this other post, but even if I add another FORMAT like the "send_to_null" mentioned in the post, the data gets indexed.
http://answers.splunk.com/answers/34747/send-filtered-data-to-syslog-and-not-index-it

Re: Data forwarded as syslog gets always indexed

FRoth — Mon, 09 Sep 2013 15:06:29 GMT

It seems that data that get forwarded via Syslog is always indexed and there is no way to avoid this. Disappointing.

Re: Data forwarded as syslog gets always indexed

sowings — Mon, 09 Sep 2013 15:17:49 GMT

First off, you might be falling afoul of the way the parsing rules work when reassigning sourcetypes. Data coming in via syslog is assigned a sourcetype (whether at inputs.conf or from the sending side by props.conf source:: or other rules). The parsing rules (including sending to the null queue) apply once, top to bottom, left to right, for the incoming sourcetype.

So let's say you have a syslog sourcetype, and you set up some transforms.conf to recognize that "oh hey, those are really Windows Event logs, like WinEventLog:Security". Great, you've changed the sourcetype. What you might then try to do is null queue some of those by setting up a transforms.conf entry relative to the [WinEventLog:Security] sourcetype. That won't work, because those parsing rules won't fire. Only if the transforms.conf entry you have for null queueing those events is assigned to the original [syslog] sourcetype, will it work.

Second, the source of the data (your first props.conf rule) may no longer be matching the filename. Instead, it's probably something like "udp:514". I don't know specifically, since you didn't include your server's inputs.conf.

So try checking against the "only one parsing pass" rule, or maybe it's the source.

Re: Data forwarded as syslog gets always indexed

FRoth — Mon, 09 Sep 2013 15:31:42 GMT

Ok, there might be a misunderstanding. The data comes in from Splunk Universal Forwarders. I am talking about filtering this content and send the filtered content to a remote syslog server before it gets indexed.
I am going to clear this up in the initial post.

Re: Data forwarded as syslog gets always indexed

sowings — Mon, 28 Sep 2020 14:44:17 GMT

I see your "send_to_null" added as part of the FORMAT key of your transforms. That's the wrong place for it. You need to list send_to_null as its OWN transform hung off of the TRANSFORMS list on your props.conf entry for the source.

Re: Data forwarded as syslog gets always indexed

FRoth — Mon, 28 Sep 2020 14:44:20 GMT

Ok, I'll try that. But the "filterWinFw,filterEventIDs" transforms apply regular expressions to filter events from the data stream I would like to index. These lines that get filtered out should be routed to a remote syslog server and not get indexed. If I add a "send_to_null" to the "filterWinFw,filterEventIDs" statement, it would send the events to the remote syslog and then everything (else) to nullQueue. But your comment gave me another idea.

Re: Data forwarded as syslog gets always indexed

sowings — Mon, 09 Sep 2013 15:53:49 GMT

Ok, then you'll have to "sendWinFwToSyslog" as one transform, then "filterWinFw" as a second, using the same matching regex, but two different DEST_KEY / FORMAT pairs.

Re: Data forwarded as syslog gets always indexed

FRoth — Mon, 09 Sep 2013 20:09:04 GMT

Jupp - thanks - please see UPDATE 4 if everything is as expected. I tried and failed.

Re: Data forwarded as syslog gets always indexed

sowings — Mon, 28 Sep 2020 14:44:39 GMT

With update4, I believe that the TRANSFORMS are sorted lexically based upon the right
hand side of the dash. Try this instead:

[source::WinEventLog...] TRANSFORMS-filter_and_send=syslogWinFw,filterWinFw,syslogEventIDs,filterEventIDs TRANSFORMS-index=winnativeindex

Note that you'll have to restart the indexer(s) for these changes to take effect after making the edits.

Re: Data forwarded as syslog gets always indexed

yannK — Mon, 09 Sep 2013 21:07:27 GMT

A possible hack is to define an index that doesn't exist on the final destination indexer.

data as sourcetype=syslog/index=whathever
-> forwarded to heavy forwarder with syslog forwarding, and splunk forwarding to another indexer
-> on last indexer the index "whathever" doesn't exit, the events are dropped (and plenty of yellow banners appears to warn you, hilarity ensue)

Re: Data forwarded as syslog gets always indexed

FRoth — Wed, 11 Sep 2013 08:04:25 GMT

Ok, that did not work. Filters are applied but no syslog forwarding. If I remove the filter transforms, everything gets forwarded.

Re: Data forwarded as syslog gets always indexed

FRoth — Wed, 11 Sep 2013 09:10:15 GMT

Hey - this worked I tried this config

[filterWinFw]
REGEX = (169\.254\.|bin\\splunkd.exe)
DEST_KEY = _MetaData:Index
FORMAT = dummy

[filterEventIDs]
REGEX=EventCode=(4656|4658|4769|5156|5158|5447)
DEST_KEY = _MetaData:Index
FORMAT = dummy

Syslog gets Forwarded, 2. Messages do not appear in my index. Is there a way to suppress the yellow banners telling me that messages could not be send to the indexer "dummy"?

Re: Data forwarded as syslog gets always indexed

sowings — Wed, 11 Sep 2013 13:28:46 GMT

Sorry that the "usual method" didn't work out for you.

Re: Data forwarded as syslog gets always indexed

sowings — Wed, 11 Sep 2013 13:30:19 GMT

Silencing the warning messages permanently might be a bit tricky; you'd have to have a fair bit of Javascript knowledge to tweak the way the messaging module behaves. I've found, though, that once you silence the warnings once (using Manager -> User Interface -> Bulletin Messages), they stay gone, at least until the indexers or the search head restart.