Getting Data In

Data forwarded as syslog gets always indexed

FRoth
Contributor

I filtered some of our Windows events before indexing. This worked perfectly.

We had the idea to send the filtered data to a remote system for archiving via syslog.

UPDATE 3

Clarification: This is what we try to accomplish:

Universal Forwarders -- in splunk format --> Filter 2 way

1. into index of Splunk Server (Indexer)

2. as syslog to --> Remote Syslog Server

Everything works find except that ALL the data that gets forwarded as syslog ALSO gets indexed and there seems to be no way to avoid this. There is no way to filter the incoming data stream an route a data stream into the indexer and another to a remote syslog server.

This is the content of the props.conf:

[source::WinEventLog...]
TRANSFORMS-filter=filterWinFw,filterEventIDs

This is the content of the transforms.conf - the old config that worked perfectly is commented out:

[filterWinFw]
REGEX = (169\.254\.|bin\\splunkd.exe)
# DEST_KEY = queue
# FORMAT = nullQueue
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslogarchive

[filterEventIDs]
REGEX=EventCode=(4656|5447|5154|5145|5156|4957|4634|4663|4658|4688|4768|4769|4689)
# DEST_KEY = queue
# FORMAT = nullQueue
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslogarchive

This is the outputs.conf

[syslog:syslogarchive]
indexAndForward = false
server = 10.1.1.12:1577

What happens as expected is the syslog forwarding. BUT - it still indexes these events which is not the desired result.
Any ideas?

UPDATE 1

I tried to change the transforms.conf to add another format that sends the events to the nullQueue - without effect:

[filterWinFw]
REGEX = (169\.254\.|bin\\splunkd.exe)
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslogarchive,send_to_null

[filterEventIDs]
REGEX=EventCode=(4656|5447|5154|5145|5156|4957|4634|4663|4658|4688|4768|4769|4689)
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslogarchive,send_to_null

[send_to_null]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

UPDATE 2

I even tried this in outputs.conf without success and tried to apply "selectiveIndexing":

[syslog]
index = false
indexAndForward = false

[tcpout]
index = false
indexAndForward = false

[syslog:syslogarchive]
index = false
indexAndForward = false
server = 10.1.1.12:1577

UPDATE 4

I even tried this:

Section in props.conf:

[source::WinEventLog...]
TRANSFORMS-sendsyslog=syslogWinFw,syslogEventIDs
TRANSFORMS-filter=filterWinFw,filterEventIDs
TRANSFORMS-index=winnativeindex

Section in transforms.conf:

[winnativeindex]
REGEX = .
FORMAT = win_native
DEST_KEY = _MetaData:Index

[filterWinFw]
REGEX = (169\.254\.|bin\\splunkd.exe)
DEST_KEY = queue
FORMAT = nullQueue

[filterEventIDs]
REGEX=EventCode=(4656|4658|4769|5156|5158|5447)
DEST_KEY = queue
FORMAT = nullQueue

[syslogWinFw]
REGEX = (169\.254\.|bin\\splunkd.exe)
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslogarchive

[syslogEventIDs]
REGEX=EventCode=(4656|4658|4769|5156|5158|5447)
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslogarchive

Result:
1. Filtered log gets indexed
2. No Syslog sending

UPDATE 5

I also tried these 3 versions - Result: Filtering = Yes, Syslog Forwarding = No.

It seems that the filters always apply before the _SYSLOG_ROUTING is done no matter what order I use.
(I checked the forwarding by removing the "filter.." statements and yes, everything gets forwarded via syslog as expected)

TRANSFORMS-send_and_filter=syslogWinFw,filterWinFw,syslogEventIDs,filterEventIDs
TRANSFORMS-index=winnativeindex

TRANSFORMS-send_and_filter=filterWinFw,syslogWinFw,filterEventIDs,syslogEventIDs
TRANSFORMS-index=winnativeindex

TRANSFORMS-send=syslogWinFw,syslogEventIDs
TRANSFORMS-xfilter=filterWinFw,filterEventIDs
TRANSFORMS-index=winnativeindex
0 Karma
1 Solution

yannK
Splunk Employee
Splunk Employee

A possible hack is to define an index that doesn't exist on the final destination indexer.

data as sourcetype=syslog/index=whathever
-> forwarded to heavy forwarder with syslog forwarding, and splunk forwarding to another indexer
-> on last indexer the index "whathever" doesn't exit, the events are dropped (and plenty of yellow banners appears to warn you, hilarity ensue)

View solution in original post

yannK
Splunk Employee
Splunk Employee

A possible hack is to define an index that doesn't exist on the final destination indexer.

data as sourcetype=syslog/index=whathever
-> forwarded to heavy forwarder with syslog forwarding, and splunk forwarding to another indexer
-> on last indexer the index "whathever" doesn't exit, the events are dropped (and plenty of yellow banners appears to warn you, hilarity ensue)

sowings
Splunk Employee
Splunk Employee

Silencing the warning messages permanently might be a bit tricky; you'd have to have a fair bit of Javascript knowledge to tweak the way the messaging module behaves. I've found, though, that once you silence the warnings once (using Manager -> User Interface -> Bulletin Messages), they stay gone, at least until the indexers or the search head restart.

0 Karma

FRoth
Contributor

Hey - this worked I tried this config

[filterWinFw]
REGEX = (169\.254\.|bin\\splunkd.exe)
DEST_KEY = _MetaData:Index
FORMAT = dummy

[filterEventIDs]
REGEX=EventCode=(4656|4658|4769|5156|5158|5447)
DEST_KEY = _MetaData:Index
FORMAT = dummy
  1. Syslog gets Forwarded, 2. Messages do not appear in my index. Is there a way to suppress the yellow banners telling me that messages could not be send to the indexer "dummy"?
0 Karma

sowings
Splunk Employee
Splunk Employee

First off, you might be falling afoul of the way the parsing rules work when reassigning sourcetypes. Data coming in via syslog is assigned a sourcetype (whether at inputs.conf or from the sending side by props.conf source:: or other rules). The parsing rules (including sending to the null queue) apply once, top to bottom, left to right, for the incoming sourcetype.

So let's say you have a syslog sourcetype, and you set up some transforms.conf to recognize that "oh hey, those are really Windows Event logs, like WinEventLog:Security". Great, you've changed the sourcetype. What you might then try to do is null queue some of those by setting up a transforms.conf entry relative to the [WinEventLog:Security] sourcetype. That won't work, because those parsing rules won't fire. Only if the transforms.conf entry you have for null queueing those events is assigned to the original [syslog] sourcetype, will it work.

Second, the source of the data (your first props.conf rule) may no longer be matching the filename. Instead, it's probably something like "udp:514". I don't know specifically, since you didn't include your server's inputs.conf.

So try checking against the "only one parsing pass" rule, or maybe it's the source.

0 Karma

sowings
Splunk Employee
Splunk Employee

Sorry that the "usual method" didn't work out for you.

0 Karma

FRoth
Contributor

Ok, that did not work. Filters are applied but no syslog forwarding. If I remove the filter transforms, everything gets forwarded.

0 Karma

sowings
Splunk Employee
Splunk Employee

With update4, I believe that the TRANSFORMS are sorted lexically based upon the right
hand side of the dash. Try this instead:


[source::WinEventLog...]
TRANSFORMS-filter_and_send=syslogWinFw,filterWinFw,syslogEventIDs,filterEventIDs
TRANSFORMS-index=winnativeindex

Note that you'll have to restart the indexer(s) for these changes to take effect after making the edits.

0 Karma

FRoth
Contributor

Jupp - thanks - please see UPDATE 4 if everything is as expected. I tried and failed.

0 Karma

sowings
Splunk Employee
Splunk Employee

Ok, then you'll have to "sendWinFwToSyslog" as one transform, then "filterWinFw" as a second, using the same matching regex, but two different DEST_KEY / FORMAT pairs.

FRoth
Contributor

Ok, I'll try that. But the "filterWinFw,filterEventIDs" transforms apply regular expressions to filter events from the data stream I would like to index. These lines that get filtered out should be routed to a remote syslog server and not get indexed. If I add a "send_to_null" to the "filterWinFw,filterEventIDs" statement, it would send the events to the remote syslog and then everything (else) to nullQueue. But your comment gave me another idea.

0 Karma

sowings
Splunk Employee
Splunk Employee

I see your "send_to_null" added as part of the FORMAT key of your transforms. That's the wrong place for it. You need to list send_to_null as its OWN transform hung off of the TRANSFORMS list on your props.conf entry for the source.

0 Karma

FRoth
Contributor

Ok, there might be a misunderstanding. The data comes in from Splunk Universal Forwarders. I am talking about filtering this content and send the filtered content to a remote syslog server before it gets indexed.
I am going to clear this up in the initial post.

0 Karma

FRoth
Contributor

It seems that data that get forwarded via Syslog is always indexed and there is no way to avoid this. Disappointing.

0 Karma

FRoth
Contributor

I found this other post, but even if I add another FORMAT like the "send_to_null" mentioned in the post, the data gets indexed.
http://answers.splunk.com/answers/34747/send-filtered-data-to-syslog-and-not-index-it

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...