topic Re: INDEX_AND_FORWARD usage in Getting Data In

INDEX_AND_FORWARD usage

gavsdavs_GR — Tue, 29 Sep 2020 13:59:07 GMT

I want to (index and) forward (to a syslog endpoint) some data that goes into a particular index on my indexer cluster.

These indexers mainly do not run inputs over and above the splunktcp://9997 listener as the data arrives at these indexers from universal forwarders and has in some cases already passed through a heavy forwarding layer, and some is direct to the indexers.

To begin with, I want to index all data and forward none (where I am now)
I want to change to indexing all data and also forwarding all data.
Then I want to index some data and forward all data (reducing what is indexed to a list of regex matches).

I don't quite know how to make sense of the INDEX_AND_FORWARD routing keys, they state I need to declare "_INDEX_AND_FORWARD_ROUTING=local" for my inputs, but in most cases, my inputs are not local.

Also, are there any good examples of how to use INDEX_AND_FORWARD based on props/transforms matches ?

Re: INDEX_AND_FORWARD usage

adonio — Tue, 09 May 2017 01:14:23 GMT

hello there,
quick search gives me more than 10 answers in this portal, here are 3 as an appetizer:
https://answers.splunk.com/answers/412969/how-do-i-configure-my-heavy-forwarder-to-filter-an.html
https://answers.splunk.com/answers/474297/how-to-route-and-filter-data-on-the-heavy-forwarde.html
https://answers.splunk.com/answers/26273/how-to-selectively-index-and-forward-with-filtering.html
and the official splunk docs for desert:
http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad
hope it helps

Re: INDEX_AND_FORWARD usage

esix_splunk — Tue, 09 May 2017 07:20:54 GMT

Index and forward will not forward data in syslog format. It will forward it as Splunk Cooked data, meaning that only a Splunk HF / Indexer on the other end can process the feed.

If you truly want to forward to a 3rd party receiver, I'd look at the CEF App for Splunk (https://splunkbase.splunk.com/app/1847/). There is a custom command included that you can use to format and send messages in syslog format out of Splunk to a 3rd party receiver.

The other option is here : http://docs.splunk.com/Documentation/Splunk/6.5.3/Forwarding/Forwarddatatothird-partysystemsd

Re: INDEX_AND_FORWARD usage

gavsdavs_GR — Tue, 29 Sep 2020 13:59:46 GMT

One of the reasons we want to do this is that we don't want to index all the data that we've brought in using our UFs. As a result, I don't think the CEF app can help us.

We want to send some of a particular sourcetype to a syslog destination.
we want to
First - Index all of it AND forward all of it.
Second - Index a small amount of it AND forward all of it

I think this means we need some content based filtering and I currently have this setup:
outputs:
{code}
[syslog:Send_to_syslog_dest]
type = tcp
server = syslog_server:10518
timestampformat = %Y-%m-%dT%H:%M:%S.%3N%z

[indexAndForward]
index = true
selectiveIndexing = true
{code}

props
{code}
[mysourcetype]
TRANSFORMS-filtering = 1-forward-all-data,6-Index-bits-of-it
{code}

transforms
{code}
[1-forward-all-data]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = Send_to_syslog_dest

[6-Index-bits-of-it]
DEST_KEY = _INDEX_AND_FORWARD_ROUTING
FORMAT = local
REGEX = (?msi)\"?(some|regex|matches|for|data)\"?
{code}

I've heard two things which make me question whether to continue on this tack.
1. That I shouldn't be trying to do this on an indexer, I should only be trying to do this on a 'heavy forwarder'
2. That I can only do this with splunktcp type 'cooked' data - as opposed to syslog format. In some cases (we have a HF layer in place) the data is arriving with us already parsed.

Having said that, this sort of seems to be working (I haven't done exhaustive testing).

Can someone answer the 2 questions at the end ?

Thanks