topic Re: Index internal logs locally and forward all other logs in Getting Data In

Index internal logs locally and forward all other logs

k31453 — Thu, 26 Nov 2020 04:01:36 GMT

As title suggest, i want to index internal logs only and forwards all other logs to forwarders or idxs.

Here is the setup :

I have one cluster and three indexes setup seperately outside cluster.
Cluster has CM, SH and three indexers.
Those Three indexers i want to use as Heavy forwarder to send all logs out to external indexes

Following is default output.conf:

[tcpout] maxQueueSize = auto forwardedindex.0.whitelist = .* forwardedindex.1.blacklist = _.* forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup) forwardedindex.filter.disable = false indexAndForward = false

Here is what I have done outputs.conf

[tcpout] defaultGroup=noforward disabled=false [indexAndForward] index=true selectiveIndexing=true [tcpout:forwarders] server:<forwarders>:9997

Below is my props.conf

[default] TRANSFORMS-forwardit = forwardit [host::*.foo.splunk.com] TRANSFORMS-routing = indexing

Below is transforms.conf

[forwardit] REGEX = . DEST_KEY = _TCP_ROUTING FORMAT = forwarders [indexing] REGEX = . DEST_KEY = _INDEX_AND_FORWARD_ROUTING FORMAT = local

Essentially all internal indexes should stay within cluster indexes but rest of index or logs forwarded to external indexes.

Re: Index internal logs locally and forward other logs

thambisetty — Wed, 25 Nov 2020 06:06:06 GMT

@k31453

The Splunk Doc is very much detailed on the question you have asked. check it out using below link.

https://docs.splunk.com/Documentation/Splunk/8.1.0/Forwarding/Routeandfilterdatad#Perform_selective_indexing_and_forwarding

Re: Index internal logs locally and forward other logs

k31453 — Wed, 25 Nov 2020 23:57:34 GMT

Hi, if the intention is to index all internal indexes, i have set _INDEX_AND_FORWARD_ROUTING and

_TCP_ROUTING which can cause the issue.

Re: Index internal logs locally and forward other logs

k31453 — Thu, 26 Nov 2020 00:21:57 GMT

For me by default i want to forward new indexes created and internal indexes has to be indexed locally. My thoughts is , setup tcpgroup for forwarders and in outputs.conf and inputs.conf i should modify but not sure how.

Re: Index internal logs locally and forward all other logs

thambisetty — Thu, 26 Nov 2020 06:57:49 GMT

@k31453

I believe you are looking for below: Note: you can only index _internal logs using this method.

https://docs.splunk.com/Documentation/Splunk/8.1.0/Forwarding/Routeandfilterdatad#Selective_indexing_and_internal_logs

Re: Index internal logs locally and forward all other logs

k31453 — Fri, 27 Nov 2020 00:48:15 GMT

Well. This tells me i have to use inputs.conf to ensure routing. By default I want to forward logs. But if i see internal logs i will index it and not forward it. This basically is telling me i have to put _INDEX_AND_FORWARD_ROUTING on all internal inputs.conf this can cause the issue.