topic Re: How to stop indexing forwarded data from heavy forwarder that indexes locally in Getting Data In

How to stop indexing forwarded data from heavy forwarder that indexes locally

damindragunatil — Mon, 01 Jul 2019 11:45:19 GMT

Reading from article : Does data indexed and forwarded from a heavy forwarder to indexer would charge twice?

Any indexed forwarded events from a Heavy forwarded are NOT licensed twice.

When Indexing and forwarding from a Heavy Forwarder, the licensing is only used at the Heavy Forwarder, since indexed Data sent to the Indexer, doesn't go through the Parsing queue (as well as the Aggregator and Typing queues).

I have setup the following on my Heavy Forwarder:

outputs.conf:

defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = rdbrsdem03.ref.clp7.local:9997
indexAndForward=true

props.conf

[source::tcp:9999]
BREAK_ONLY_BEFORE=^CEF\:0\|

So on my heavy forwarder, I am sending indexed data to my indexer (rdbrsdem03), and it also filters all events that start with CEF:0|

When I check licensing it seems as if the events ARE being indexed on both the Heavy Forwarder and Indexer.

Can someone provide me with a search possibly using the 'summary' index that proves the events are only being index at the Heavy Forwarder, please?

I have a developer license at the moment so would like to prove that events that need to be indexed at the Heavy Forwarder (due to local users in a remote site being able to search events of their local hardware events) and then not being reindexed (in effect doubling licensing costs) on the Indexer.

Hope this all makes sense, please let me know if there is anything further you may need.

kind regards

Damindra

Re: How to stop indexing forwarded data from heavy forwarder that indexes locally

richgalloway — Mon, 01 Jul 2019 12:36:20 GMT

Where did you read that index-and-forward does not count twice against your license? I believe that's incorrect, but would like to see your source.

Re: How to stop indexing forwarded data from heavy forwarder that indexes locally

damindragunatil — Mon, 01 Jul 2019 15:28:53 GMT

Hiya, the source of the answer was here on Splunk Answers

https://answers.splunk.com/answers/337523/does-data-indexed-and-forwarded-from-a-heavy-forwa.html

kind regards

Damindra

Re: How to stop indexing forwarded data from heavy forwarder that indexes locally

richgalloway — Mon, 01 Jul 2019 18:24:44 GMT

Thanks for the citation. That answer has since changed.

Information on Answers is not official and not always definitive. See this answer: https://answers.splunk.com/answers/506909/heavy-forwarder-as-indexer-and-license-usage.html
I'm struggling to find this mentioned in official Splunk docs.

Re: How to stop indexing forwarded data from heavy forwarder that indexes locally

skalliger — Mon, 01 Jul 2019 19:19:19 GMT

We recently had this discussion on the Slack usergroups. A heavy forwarder doing indexing is an *indexer. * License usage gets applied when events get written to disk. This means, when you index twice, your license gets hit twice also.

Skalli

Re: How to stop indexing forwarded data from heavy forwarder that indexes locally

woodcock — Sat, 06 Jul 2019 03:56:05 GMT

You have no configurations that "filter". The BREAK_ONLY_BEFORE=^CEF\:0\| is a (poorly-performing) LINE_BREAKING configuration. Even so, I am unclear on your goal. Please fill out this chart:

| NODE|  IDX?  |  FWD?  |
+-----+--------+--------+
|  HF | YES/NO | YES/NO |
| IDX | YES/NO |   N/A  |

Re: How to stop indexing forwarded data from heavy forwarder that indexes locally

damindragunatil — Sat, 06 Jul 2019 11:19:01 GMT

| NODE| IDX? | FWD? |
2. +-----+--------+--------+
3. | HF | YES/| YES|
4. | IDX | YES/ | N/A |

Hope this makes sense, the reason is there needs to be local searching on the HF.

What would you advise in regards to the LINE_BREAKING?

thanks