topic Re: Send data via Splunk REST API to a Heavy Forwarder requires index on the HF? in Getting Data In

Send data via Splunk REST API to a Heavy Forwarder requires index on the HF?

xiaoweiwu — Fri, 17 May 2019 14:31:53 GMT

We have indexer cluster setup. We are trying to use the REST API on the Heavy Forwarder to receive data update. But i am not sure if creating index on the HF is required to make sure indexer gets the data.

I do see an earlier posting is similar to this one. Since it's an old posting, i just want to confirm If using REST API on the HF, while output.conf can point to the indexer/cluster would work, with no index stored on the HF.
https://answers.splunk.com/answers/38176/data-indexed-via-rest-api-then-forwarded-on-to-another-indexer.html?utm_source=typeahead&utm_medium=newquestion&utm_campaign=no_votes_sort_relev#comment-746203

Re: Send data via Splunk REST API to a Heavy Forwarder requires index on the HF?

starcher — Fri, 17 May 2019 18:18:38 GMT

HTTP Event collector is better choice than trying to post to the api.

Re: Send data via Splunk REST API to a Heavy Forwarder requires index on the HF?

koshyk — Fri, 17 May 2019 18:46:50 GMT

indexes.conf is NOT required at HF.
But it is very important that your HF forwards (outputs.conf) to Indexers
props.conf/tranforms.conf is required at HF and indextime changes happen at HF though it sends data to indexers

Re: Send data via Splunk REST API to a Heavy Forwarder requires index on the HF?

xiaoweiwu — Mon, 20 May 2019 13:04:01 GMT

I did look at my HF.
There is no entry in the indexes.conf at this location /opt/splunk/etc/system/default.

Here is outputs.conf in /opt/splunk/etc/system/local

[tcpout]
defaultGroup = default-autolb-group
indexAndForward = 1

[tcpout:default-autolb-group]
server = xx.xx.xx.xx:9997

[tcpout-server://xx.xx.xx.xx:9997]

Re: Send data via Splunk REST API to a Heavy Forwarder requires index on the HF?

xiaoweiwu — Mon, 20 May 2019 13:04:54 GMT

i do want to mention. I have an index on the HF. I am not sure why it is not in the indexes.conf.

Re: Send data via Splunk REST API to a Heavy Forwarder requires index on the HF?

koshyk — Mon, 20 May 2019 15:55:03 GMT

it might be one or another app contain indexes.conf
if you do a find on all apps with indexes.conf or a btool then it shows which app have created it incorrectly

/opt/splunk/bin/splunk cmd btool indexes list --debug > /tmp/indexes.btool

in your HF

Re: Send data via Splunk REST API to a Heavy Forwarder requires index on the HF?

xiaoweiwu — Wed, 30 Sep 2020 00:38:18 GMT

ok, I was able to find on the HF at this location for the indexes.conf.

/opt/splunk/etc/apps/search/local

And I've found the index in that indexes.conf file.
[myindex]
coldPath = $SPLUNK_DB/myindex/colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB/myindex/db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB/myindex/thaweddb

Re: Send data via Splunk REST API to a Heavy Forwarder requires index on the HF?

xiaoweiwu — Mon, 20 May 2019 16:55:59 GMT

I think i've gone on a tangent. All I wanted to know is for REST API, it's pushing data to this URI.
http://[Splunk IP]:8089/services/receivers/simple?index=myindex&source=sccm.py&sourcetype=web_event
If no index is required on the HF, then it will be sent to the indexer directly.

Is that correct?

Re: Send data via Splunk REST API to a Heavy Forwarder requires index on the HF?

xiaoweiwu — Tue, 21 May 2019 18:06:19 GMT

Not to ignore your comments. Why is the event collector is better? Just want to know your view point.