topic Re: Custom index for forwarded data in Splunk Search

Custom index for forwarded data

eholz1 — Wed, 30 Sep 2020 00:50:40 GMT

Hello all,
I have a working universal forwarder that happily sends data to my Enterprise indexer.
The data shows up under the forwarder's hostname on the indexer.
I would like to have a custom index for the data that comes from the Universal forwarder (my_fwd_server.net)
I can search the data by entering the hostname in the search field: host="my_fwd_server.net

I would like to be able to create a search string like: host="my_fwd_server.net" index="fwd_index", etc.
Is this possible or is this unnecessary for the data coming from the forwarder since I know the hostname?

Thanks for outstanding product,

eholz1

Re: Custom index for forwarded data

somesoni2 — Mon, 10 Jun 2019 20:39:44 GMT

You want to create a new index for each forwarder server OR just a new index which will hold data from all forwarders? Which index your data is going to right now?

Read this to understand why people generally create multiple indexes. If it's just for having a separate search for each forwarder, then I don't think you need a new index for each forwarder. The field host is a metadata and can uniquely identify data coming from that host (or data that has host field set your forwarder's name)
https://docs.splunk.com/Documentation/Splunk/7.3.0/Indexer/Setupmultipleindexes#Why_have_multiple_indexes.3F

Re: Custom index for forwarded data

eholz1 — Tue, 11 Jun 2019 15:35:44 GMT

Hello somesoni2,
Thank you for the reply - much appreciated.

I have only one forwarder. currently the data is going into the "main" index. I just thought that it would be a
good idea or practice to put the data from the forwarder in a different index.

As always, I am open to suggestions

eholz

Re: Custom index for forwarded data

eholz1 — Tue, 11 Jun 2019 15:46:51 GMT

Hello Again,
just modified the inputs.conf file on the forwarder to see if it will use the
custom index I created on the indexer.
thanks

Re: Custom index for forwarded data

DavidHourani — Tue, 11 Jun 2019 16:21:23 GMT

Hi @eholz,

Yes by all means split your data into multiple indexes. Also, make sure you have a different sourcetype for the various types of data you're sending.

You want to keep your data well separated for better scalability, easier access control management and performance. If you mix everything up into one index then you will just end up slowing down all your searches because Splunk would have to read everything to find the relevant information you're looking for in your queries.

Before defining a new target index on your forwarder in inputs.conf, make sure you create it first on your indexer as follows :
https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Setupmultipleindexes#Create_events_indexes_2

Let me know if you need more help.

Cheers,
David

Re: Custom index for forwarded data

eholz1 — Tue, 11 Jun 2019 16:32:02 GMT

Hello David,

Correct, and I did just as you suggest, and it even works!!

Thanks Again,

Eholz1