topic Re: How to separate hosts/other fields into other data indexes? in Getting Data In

How to separate hosts/other fields into other data indexes?

bigfatyeastroll — Thu, 17 Sep 2020 22:45:11 GMT

I've got several data indexes (only one server) already that are separated by forwarders or listener ports. However, I have several devices that can only send to the default syslog port. How can I "pull out" events and place them into different indexes?

Is there a way in the Splunk Enterprise GUI to do this?

Re: How to separate hosts/other fields into other data indexes?

jscraig2006 — Sat, 11 Jan 2020 00:37:36 GMT

You can't. Once the data is indexed, you can't move the data to another index. Why not stand up a syslog server with a UF? Then you can send your syslog data to different directories based upon host and then send them to the peer based upon what the data type is. Network, firewall etc.

Re: How to separate hosts/other fields into other data indexes?

gcusello — Sat, 11 Jan 2020 06:39:37 GMT

Hi @bigfatyeastroll,
as @jscraig2006 said once indexed it isn't possible to change index!
I suggest you rethink your data structure: Splunk isn't a database where you separate data into tables, the correct way to identify a data flow is not the index but the sourcetype, to which all knowledge objects normally refer ( fields, tags, eventtypes).
Splunk Best Practices say that there are two / three reasons to put the data in different indices:

retention,
access rights;

in other words, you should put in the same index data with the same retention period and the same access rights, not because they are related to different things.
It is also advisable not to put data with very different volumes in the same index: e.g. I would never combine data from a stream with few events per day with data from a stream with millions of events per day.
It is also not a good thing to have too many indices because they are more complex to manage.

To return to your request, it's not possible to change the index to a data after it has been indexed, but it is possible to do it first: you need to find a way to identify it (e.g. the host of origin or a regex that identifies it) and then submit it to a transformation; for more information look at https://docs.splunk.com/Documentation/SplunkCloud/8.0.1/Data/Advancedsourcetypeoverrides where they explain how to override a main field.

To override the index you have to do something like this on your Indexers or (when present) on your Heavy Forwarders:

on transforms.conf

 [overrideindex]
 DEST_KEY =_MetaData:Index
 REGEX = your_regex
 FORMAT = my_new_index

on props.conf

 [mysourcetype]
 TRANSFORMS-index = overrideindex

Ciao.
Giuseppe

Re: How to separate hosts/other fields into other data indexes?

bigfatyeastroll — Mon, 13 Jan 2020 15:02:12 GMT

Great, thanks!

Those 2 reasons you gave are the reasons I am separating them out. One group has multiple devices and some are only able to send to default syslog (which other groups have devices like this as well).

Since Splunk is not my main job role, I was hoping there would be an "easy" GUI way.