Getting Data In

How to separate hosts/other fields into other data indexes?

bigfatyeastroll
Path Finder

I've got several data indexes (only one server) already that are separated by forwarders or listener ports. However, I have several devices that can only send to the default syslog port. How can I "pull out" events and place them into different indexes?

Is there a way in the Splunk Enterprise GUI to do this?

Labels (1)
0 Karma
1 Solution

gcusello
Legend

Hi @bigfatyeastroll,
as @jscraig2006 said once indexed it isn't possible to change index!
I suggest you rethink your data structure: Splunk isn't a database where you separate data into tables, the correct way to identify a data flow is not the index but the sourcetype, to which all knowledge objects normally refer ( fields, tags, eventtypes).
Splunk Best Practices say that there are two / three reasons to put the data in different indices:

  • retention,
  • access rights;

in other words, you should put in the same index data with the same retention period and the same access rights, not because they are related to different things.
It is also advisable not to put data with very different volumes in the same index: e.g. I would never combine data from a stream with few events per day with data from a stream with millions of events per day.
It is also not a good thing to have too many indices because they are more complex to manage.

To return to your request, it's not possible to change the index to a data after it has been indexed, but it is possible to do it first: you need to find a way to identify it (e.g. the host of origin or a regex that identifies it) and then submit it to a transformation; for more information look at https://docs.splunk.com/Documentation/SplunkCloud/8.0.1/Data/Advancedsourcetypeoverrides where they explain how to override a main field.

To override the index you have to do something like this on your Indexers or (when present) on your Heavy Forwarders:

on transforms.conf

 [overrideindex]
 DEST_KEY =_MetaData:Index
 REGEX = your_regex
 FORMAT = my_new_index

on props.conf

 [mysourcetype]
 TRANSFORMS-index = overrideindex

Ciao.
Giuseppe

View solution in original post

gcusello
Legend

Hi @bigfatyeastroll,
as @jscraig2006 said once indexed it isn't possible to change index!
I suggest you rethink your data structure: Splunk isn't a database where you separate data into tables, the correct way to identify a data flow is not the index but the sourcetype, to which all knowledge objects normally refer ( fields, tags, eventtypes).
Splunk Best Practices say that there are two / three reasons to put the data in different indices:

  • retention,
  • access rights;

in other words, you should put in the same index data with the same retention period and the same access rights, not because they are related to different things.
It is also advisable not to put data with very different volumes in the same index: e.g. I would never combine data from a stream with few events per day with data from a stream with millions of events per day.
It is also not a good thing to have too many indices because they are more complex to manage.

To return to your request, it's not possible to change the index to a data after it has been indexed, but it is possible to do it first: you need to find a way to identify it (e.g. the host of origin or a regex that identifies it) and then submit it to a transformation; for more information look at https://docs.splunk.com/Documentation/SplunkCloud/8.0.1/Data/Advancedsourcetypeoverrides where they explain how to override a main field.

To override the index you have to do something like this on your Indexers or (when present) on your Heavy Forwarders:

on transforms.conf

 [overrideindex]
 DEST_KEY =_MetaData:Index
 REGEX = your_regex
 FORMAT = my_new_index

on props.conf

 [mysourcetype]
 TRANSFORMS-index = overrideindex

Ciao.
Giuseppe

bigfatyeastroll
Path Finder

Great, thanks!

Those 2 reasons you gave are the reasons I am separating them out. One group has multiple devices and some are only able to send to default syslog (which other groups have devices like this as well).

Since Splunk is not my main job role, I was hoping there would be an "easy" GUI way.

0 Karma

jscraig2006
Communicator

You can't. Once the data is indexed, you can't move the data to another index. Why not stand up a syslog server with a UF? Then you can send your syslog data to different directories based upon host and then send them to the peer based upon what the data type is. Network, firewall etc.

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...