Getting Data In

forward index to new index

MasteringIT
Explorer

I inherited an old splunk environment where all data was indexed into the main index. I have setup a new environment with multiple indexes and some parsing rules on a heavy forwarder (These configs work perfectly with the universal forwarders I have deployed). 

How would I forward the data from the original main index, into the heavy forwarder for redistribution into the new indexes?

Labels (1)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @MasteringIT,

you could extract all the data with a simple search "index=main" and export raw data, then you can index them again in the correct indexes.

Otherwise, you could create some eventtypes like the following

index=main OR index=index1

and use them instad index=index1.

In this way you continue to use data in main index until they exceed the retention time and then they are discarded, in the meantime, new logs are stored in the correct indexes.

I hint this approach because it's a good practice to use eventtypes instead index definition in dashboards, so you can separate index definitions from the searches in dashboards.

Ciao.

Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @MasteringIT,

you could extract all the data with a simple search "index=main" and export raw data, then you can index them again in the correct indexes.

Otherwise, you could create some eventtypes like the following

index=main OR index=index1

and use them instad index=index1.

In this way you continue to use data in main index until they exceed the retention time and then they are discarded, in the meantime, new logs are stored in the correct indexes.

I hint this approach because it's a good practice to use eventtypes instead index definition in dashboards, so you can separate index definitions from the searches in dashboards.

Ciao.

Giuseppe

0 Karma

MasteringIT
Explorer

Talking to my Cyber team, they would like the first option more. 

I apologize for my ignorance, I have only been working on Splunk for the last month, how would I go about importing the raw data through my Heavy Forwarder? Most of the events I am parsing through are various Linux logs or Windows Event logs. 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @MasteringIT,

you have to extract the raw logs divided by sourcetype and host, then you have to manually load them, indicating for each index, sourcetype and host, or put them in a folder structure creating specific inputs for each folder.

It isn't a quick solution, for this reason I hinted the second solution!

If you have few hosts and sourcetypes you can do, otherwise, it's better the other!

Ciao.

Giuseppe

Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...