Reporting

What is recommended on moving select data in the default main index to a specific separate index?

sat94541
Communicator

When we first stood up some of our systems the events ended up in the main index instead of the windows, firewall and syslog indexes. We have since addressed this issue, but now we have Windows, Linux and firewall entries that we would like to export from main and move to the correct indexes.

The last time this happened (it has been a few years), I was able to perform a search using the splunk search capability that allowed us to save the content to a file in a raw format and then reimport the data. I did this with the help of support while working for another customer. Can someone help me do this again.

1 Solution

rbal_splunk
Splunk Employee
Splunk Employee

Splunk uses “main” as default index, which mean when index is not defined for data source it will index data in main.

It is common to see people miss it in early days of deployment and later it need some work to separate the data.

We do have option export data and later import as documented in links below, but some time export and import poses challenges.

http://answers.splunk.com/answers/25174/how-to-export-import-events-from-indexes.html
http://blogs.splunk.com/2013/09/15/exporting-large-results-sets-to-csv/

Here is an alternate that I like

1) Determine the buckets that hold your data using the search

| eval cd=_cd | eval bkt= _bkt | table cd bkt index splunk_server _raw

2) After that refine the above search to get the list of buckets that hold the required data

| eval cd=_cd | eval bkt= _bkt | table cd bkt index splunk_server _raw | dedup bkt | table bkt

3) Now let’s say your desire is to move the date with sourcetype=iis from data from index=main to index==iis , my suggestion will to move old data into index "iisold".

4) So, create new index "wineventold”

5) Now copy the list of above buckets from (2) from the $SPLUNK_DB/splunk/defaultdb/db or $SPLUNK_DB/splunk/defaultdb/colddb to $SPLUNK_DB/iisold/wineventold/db

6) Once the buckets are copied to new index you can check the data using (index=iisold will). As a next steps remove from (index=main index=iisold |delete ) . For delete refer link http://docs.splunk.com/Documentation/Splunk/6.1.4/SearchReference/Delete

7)In addition " index=iisold " might also have data from some other sourctype , you will need to delete that data using something like (index=main sourcetype= | delete)

😎 Now your old data sit on index= iisold and new data index=iis, to search this entire data you will need to have (index=iis OR index=iisold )

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

You may be thinking of the exporttool and importtool commands. These are only necessary if you're moving some of the data from main. If you're moving all data in main, you can simply copy the buckets/folders from main to the new index (while Splunk is stopped, and not taking into account if you're using clustering).

0 Karma

rbal_splunk
Splunk Employee
Splunk Employee

Splunk uses “main” as default index, which mean when index is not defined for data source it will index data in main.

It is common to see people miss it in early days of deployment and later it need some work to separate the data.

We do have option export data and later import as documented in links below, but some time export and import poses challenges.

http://answers.splunk.com/answers/25174/how-to-export-import-events-from-indexes.html
http://blogs.splunk.com/2013/09/15/exporting-large-results-sets-to-csv/

Here is an alternate that I like

1) Determine the buckets that hold your data using the search

| eval cd=_cd | eval bkt= _bkt | table cd bkt index splunk_server _raw

2) After that refine the above search to get the list of buckets that hold the required data

| eval cd=_cd | eval bkt= _bkt | table cd bkt index splunk_server _raw | dedup bkt | table bkt

3) Now let’s say your desire is to move the date with sourcetype=iis from data from index=main to index==iis , my suggestion will to move old data into index "iisold".

4) So, create new index "wineventold”

5) Now copy the list of above buckets from (2) from the $SPLUNK_DB/splunk/defaultdb/db or $SPLUNK_DB/splunk/defaultdb/colddb to $SPLUNK_DB/iisold/wineventold/db

6) Once the buckets are copied to new index you can check the data using (index=iisold will). As a next steps remove from (index=main index=iisold |delete ) . For delete refer link http://docs.splunk.com/Documentation/Splunk/6.1.4/SearchReference/Delete

7)In addition " index=iisold " might also have data from some other sourctype , you will need to delete that data using something like (index=main sourcetype= | delete)

😎 Now your old data sit on index= iisold and new data index=iis, to search this entire data you will need to have (index=iis OR index=iisold )

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...