Reporting

What is recommended on moving select data in the default main index to a specific separate index?

Communicator

When we first stood up some of our systems the events ended up in the main index instead of the windows, firewall and syslog indexes. We have since addressed this issue, but now we have Windows, Linux and firewall entries that we would like to export from main and move to the correct indexes.

The last time this happened (it has been a few years), I was able to perform a search using the splunk search capability that allowed us to save the content to a file in a raw format and then reimport the data. I did this with the help of support while working for another customer. Can someone help me do this again.

1 Solution

Splunk Employee
Splunk Employee

Splunk uses “main” as default index, which mean when index is not defined for data source it will index data in main.

It is common to see people miss it in early days of deployment and later it need some work to separate the data.

We do have option export data and later import as documented in links below, but some time export and import poses challenges.

http://answers.splunk.com/answers/25174/how-to-export-import-events-from-indexes.html
http://blogs.splunk.com/2013/09/15/exporting-large-results-sets-to-csv/

Here is an alternate that I like

1) Determine the buckets that hold your data using the search

| eval cd=_cd | eval bkt= _bkt | table cd bkt index splunk_server _raw

2) After that refine the above search to get the list of buckets that hold the required data

| eval cd=_cd | eval bkt= _bkt | table cd bkt index splunk_server _raw | dedup bkt | table bkt

3) Now let’s say your desire is to move the date with sourcetype=iis from data from index=main to index==iis , my suggestion will to move old data into index "iisold".

4) So, create new index "wineventold”

5) Now copy the list of above buckets from (2) from the $SPLUNK_DB/splunk/defaultdb/db or $SPLUNK_DB/splunk/defaultdb/colddb to $SPLUNK_DB/iisold/wineventold/db

6) Once the buckets are copied to new index you can check the data using (index=iisold will). As a next steps remove from (index=main index=iisold |delete ) . For delete refer link http://docs.splunk.com/Documentation/Splunk/6.1.4/SearchReference/Delete

7)In addition " index=iisold " might also have data from some other sourctype , you will need to delete that data using something like (index=main sourcetype= | delete)

😎 Now your old data sit on index= iisold and new data index=iis, to search this entire data you will need to have (index=iis OR index=iisold )

View solution in original post

Splunk Employee
Splunk Employee

You may be thinking of the exporttool and importtool commands. These are only necessary if you're moving some of the data from main. If you're moving all data in main, you can simply copy the buckets/folders from main to the new index (while Splunk is stopped, and not taking into account if you're using clustering).

0 Karma

Splunk Employee
Splunk Employee

Splunk uses “main” as default index, which mean when index is not defined for data source it will index data in main.

It is common to see people miss it in early days of deployment and later it need some work to separate the data.

We do have option export data and later import as documented in links below, but some time export and import poses challenges.

http://answers.splunk.com/answers/25174/how-to-export-import-events-from-indexes.html
http://blogs.splunk.com/2013/09/15/exporting-large-results-sets-to-csv/

Here is an alternate that I like

1) Determine the buckets that hold your data using the search

| eval cd=_cd | eval bkt= _bkt | table cd bkt index splunk_server _raw

2) After that refine the above search to get the list of buckets that hold the required data

| eval cd=_cd | eval bkt= _bkt | table cd bkt index splunk_server _raw | dedup bkt | table bkt

3) Now let’s say your desire is to move the date with sourcetype=iis from data from index=main to index==iis , my suggestion will to move old data into index "iisold".

4) So, create new index "wineventold”

5) Now copy the list of above buckets from (2) from the $SPLUNK_DB/splunk/defaultdb/db or $SPLUNK_DB/splunk/defaultdb/colddb to $SPLUNK_DB/iisold/wineventold/db

6) Once the buckets are copied to new index you can check the data using (index=iisold will). As a next steps remove from (index=main index=iisold |delete ) . For delete refer link http://docs.splunk.com/Documentation/Splunk/6.1.4/SearchReference/Delete

7)In addition " index=iisold " might also have data from some other sourctype , you will need to delete that data using something like (index=main sourcetype= | delete)

😎 Now your old data sit on index= iisold and new data index=iis, to search this entire data you will need to have (index=iis OR index=iisold )

View solution in original post

State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!