topic How to move partial data from index to another index? in Getting Data In

How to move partial data from index to another index?

bogdan_nicolesc — Tue, 06 Sep 2022 14:35:23 GMT

Hi all,

I need some help.

I recently was wondering whether or not i could export some data from some index and import that data to a new index which is live, as in getting data in.

How can i do that?

Any step by step tutorial of some kind somewhere?

Thx.

Re: Move partial data from index to another index.

tiagofbmm — Fri, 02 Nov 2018 15:57:34 GMT

Maybe I'm missing something, but why not use the collect command ?

Re: Move partial data from index to another index.

bogdan_nicolesc — Mon, 05 Nov 2018 08:42:24 GMT

Hi,

I don't know what is that ... (?!) Any step by step tutorial using that?

Also, the data i want to move is ALREADY indexed in one old index with a bunch of other old data. Is a universal index and now i want to export specific data to a new index.

(Make any sense?!)

I tried to export that data by looking for source type using raw, json, xml, csv, but i think i'm doing something wrong as i cannot find exported data in my searches.

Any help/idea?

Thank you.

Re: Move partial data from index to another index.

tiagofbmm — Mon, 05 Nov 2018 08:58:22 GMT

If you are trying to move data in the same Splunk Environment between one index and another, you can just do this:

index=A | collect index=B

Regards

Tiago Matos

Re: Move partial data from index to another index.

bogdan_nicolesc — Mon, 05 Nov 2018 09:06:12 GMT

Hi tiagofbmm,

I don't want to move the same data (if this command is doing what i think is doing) but i want to move a sourcetype from one index (which is on old version of splunk) to a new index from a new version of splunk.

Thank you.

Re: Move partial data from index to another index.

tiagofbmm — Tue, 29 Sep 2020 21:53:28 GMT

A sourcetype is not data. A sourcetype is metadata that tells Splunk how it processes and shows data to you.

For instance, if you want to move the sourcetype Snare:Security, then install the Splunk_TA_Windows in your new environment and you're done.

If instead you want to move one index from an old Splunk to a new Splunk, then go to the location where your index is, $SPLUNK_HOME/var/lib/splunk/, and copy it to your new Splunk. After this, you also need to tell your new Splunk that this index exists, for which you need to create a stanza in indexes.conf on your new index.

[indexname]
homePath = $SPLUNK_DB/indexname/db
coldPath = $SPLUNK_DB/indexname/colddb
thawedPath = $SPLUNK_DB/indexname/thaweddb

Re: Move partial data from index to another index.

bogdan_nicolesc — Mon, 05 Nov 2018 10:47:51 GMT

I am already past by that, now the real 1mil$ question is if there is some sort of option/solution to tell splunk to export just this:
==>>>>> source="WinEventLog:Security"<<<<<===

From this: index=main

As i was saying above and earlier, i can export in csv, json, xml, or even raw events, but for some reason, i don't know why, is exporting data without >>>FIELDS<<< so i cannot use that data in a previously built dashboard.

So ... back to my question (as we dived more into my question), Is there, any kind/sort of SOP so i can use to export and then import data correctly into a new index?

It is doable?!

Is even possible to do this or i'm dreaming of green horses on walls?

Thank you.

Re: Move partial data from index to another index.

tiagofbmm — Mon, 05 Nov 2018 11:29:18 GMT

When you try to move data into another index, you can specify the sourcetype that data will have in the other index.

Collect moves raw data from one index to another. If you want the fields that you may have EXTRACTED, CALCULATED, FIELDALIAS, whatever else, then these are associated to a specific sourcetype.

When issuing the collect command, specify sourcetype=.

You can also specify the source too:

https://docs.splunk.com/Documentation/Splunk/7.1.2/SearchReference/Collect

Re: Move partial data from index to another index.

bogdan_nicolesc — Mon, 05 Nov 2018 11:35:17 GMT

I don't want to be rude or something, but i think i need some translation here as i don't understand a bit from what is in that documentation. I'm fairly new to splunk.

Thanks.

Re: Move partial data from index to another index.

tiagofbmm — Mon, 05 Nov 2018 12:00:35 GMT

Imagine you want to copy source A from index X to index Y with the same source.

Execute this: index=X source=A | collect index=Y source=A sourcetype=

Re: Move partial data from index to another index.

bogdan_nicolesc — Mon, 05 Nov 2018 12:39:12 GMT

Thank you,

This was what i was asking for (tutorial/explanation).

And sourcetype= i believe that you type it by mistake?!

Re: Move partial data from index to another index.

tiagofbmm — Mon, 05 Nov 2018 13:03:25 GMT

Both sourcetype and source can be set optionally

Re: Move partial data from index to another index.

bogdan_nicolesc — Mon, 05 Nov 2018 14:17:22 GMT

Worked. Thank you for your time.

Re: Move partial data from index to another index.

bogdan_nicolesc — Mon, 05 Nov 2018 14:18:46 GMT

index=X source=A | collect index=Y source=A (optionally sourcetype=)

From:
index=X source=A

To:
collect index=Y source=A

Details:
In order to copy one source and/or sourctype, from one old index (even if it's on old version of splunk) you need to type in splunk search:

index=X source=A | collect index=Y source=A

Where index=X source=A indicates the old index

And collect index=Y source=A is the new index.

In order to work, you must have pipe |.

Re: Move partial data from index to another index.

tiagofbmm — Mon, 05 Nov 2018 14:25:13 GMT

Would you please accept my answer and upvote the help comments I took my time to do?

Re: Move partial data from index to another index.

911 — Tue, 06 Sep 2022 08:52:11 GMT

We have a ticket index i.e index=incident the status of the ticket is Assigned but I need to write the same data back into the incident index with new status = Orphaned

Also need to changed the _time

index=incident Incident_Number="XXXXXX" ticketStatus=Assigned
| rex mode=sed "s/Status=\"Assigned\"/Status=\"Orphaned\"/"
| rex mode=sed "s/ticketStatus=\"Assigned\"/ticketStatus=\"Orphaned\"/"
| collect index=incident