Getting Data In

How to move partial data from index to another index?

bogdan_nicolesc
Communicator

Hi all,

I need some help.

I recently was wondering whether or not i could export some data from some index and import that data to a new index which is live, as in getting data in.

How can i do that?

Any step by step tutorial of some kind somewhere?

Thx.

Labels (1)
0 Karma
1 Solution

bogdan_nicolesc
Communicator

index=X source=A | collect index=Y source=A (optionally sourcetype=)

From:
index=X source=A

To:
collect index=Y source=A

Details:
In order to copy one source and/or sourctype, from one old index (even if it's on old version of splunk) you need to type in splunk search:

index=X source=A | collect index=Y source=A

Where index=X source=A indicates the old index

And collect index=Y source=A is the new index.

In order to work, you must have pipe |.

View solution in original post

bogdan_nicolesc
Communicator

index=X source=A | collect index=Y source=A (optionally sourcetype=)

From:
index=X source=A

To:
collect index=Y source=A

Details:
In order to copy one source and/or sourctype, from one old index (even if it's on old version of splunk) you need to type in splunk search:

index=X source=A | collect index=Y source=A

Where index=X source=A indicates the old index

And collect index=Y source=A is the new index.

In order to work, you must have pipe |.

tiagofbmm
Influencer

Would you please accept my answer and upvote the help comments I took my time to do?

0 Karma

tiagofbmm
Influencer

Maybe I'm missing something, but why not use the collect command ?

bogdan_nicolesc
Communicator

Hi,

I don't know what is that ... (?!) Any step by step tutorial using that?

Also, the data i want to move is ALREADY indexed in one old index with a bunch of other old data. Is a universal index and now i want to export specific data to a new index.

(Make any sense?!)

I tried to export that data by looking for source type using raw, json, xml, csv, but i think i'm doing something wrong as i cannot find exported data in my searches.

Any help/idea?

Thank you.

0 Karma

tiagofbmm
Influencer

If you are trying to move data in the same Splunk Environment between one index and another, you can just do this:

index=A | collect index=B

Regards

Tiago Matos

bogdan_nicolesc
Communicator

Hi tiagofbmm,

I don't want to move the same data (if this command is doing what i think is doing) but i want to move a sourcetype from one index (which is on old version of splunk) to a new index from a new version of splunk.

Thank you.

0 Karma

tiagofbmm
Influencer

A sourcetype is not data. A sourcetype is metadata that tells Splunk how it processes and shows data to you.

For instance, if you want to move the sourcetype Snare:Security, then install the Splunk_TA_Windows in your new environment and you're done.

If instead you want to move one index from an old Splunk to a new Splunk, then go to the location where your index is, $SPLUNK_HOME/var/lib/splunk/, and copy it to your new Splunk. After this, you also need to tell your new Splunk that this index exists, for which you need to create a stanza in indexes.conf on your new index.

[indexname]
homePath = $SPLUNK_DB/indexname/db
coldPath = $SPLUNK_DB/indexname/colddb
thawedPath = $SPLUNK_DB/indexname/thaweddb

0 Karma

bogdan_nicolesc
Communicator

I am already past by that, now the real 1mil$ question is if there is some sort of option/solution to tell splunk to export just this:
==>>>>> source="WinEventLog:Security"<<<<<===

From this: index=main

As i was saying above and earlier, i can export in csv, json, xml, or even raw events, but for some reason, i don't know why, is exporting data without >>>FIELDS<<< so i cannot use that data in a previously built dashboard.

So ... back to my question (as we dived more into my question), Is there, any kind/sort of SOP so i can use to export and then import data correctly into a new index?

It is doable?!

Is even possible to do this or i'm dreaming of green horses on walls?

Thank you.

0 Karma

tiagofbmm
Influencer

When you try to move data into another index, you can specify the sourcetype that data will have in the other index.

Collect moves raw data from one index to another. If you want the fields that you may have EXTRACTED, CALCULATED, FIELDALIAS, whatever else, then these are associated to a specific sourcetype.

When issuing the collect command, specify sourcetype=.

You can also specify the source too:

https://docs.splunk.com/Documentation/Splunk/7.1.2/SearchReference/Collect

0 Karma

bogdan_nicolesc
Communicator

I don't want to be rude or something, but i think i need some translation here as i don't understand a bit from what is in that documentation. I'm fairly new to splunk.

Thanks.

0 Karma

tiagofbmm
Influencer

Imagine you want to copy source A from index X to index Y with the same source.

Execute this: index=X source=A | collect index=Y source=A sourcetype=

0 Karma

bogdan_nicolesc
Communicator

Thank you,

This was what i was asking for (tutorial/explanation).

And sourcetype= i believe that you type it by mistake?!

0 Karma

tiagofbmm
Influencer

Both sourcetype and source can be set optionally

0 Karma

bogdan_nicolesc
Communicator

Worked. Thank you for your time.

0 Karma

911
Engager

We have a ticket index i.e index=incident the status of the ticket is Assigned but I need to write the same data back into the incident index with new status = Orphaned

Also need to changed the _time

index=incident Incident_Number="XXXXXX" ticketStatus=Assigned
| rex mode=sed "s/Status=\"Assigned\"/Status=\"Orphaned\"/"
| rex mode=sed "s/ticketStatus=\"Assigned\"/ticketStatus=\"Orphaned\"/"
| collect index=incident

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...