topic Re: How can i redirect data from an index to a new one in Getting Data In

How can i redirect data from an index to a new one

baroudiem — Tue, 22 Jan 2019 13:32:10 GMT

Hello,
I would like to know how can i redirect data from a general index to a new one ?

Example :
General index : hostname / ip / port

New index : only port

Thanks.

Re: How can i redirect data from an index to a new one

ccl0utier — Tue, 22 Jan 2019 15:18:45 GMT

As in "I would like to duplicate a subset of selected events data to another index on top of the current one"?

I suggest you please clarify your question.

Re: How can i redirect data from an index to a new one

gcusello — Wed, 23 Jan 2019 13:49:43 GMT

Hi baroudiem,
if you want to send events that contain a string, you can override the index value depending on a regex On your indexer or heavy forwarder:

# etc/system/local/transforms.conf 

     [overrideindex]
     DEST_KEY =_MetaData:Index
     REGEX = .
     FORMAT = my_new_index

  #etc/system/local/props.conf 

     [mysourcetype]
     TRANSFORMS-index = overrideindex

If instead you want to duplicate in another index a subset of your fields (not all the events), it isn't a good idea to index twice because you have a double license consuption , the best thing is to schedule a search that extract the fields you want and then collect them in one of the following ways:

if they aren't so much, in a lookup using the outputlookup command;
if they are many, in a summary index using the collect command.

Bye.
Giuseppe

Re: How can i redirect data from an index to a new one

gcusello — Wed, 31 Jul 2019 12:54:07 GMT

Hi baroudiem,
if you're satisfied by this answer, please accept and/or upvote it.

Bye, see next time.
Giuseppe

Re: How can i redirect data from an index to a new one

FredericHebert — Wed, 26 Jul 2023 20:50:12 GMT

Hi, this seems to be based on [mysourcetype].

So, if an index gets dozens of sourcetypes treated in the HF, I will need to overrride each one of them individually.

I need to redirect for a short period of time targeting nullQueue for the remaining of the day. All this is detected via alerts throttled upon thresholds crossing.

Once the theshold crossed, I need a "kill switch" that would flush and data into an index based on an allowed ingestion threshold (plus 5%). I thought of overriding from MyIndex to nullQueue using props/transforms files but I need it to be simply and efficient.

This needs to take precedence on all MyIndex related props/transforms that would still exist, but would simply be left aside.

I would deliver "on the fly" an app that would contain props/transforms (all data targeting MyIndex redirect it to nullQueue) and restart my splunk HF service. at midnight, i would simple delete to "on the fly" an app and restart my splunk HF, falling back to the previously left aside exsiting MyIndex related props/transforms.

Re: How can i redirect data from an index to a new one

gcusello — Thu, 27 Jul 2023 03:25:31 GMT

Hi @FredericHebert ,

please open a new question even if on the same topic of this question, because in this way, you'll have more attention from more people and the a quicker an maybe better answer.

Anyway, I usually use sourcetype in props.conf , but you can also use host or source but with a little different format:

[host::<your_host>] or [source::<your_source>]

For my knowledhe it isn't possible to define a validity period for a conf file: you should manually (or using a scheduled shell script) modify a conf file and restart the Heavy Forwarder, in my opinion isn't an efficient solution!

Ciao.

Giuseppe