How to copy/reindex logs to another index continuously

[jpillai]

Hi all,

We have an index say index1 with a log retention of 7 days where we receive logs for different applications. Now we have a requirement to copy all ERROR logs from an application to another index, say index2 continuously.  The intention here is to have index2 with a higher retention so we can have access to the error logs for longer period.

What is the best way to implement such a mechanism. Its okay to run this job every day, or every 6 hour or so. Would be best to retain all fields and field extractions in logs in target index as well.

[jpillai]

The reason I need to ''copy the logs instead of splitting is that, there are some searches thats using index1 for searching the same data.  So we still need to support that dependancy. As the log volume thats getting copied is relatively small, the additional license usage is not a major issue.

It looks like scheduling search with collect is what fits the requirement we have

[gcusello]

Hi @jpillai ,

ok, my hint is to evaluate the effort to cheng the searches that use these indexes.

Anyway, let us know if we can help you more, or, please, accept one answer for the other people of Community.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

[PickleRick]

Different retention periods is the textbook case of splitting data between different indexes. So instead of copying you might consider simply sending some of your data to one index and the rest (the error logs) to another index. Otherwise - if you copy the events from one index to another - those events will be counted twice against your license which doesn't make much sense.

[jpillai]

The reason I need to ''copy the logs instead of splitting is that, there are some searches thats using index1 for searching the same data.  So we still need to support that dependancy. As the log volume thats getting copied is relatively small, the additional license usage is not a major issue.

[gcusello]

Hi @jpillai ,

do you want to divide logs between the two indexes or copy a part of logs (ERRORS) in the second one?

in this second case you pay twice the license.

Anyway, if you want to divide logs, you have to find the regex to identify logs fo index2 and put these props.conf and transforms.conf in your indexes or (if present) on Heavy Forwarders:

# etc/system/local/transforms.conf 
[overrideindex]
DEST_KEY =_MetaData:Index
REGEX = .
FORMAT = my_new_index

#etc/system/local/props.conf 
[mysourcetype]
TRANSFORMS-index = overrideindex

the REGEX to use is the one you identified.

If instead you want to copy ERROR logs in the second index, you can use the collect command, but you pay twice the license (using the original sourcetype) or you must use the stash sourcetype (in this case you don't pay twice the license.

In othe words, you have to schedule a search like the following (if "ERROR" is the string to identify logs to copy):

index=index1 "ERROR"
| collect index=index2

My hint is to override index value for the logs that you want to index in index2.

Ciao.

Giuseppe