Splunk Search

How to copy/reindex logs to another index continuously

jpillai
Path Finder

Hi all,

We have an index say index1 with a log retention of 7 days where we receive logs for different applications. Now we have a requirement to copy all ERROR logs from an application to another index, say index2 continuously.  The intention here is to have index2 with a higher retention so we can have access to the error logs for longer period.

 

What is the best way to implement such a mechanism. Its okay to run this job every day, or every 6 hour or so. Would be best to retain all fields and field extractions in logs in target index as well.

 

Labels (1)
0 Karma

jpillai
Path Finder

The reason I need to ''copy the logs instead of splitting is that, there are some searches thats using index1 for searching the same data.  So we still need to support that dependancy. As the log volume thats getting copied is relatively small, the additional license usage is not a major issue.

 

It looks like scheduling search with collect is what fits the requirement we have

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @jpillai ,

ok, my hint is to evaluate the effort to cheng the searches that use these indexes.

Anyway, let us know if we can help you more, or, please, accept one answer for the other people of Community.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Different retention periods is the textbook case of splitting data between different indexes. So instead of copying you might consider simply sending some of your data to one index and the rest (the error logs) to another index. Otherwise - if you copy the events from one index to another - those events will be counted twice against your license which doesn't make much sense.

jpillai
Path Finder

The reason I need to ''copy the logs instead of splitting is that, there are some searches thats using index1 for searching the same data.  So we still need to support that dependancy. As the log volume thats getting copied is relatively small, the additional license usage is not a major issue.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @jpillai ,

do you want to divide logs between the two indexes or copy a part of logs (ERRORS) in the second one?

in this second case you pay twice the license.

Anyway, if you want to divide logs, you have to find the regex to identify logs fo index2 and put these props.conf and transforms.conf in your indexes or (if present) on Heavy Forwarders:

# etc/system/local/transforms.conf 
[overrideindex]
DEST_KEY =_MetaData:Index
REGEX = .
FORMAT = my_new_index

#etc/system/local/props.conf 
[mysourcetype]
TRANSFORMS-index = overrideindex

the REGEX to use is the one you identified.

If instead you want to copy ERROR logs in the second index, you can use the collect command, but you pay twice the license (using the original sourcetype) or you must use the stash sourcetype (in this case you don't pay twice the license.

In othe words, you have to schedule a search like the following (if "ERROR" is the string to identify logs to copy):

index=index1 "ERROR"
| collect index=index2

My hint is to override index value for the logs that you want to index in index2.

Ciao.

Giuseppe

 

Get Updates on the Splunk Community!

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

See your relevant APM services, dashboards, and alerts in one place with the updated ...

As a Splunk Observability user, you have a lot of data you have to manage, prioritize, and troubleshoot on a ...