Splunk Search

Put Data in Separate Index Based on Timestamp II

chris
Motivator

I know this Question has been asked before (http://answers.splunk.com/questions/712/put-data-in-separate-index-based-on-timestamp) but we will start with end of year tests soon. Some of our test servers will simulate what will happen on Dec 31st at midnight. We would like to have the data from those test servers in a different index somehow.

I'd like to know if anyone has done anaything similar before. We're thinking about setting up a temporary indexer and then reconfigure syslog and our Splunk Forwarders to make sure that our main data does not get polluted.

Any Ideas?

Thanks Chris

Tags (3)
0 Karma
1 Solution

Lowell
Super Champion

Another option is to use a transformer to set the _MetaData:Index property. I would only suggest this if you have very simmilar timestamps across all of your events; otherwise writing a proper regular expression will be very difficult.

This example assumes that only events for Dec 31 2010 and Jan 1 2011 will occur for this test. In other words, if you forget to correct your clock and the system rolls over to Jan 2, 2011 the that your event will end up in your current index. Here is an example set of config files: (I would recommend you put them in an app that you disable as soon as your testing period is done. You obviously don't want your real events on Dec 31 and Jan 1 to end up in your testing index.)

props.conf

[syslog]
TRANSFORMS-year_end_testing = route_index_YE_testing

[sourcetype-n]
TRANSFORMS-year_end_testing = route_index_YE_testing
...

transforms.conf:

[route_index_YE_testing]
REGEX = ^(Dec\s+31|Jan\s+1)\s
FORMAT = test_ye
DEST_KEY = _MetaData:Index

In this example, "test_ye" is the name of your testing index which you must create. Also, "sourcetype-n" is a placeholder. You must explicitly list out all all sourcetypes that will be involved. And each sourcetype must use this transformer (or a simmilar transformer, if you create a different transformer for your timestamp formats) of only part of your data will be routed to the correct location.


If you aren't very familiar with indexing routing like this, fluent with writing and testing regular expression, or don't have full control over your sourcetypes than one other options would probably be better. They all have different pros/cons, and this could be rather tricky to get right on the first try....

View solution in original post

Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...