Deployment Architecture

Replicate indexed events at certain times


I'm trying to get data that I'm indexing at one location to be replicated to another Splunk Indexer at a remote site ONLY during a daily time window (1AM to 3AM).

I've thought of a few options but most of them would involve unwanted side effects such as:

Restarting one of my instances (eg. freezing/ copying/ thawting the buckets);
Not ensuring there are no gaps in the data (eg. forwarder instance being kicked in by a script);
Doubling my license expenditure.
Messing up my data format (eg. Summary Indexing + outputs.conf)

I'm thinking about developing something that would make use of the REST API (not sure about the license implications of that) but maybe someone has already devised a more practical way?

Tags (2)
0 Karma

Splunk Employee
Splunk Employee

sounds more or less like a backup. I would suggest that simply copying the buckets over is the right way. you can copy cold and warm buckets while the system is running, and of course you don't have to copy buckets that were already copied. hot buckets are trickier. you can in fact just copy the journal.gz file from each hot bucket each day. however, at the end of each day, you need to delete the hot buckets from the previous day (as they will be either modified or have rolled to warm).

it's actually possible to copy only the journal.gz files from the warm and cold buckets also, and then rebuild the rest of the bucket after the copy is complete. this would be preferred if bandwidth and time are issues, which they seem to be.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!