topic Re: methods for removing specific data performance in Monitoring Splunk

methods for removing specific data performance

brettcave — Mon, 21 Jan 2013 12:23:08 GMT

hi, i've been reading over the threads on the topic, and most splunkers are looking at removing some data for space-saving purposes, but I haven't come across any responses addressing performance.

our splunk data has a few different data types - analytics data, backup reports, internal testing logs and some others. while we want to keep analytics data for indefinite periods, I am finding that the reports are becoming very sluggish, and was wondering what approach would be best to strip out specific data. Would the ' | delete ' command prevent the events from being scanned during reporting and improve performance? Or is there another mechanism that would work better for this?

The sort of approach I am looking for, as an example, would be to remove / archive / purge:

backup logs older than 14 days
analytics / web access logs flagged as internal users or from certain IP addresses ("internal" based on a field extractor, IP based on inputlookup )
application logs matching certain search terms older than 30 days

Re: methods for removing specific data performance

brettcave — Mon, 21 Jan 2013 12:28:17 GMT

busy reading up on creating multiple indexes http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Setupmultipleindexes it offers what i am looking for.

Re: methods for removing specific data performance

brettcave — Mon, 21 Jan 2013 12:57:14 GMT

I am following the example to try redirect a sourcetype to an alternative index. The sourcetype definition uses an extract. After adding the transforms to props.conf, I don't see the transformation in the web manager. This is what I have:

props.conf:

[my_sourcetype]
EXTRACT-backuplog = STATUS: \[Name: (?P<JobName>[^,]+), Result: (?P<JobResult>[^;]+);
TRANSFORMS-index = IndexRedirect

transforms.conf

[IndexRedirect]
REGEX = STATUS: \[Job:
DEST_KEY = _MetaData:Index
FORMAT = backup_index

How would I configure an extractor and a transform for a sourcetype?

Re: methods for removing specific data performance

brettcave — Mon, 21 Jan 2013 13:51:29 GMT

what happens to data with a sourcetype of my_sourcetype that doesn't match the regex? does it go to the main index?