Monitoring Splunk

methods for removing specific data performance

brettcave
Builder

hi, i've been reading over the threads on the topic, and most splunkers are looking at removing some data for space-saving purposes, but I haven't come across any responses addressing performance.

our splunk data has a few different data types - analytics data, backup reports, internal testing logs and some others. while we want to keep analytics data for indefinite periods, I am finding that the reports are becoming very sluggish, and was wondering what approach would be best to strip out specific data. Would the ' | delete ' command prevent the events from being scanned during reporting and improve performance? Or is there another mechanism that would work better for this?

The sort of approach I am looking for, as an example, would be to remove / archive / purge:

  • backup logs older than 14 days
  • analytics / web access logs flagged as internal users or from certain IP addresses ("internal" based on a field extractor, IP based on inputlookup )
  • application logs matching certain search terms older than 30 days
Tags (2)
0 Karma
1 Solution

brettcave
Builder

busy reading up on creating multiple indexes http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Setupmultipleindexes it offers what i am looking for.

View solution in original post

0 Karma

brettcave
Builder

what happens to data with a sourcetype of my_sourcetype that doesn't match the regex? does it go to the main index?

0 Karma

brettcave
Builder

I am following the example to try redirect a sourcetype to an alternative index. The sourcetype definition uses an extract. After adding the transforms to props.conf, I don't see the transformation in the web manager. This is what I have:

props.conf:

[my_sourcetype]
EXTRACT-backuplog = STATUS: \[Name: (?P<JobName>[^,]+), Result: (?P<JobResult>[^;]+);
TRANSFORMS-index = IndexRedirect

transforms.conf

[IndexRedirect]
REGEX = STATUS: \[Job:
DEST_KEY = _MetaData:Index
FORMAT = backup_index

How would I configure an extractor and a transform for a sourcetype?

0 Karma

brettcave
Builder

busy reading up on creating multiple indexes http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Setupmultipleindexes it offers what i am looking for.

0 Karma
Get Updates on the Splunk Community!

New This Month in Splunk Observability Cloud - Metrics Usage Analytics, Enhanced K8s ...

The latest enhancements across the Splunk Observability portfolio deliver greater flexibility, better data and ...

Alerting Best Practices: How to Create Good Detectors

At their best, detectors and the alerts they trigger notify teams when applications aren’t performing as ...

Discover Powerful New Features in Splunk Cloud Platform: Enhanced Analytics, ...

Hey Splunky people! We are excited to share the latest updates in Splunk Cloud Platform 9.3.2408. In this ...