Splunk Search

What considerations should I make when rewriting metadata for best efficiency?

brent_weaver
Builder

I am in a situation where I need to rewrite metadata for each and every event. I need to rewrite index and sourcetype for starters. This is in a distributed environment with heavy forwarders in front of the indexers.

What considerations should I make?

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Okay, that's a conversation that really ought to be spoken over a lot of beer. You are basically asking "what are the considerations when (re)architecting an entire splunk ecology?" Without knowing more about your use case(s), I could wear down my fingers expounding on the internet without providing you the insights you most need.

Separating data by index and sourcetype -- segregating data with regard to how that data is going to be used -- is one key to efficiency of access, as long as you don't go too far. (Pretty much like normalization in relational databases. You normalize the overall design, then denormalize selectively to achieve maximum workability for your real-world applications.)

When considering what indexes to create, consider your users and their various roles, as well as the sensitivity of the particular classes of data involved. Consider frequency of access to each type of data, and consider granularity...whether the data will be generally needed at the detail level, or whether (and to what degree) aggregates in summary indexes would adequately meet most needs.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

The metadata overwrite operation (transforms) will happen on the Heavy forwarder, so make sure you've sufficient number of heavy forwarders (at least one per indexer you have) with decent h/w configurations. The reference h/w size will depend upon the data load you'll per indexer. This may help.
http://docs.splunk.com/Documentation/Splunk/6.5.2/Capacity/Performancechecklist

0 Karma

woodcock
Esteemed Legend

Do you mean redesign or do you really mean that you are going to modify the data for buckets on the indexers after data has been indexed? I strongly advise against the latter.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...