Splunk Search

Disable Sources.data

vbumgarn
Path Finder

There's an app we have that is writing a file per transaction, and unfortunately, part of the useful information is in the filename itself.

But, the Sources.data has grown to such an extent that it is causing performance issues as it is constantly rewritten. It is currently 2.3 gigs.

What's the most effective way to essentially disable Sources.data? I have no use for it on this index.

Tags (1)
0 Karma

yannK
Splunk Employee
Splunk Employee

This is possible to disable the global metadata since 4.3.3
And in Splunk 5.0 the global metadata are deprecated.

To disable the Global Metdata and restore full speed indexing :

Add option to indexes.conf that disables global metadata generation to handle deployments with rapidly growing sources.data file. (SPL-47689)

The side consequence will be that the default summary page will not display values for the "last source/sourcetype/host".

to apply, edit indexes.conf and add :
disableGlobalMetadata = true

see http://docs.splunk.com/Documentation/Splunk/4.3.4/Admin/Indexesconf

vbumgarn
Path Finder

The eventual answer was to do a number of things...

  1. Create a transform to copy the source into a new field. Be sure to add to fields.conf.
  2. Create a transform to set source to a common string.
  3. Create an alias to allow searching the new field as source.

This effectively short circuits Sources.meta. It would be nice if there was a setting in props.conf to do all of this for you. Basically, a setting that says "I still want to be able to search this field, but I don't care if it works for the metadata command."

0 Karma

dwaddle
SplunkTrust
SplunkTrust

You can't. The Sources.data file is critical to Splunk's operation - part of its use is to help Splunk at search time to select buckets that may be relevant to a search. (If you did a search on source="/var/log/httpd/access*" then Splunk could quickly determine if a particular index bucket had any matches at all by just scanning the sources list.)

A better solution might be to rewrite the value of "source" using props.conf and transforms.conf. You would then want to pull out the "useful information" from the filename and put it into an otherwise indexed field. Basically, making the source somewhat less specific but still making the useful information available to you.

0 Karma

vbumgarn
Path Finder

I was under the impression those metadata files are only used by the |metadata commands, and the actual field values needed for search are in the tsidx files.

I was going down the route of copying source into orig_source and then trying to use FIELDALIAS to make it available as source, but the filenames have spaces in them, and I can't for the life of me make an indexed field work properly with spaces in the value.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...