reducing metadata size

[a212830]

Hi,

I recall hearing at Splunk Conf2013 that it's a good idea to keep your metadata size reasonable. I process lots of logfiles which are in the format /apps/logs/YYYY/MM/DD/system-hostname.log. I'd like to change the source to be just the system-hostname.log, and eliminate the date. Is that possible? If so, how?

[gkanapathy]

it's actually not as necessary as it was in earlier (say, 4.2 or earlier) versions of Splunk. also, it seems like you're only generating a new source name per day (rather than say, per-hour), which typically won't result in unmanageable metadata, even in the older versions in most cases. but yes, if you want to do it, or you're on an older version, or you have a more complex environment that results in a lot more source names, you can do an index-time transform.

e.g., transforms.conf:

[reduce-source-name]
SOURCE_KEY = MetaData::Source
REGEX = ([^\/\\]*)$
FORMAT = host::$1
DEST_KEY = MetaData::Source

(possibly the REGEX and FORMAT could be optimized for the specific path) and then something in props.conf to invoke the rule, e.g.,

[mysourcetype]
TRANSFORMS-sourcename = reduce-source-name

or

[source::/app/logs/*/*/*/system-*.log]
TRANSFORMS-sourcename = reduce-source-name