topic Is there a better way to edit my current inputs.conf for sourcetypes defined by path? in Getting Data In

Is there a better way to edit my current inputs.conf for sourcetypes defined by path?

daniel333 — Fri, 07 Oct 2016 17:39:41 GMT

All,

I have a dozen+ inputs I am creating. I feel there there should be a smarter way of doing this. As you can see, I am naming the sourcetype after the log.

[monitor:///opt/paidsearch/autopilot/logs/collateral.log]
index=paidsearch
sourcetype=paidsearch:collateral

[monitor:///opt/paidsearch/autopilot/logs/partner.log]
index=paidsearch
sourcetype=paidsearch:partner

Re: Is there a better way to edit my current inputs.conf for sourcetypes defined by path?

javiergn — Fri, 07 Oct 2016 18:08:49 GMT

Yes there is.
(NOTE I HAVEN'T TESTED THE BELOW)

Collection layer (normally a universal forwarder):

 [monitor:///opt/paidsearch/autopilot/logs/*.log]
 index=paidsearch
 sourcetype=paidsearch:rename

Parsing layer (before indexing, usually a heavy forwarder or indexer):

# props.conf
[paidsearch:rename]
TRANSFORMS-changesourcetype = set_paidsearch_sourcetype_from_filename


# transforms.conf 
[set_paidsearch_sourcetype_from_filename]
SOURCE_KEY = MetaData::Source
REGEX = ([^\/]+)\.\w+$
FORMAT = paidsearch:$1
DEST_KEY = MetaData::Sourcetype
WRITE_META = true

Re: Is there a better way to edit my current inputs.conf for sourcetypes defined by path?

ddrillic — Fri, 07 Oct 2016 18:08:50 GMT

There must be all kinds of ways - I create tiny little Java programs for this type of cases...

Re: Is there a better way to edit my current inputs.conf for sourcetypes defined by path?

twinspop — Fri, 07 Oct 2016 20:22:34 GMT

I usually think changing sourcetype is evil and should be avoided. But this is nice. I like it.

Re: Is there a better way to edit my current inputs.conf for sourcetypes defined by path?

Masa — Mon, 10 Oct 2016 21:00:46 GMT

inputs.conf

 [monitor:///opt/paidsearch/autopilot/logs]
 index=paidsearch
 sourcetype=paidsearch

props.conf (at the same Splunk instance because this is input stage configuration)

 [source::/opt/paidsearch/autopilot/logs/collateral.log]
 sourcetype=paidsearch:collateral
 [source::/opt/paidsearch/autopilot/logs/partner.log]
 sourcetype=paidsearch:partner