Getting Data In
Highlighted

Rename sourcetype to a non-existent type

Explorer

Hi,

I am trying to reset/rename the sourcetype based on the filename - which appears to work fine, if the sourcetype it is being renamed to exists in props.conf. But, what happens if it doesnt exist ?

I have an inital sourcetype based on json.

props.conf:
[clone-json]
pulldowntype = true
INDEXED
EXTRACTIONS = json
KVMODE = none
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
TIMESTAMP
FIELDS = timeStamp
TRANSFORMS-fs = force-sourcetype-st

This works perfectly, but now I need to change the sourcetype based on the filename, therefore the 'TRANSFORMS-fs = force-sourcetype-st' setting at the bottom.

If the source file is, /DATA/12345/interfaces.20160611.gz

transforms.conf:
[force-sourcetype-st]
DESTKEY = MetaData:Sourcetype
SOURCE
KEY = MetaData:Source
REGEX = \/\d+\/(\w+).\d+.gz$
FORMAT = sourcetype::$1
INDEXEDEXTRACTIONS = json
WRITE
META = true

So with the above, configurations, I am able to reset the sourcetype to interfaces and that works, however, when I look at the data in splunk it is duplicated. That is, if I pass in 1 record and do .... | stats count by id - it returns 2 instead of 1.

In this instance there is no sourcetype interfaces defined in the props.conf, so although I can change the sourcetype to interfaces, that type doesnt actually defined anywhere.

If I create the sourcetype interfaces, it all works fine.

You ask, why not just create the type if that makes it work ?
Well, I dont know what types are likely to come into the system, so I am trying to make it completely dynamic in nature.

I dont know if the problem is due to the INDEXED_EXTRACTION not being known, or its set to a default that is not json - or if there is some other metadata value I need to change to tell splunk the format and only to create 1 record.

Or worst case, I have to predefine all possible sourcetypes - even though that are all json in nature.

0 Karma
Highlighted

Re: Rename sourcetype to a non-existent type

Explorer

Solved, the issue was a props.conf miss-understanding by me - indextime vs runtime:

[source::.../json-files/*/*.gz]
KV_MODE = none
INDEXED_EXTRACTIONS = json

[clone-json]
pulldown_type = true
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
TIMESTAMP_FIELDS = timeStamp
TRANSFORMS-fs = force-sourcetype-st

Setting a source with KVMODE to none and the INDEXEXTRACTIONS there as well seems to have solved the issue.

View solution in original post

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.