props source stanza will not match

[zapping575]

I have these two files:

• EventLogger.log
• HOSTNAME-eventlog-TIMESTAMP.xml

The structure of their content is the same. Except that the .xml file may contain "<eventlog>" and "</eventlog>" at beginning and end.

So I thought to use the same sourcetype for them.

This stanza is working fine:

[source::/tmp/logs/.../qwe/.../*EventLogger.log*]
priority = 100
TZ = UTC
sourcetype = eventlogger

This one however, is not (splunk will apparently default and make a "xml-2" sourcetype):

[source::/tmp/logs/.../qwe/.../*eventlog*.xml*]
priority = 100
TZ = UTC
sourcetype = eventlogger

Using btool, I did confirm that on my indexer, the two stanzas above are actually present.

I feel like I am missing out on something here.

[PickleRick · Accepted solution]

The sourcetyper config works at the originating input component level.

sourcetype = <string>
* Can only be set for a [source::...] stanza.
* Anything from that <source> is assigned the specified source type.
* Is used by file-based inputs, at input time (when accessing logfiles) such
  as on a forwarder, or indexer monitoring local files.
* sourcetype assignment settings on a system receiving forwarded Splunk data
  are not be applied to forwarded data.
* For log files read locally, data from log files matching <source> is
  assigned the specified source type.
* Default: empty string

[zapping575]

This was the deciding tip. Turs out there was an issue with the deployment server which is why the files were not rolled out to the forwarder. Thank you very much!

[gcusello]

Hi @zapping575 ,

sorry for the question: how is it possible to use the same sourcetype with data source different?

I suppose that the parsing rules aren't the same for both formats.

I hint to use two sourcetypes.

Ciao.

Giuseppe

[zapping575]

Hi @gcusello 

Thats a good question actually. I was just presuming that splunk software would support the use case, where two different files have the same kind of data and can thus be treated as the same sourcetype.

I ended up creating a second sourcetype, as suggested but the outcome is still the same. I also introduced a filter to drop undesired events before ingestion (only keep events that begin with "<event ").

props.conf

[source::/tmp/logs/.../qwe/.../*EventLogger.log*]
priority = 100
TZ = UTC
sourcetype = eventlogger

[eventlogger]
KV_MODE = xml
TIME_PREFIX = timestamp="
TIME_FORMAT = %Y-%m-%dT%H:%M:%S%z
MAX_TIMESTAMP_LOOKAHEAD = 25
REPORT-mydefault = sourcefields-mydefault



[source::/tmp/logs/.../qwe/.../*eventlog*.xml*]
priority = 100
TZ = UTC
sourcetype = eventlog

[eventlog]
KV_MODE = xml
TIME_PREFIX = timestamp="
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE_DATE = false
LINE_BREAKER = <event
TIME_FORMAT = %Y-%m-%dT%H:%M:%S%z
MAX_TIMESTAMP_LOOKAHEAD = 25
REPORT-mydefault = sourcefields-mydefault
TRANSFORMS-null=setnull,eventlog-strip

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[eventlog-strip]
REGEX = ^<event\s
DEST_KEY = queue
FORMAT = indexQueue

[sourcefields-mydefault]
SOURCE_KEY = source
REGEX = /tmp/logs/(.*?)/(.*?)/(.*?)/
FORMAT = asd::$1 qwe::$2 yxc::$3

[gcusello]

Hi @zapping575 ,

as you said, using two sourcetypes you can correctly parse both the data sources and using a similar name for the sourcetypes you can use both of them using asterisk:

sourcetype=eventlog*

or (better) creating an eventtype

sourcetype IN (eventlog, eventlogger)

Ciao.

Giuseppe

[zapping575]

Hi @gcusello 

apologies if my previous post may have been ambiguous.

Out of the two source/sourcetype stanza combinations I posted, only the first one (eventlogger) is working as expected.

The second one does not work. Splunk just defaults to an "xml-2" sourcetype.