Why are my sourcetypes constantly changing, and how do I prevent this?

[Michael]

I have a small LAN with a couple dozen servers all running Solaris. They are sending into a single instance of Splunk Enterprise via syslog.

I have created field extractions, dashboards, reports, etc. and everyone was happy -- then the sourcetypes started changing themselves, thereby breaking the field extractions etc.. To illustrate this, I pulled up logons for the same user, to the same system over a 30 day period and noted three different sourcetypes in that time: "udp:514", "authlog-too_small", and "syslog". The source was always udp:514, and other than the sourcetype changing, I can tell no other difference in the events.

I've been having this issue for a long time and have tried adding sourcetype=syslog anywhere I can (inputs.conf, props.conf).

Other sourcetypes also change on other things, not use the username on this logon event.

This did this on 6.3, and now is still doing it on 6.5.

I don't care so much that I can't force the sourcetype to syslog, as long as it stops changing randomly!

ANY ideas would be appreciated, even untested ones...!
thanks,
Mike

[koshyk]

please provide a copy of your inputs.conf for collecting the syslog

[somesoni2 · Accepted solution]

It seems like you're not assigning sourcetypes when setting up the data input (in inputs.conf on syslog server where you're receiving data on UDP port), thus letting Splunk decide the sourcetype dynamically based on event content, the reason for multiple sourcetypes. Ideally, you should setup inputs.conf entry for each UDP input and assign sourcetype to each. In your case, you want to set the sourcetype to "syslog" for all UDP input, then add this to inputs.conf ($SPLUNK_HOME/etc/system/local/inputs.conf OR $SPLUNK_HOME/etc/apps/AnyAppYouCreated/local/inputs.conf)

[default]
sourcetype=syslog

Now all inputs.conf entries which doesn't have explicitly sourcetype assigned will have a common sourcetype name as "syslog".

[Michael]

Unfortunately, this is on a system in another part of the campus, and I'll have to go check this later. I'm pretty sure I have an entry in inputs.conf -- however, it most likely says:

[udp:514]
sourcetype=syslog

I hadn't thought of setting default to syslog -- will give it a try and let you know.

Thanks!

[Michael]

Looks like that did the trick.

I did have:
[udp:514]
sourcetype=syslog

then added under [default]
sourcetype=syslog

Thanks!