topic Define sourctype based on the host - via a lookup? in Splunk Search

Define sourctype based on the host - via a lookup?

robf — Tue, 01 Oct 2013 10:17:09 GMT

We have many different data sources which can only send on 514 UDP.

I need to define the sourcetype based on the host value.

I can see this can be done easily using a regex as described here

Advancedsourcetypeoverrides

However we have thousands of hosts with no obvious naming conventions.

so what i would want to do is maintain a list of hostnames and refernce that and say

if in ListA; sourcetype=typeA
if in ListB; sourcetype=typeB

etc

i really dont want to have several huge regex like

there is also no easy way to to regex based on the pattern of the events!

any ideas?!

Re: Define sourctype based on the host - via a lookup?

Ayn — Tue, 01 Oct 2013 10:59:10 GMT

Sorry, mechanisms like lookup are simply not available in that stage of the indexing pipeline. Lookups are pure search-time operations. Transforms like sourcetype rewriting, event filtering etc can be done with regexes only.

Re: Define sourctype based on the host - via a lookup?

robf — Tue, 01 Oct 2013 11:13:29 GMT

thanks. so is the only way to do a huge regex as far as you know?

Re: Define sourctype based on the host - via a lookup?

kristian_kolb — Tue, 01 Oct 2013 11:24:47 GMT

Ayn is right, unfortunately. You can set up Splunk to listen on more than one port (515,516,517 etc) and specify in inputs.conf on the receiving end that dictates the sourcetype to be used;

[udp://:514]
connection_host = dns
sourcetype = type_a

[udp://:515]
connection_host = dns
sourcetype = type_b

[udp://:516]
connection_host = dns
sourcetype = type_c

Unfortunately, you'd need to reconfigure a substantial number of your hosts to send to another port. And all data coming from each host would have the same sourcetype. (though you could override it, of course).

Re: Define sourctype based on the host - via a lookup?

robf — Tue, 01 Oct 2013 11:32:59 GMT

thanks but unfortunately a number of devices cannot change their remote syslog port number

Re: Define sourctype based on the host - via a lookup?

kristian_kolb — Tue, 01 Oct 2013 12:35:23 GMT

Depending on your setup, i.e. if it's just the port that can't be changed, you might be able to do a similar operation, but by setting up a few syslog servers (rsyslog, splunk or other), and changing the destination ip on the sending hosts.

Then you install a forwarder on each syslog server... and set the sourcetype in inputs.conf. A bit messy - to say the least - for the initial setup, but hopefully fairly straightforward once it's set up.

Re: Define sourctype based on the host - via a lookup?

MuS — Fri, 05 Sep 2014 05:55:06 GMT

Hi robf,

@Ayn and @kristian.kolb are both right, but you could try something like this and use a slim regex in transforms.conf:

[yourHostTransforms]
SOURCE_KEY = host
DEST_KEY = MetaData:Sourcetype
REGEX = host\d+
FORMAT = sourcetype::$1

I did something for an index re-write lately, but did not test it for sourcetype yet.

hope this helps ...

cheers, MuS

Re: Define sourctype based on the host - via a lookup?

kristian_kolb — Fri, 05 Sep 2014 08:13:11 GMT

?? that would mean one sourcetype per host... assuming that you'd put the whole REGEX as capturing group.

And the hosts were not named in such a fashion. .. and @robf has already been down that road.

Re: Define sourctype based on the host - via a lookup?

MuS — Fri, 05 Sep 2014 08:23:22 GMT

Hi /k, I used the information available here in this post and showed an example what can be done. There is nothing mentioned about hosts not being named in such a fashion.

Re: Define sourctype based on the host - via a lookup?

kristian_kolb — Fri, 05 Sep 2014 13:08:10 GMT

"thousands of hosts with no obvious naming conventions".
He wants 2 different sourcetypes.

But the post is almost a year old, and maybe he solved the problem already.

🙂

Re: Define sourctype based on the host - via a lookup?

MuS — Fri, 05 Sep 2014 13:19:30 GMT

HeHe, let's call this selective memory 🙂 Only the regex example for the hosts was left while writing it. Never mind, maybe someone else can use this.

cheers, MuS