Splunk Search

Define sourctype based on the host - via a lookup?

robf
Path Finder

We have many different data sources which can only send on 514 UDP.

I need to define the sourcetype based on the host value.

I can see this can be done easily using a regex as described here

Advancedsourcetypeoverrides

However we have thousands of hosts with no obvious naming conventions.

so what i would want to do is maintain a list of hostnames and refernce that and say

if in ListA; sourcetype=typeA
if in ListB; sourcetype=typeB

etc

i really dont want to have several huge regex like

host1|host2|host3|host4|.................|host230|host231....

there is also no easy way to to regex based on the pattern of the events!

any ideas?!

0 Karma

MuS
Legend

Hi robf,

@Ayn and @kristian.kolb are both right, but you could try something like this and use a slim regex in transforms.conf:

[yourHostTransforms]
SOURCE_KEY = host
DEST_KEY = MetaData:Sourcetype
REGEX = host\d+
FORMAT = sourcetype::$1

I did something for an index re-write lately, but did not test it for sourcetype yet.

hope this helps ...

cheers, MuS

0 Karma

MuS
Legend

HeHe, let's call this selective memory 🙂 Only the regex example for the hosts was left while writing it. Never mind, maybe someone else can use this.

cheers, MuS

0 Karma

kristian_kolb
Ultra Champion
  • "thousands of hosts with no obvious naming conventions".
  • He wants 2 different sourcetypes.

But the post is almost a year old, and maybe he solved the problem already.

🙂

/K

0 Karma

MuS
Legend

Hi /k, I used the information available here in this post and showed an example what can be done. There is nothing mentioned about hosts not being named in such a fashion.

0 Karma

kristian_kolb
Ultra Champion

?? that would mean one sourcetype per host... assuming that you'd put the whole REGEX as capturing group.

And the hosts were not named in such a fashion. .. and @robf has already been down that road.

0 Karma

kristian_kolb
Ultra Champion

Ayn is right, unfortunately. You can set up Splunk to listen on more than one port (515,516,517 etc) and specify in inputs.conf on the receiving end that dictates the sourcetype to be used;

[udp://:514]
connection_host = dns
sourcetype = type_a

[udp://:515]
connection_host = dns
sourcetype = type_b

[udp://:516]
connection_host = dns
sourcetype = type_c

Unfortunately, you'd need to reconfigure a substantial number of your hosts to send to another port. And all data coming from each host would have the same sourcetype. (though you could override it, of course).

/K

kristian_kolb
Ultra Champion

Depending on your setup, i.e. if it's just the port that can't be changed, you might be able to do a similar operation, but by setting up a few syslog servers (rsyslog, splunk or other), and changing the destination ip on the sending hosts.

Then you install a forwarder on each syslog server... and set the sourcetype in inputs.conf. A bit messy - to say the least - for the initial setup, but hopefully fairly straightforward once it's set up.

/K

0 Karma

robf
Path Finder

thanks but unfortunately a number of devices cannot change their remote syslog port number

0 Karma

Ayn
Legend

Sorry, mechanisms like lookup are simply not available in that stage of the indexing pipeline. Lookups are pure search-time operations. Transforms like sourcetype rewriting, event filtering etc can be done with regexes only.

robf
Path Finder

thanks. so is the only way to do a huge regex as far as you know?

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...