How to separate logs from the same source to their respective sourcetypes based on regex?

[tayyujie]

I am running pfSense in my environment. Currently, I am sending logs through UDP 50000, and my source type is pfsense-firewall. My props.conf and transforms.conf look like this:

props.conf

[source::udp:50000]
TRANSFORMS-pfsense-firewall = pfsense-firewall
SHOULD_LINEMERGE = true
TRUNCATE = 0
MUST_NOT_BREAK_AFTER = pf: .* rule ([-\d]+\/\d+)\(.*?\):
MUST_BREAK_AFTER = pf: .* (<|>) +(\d+\.\d+\.\d+\.\d+)\.?(\d*)\:
REPORT-pfsense-firewall = pfsense-firewall

transforms.conf

[pfsense-firewall]
REGEX = .* (?pass|block) .* (?TCP|UDP|IGMP|ICMP) .* (?(\d+\.\d+\.\d+\.\d+))\.?(?(\d*)) [<|>] (?(\d+\.\d+\.\d+\.\d+))\.?(?(\d*)): (.*)
CLEAN_KEYS = 1
MV_ADD = 0

I am also currently sending virus logs from havp (pfSense antivirus package) through the same port. The logs from pfSense are multi-lined, but the logs from havp are single-lined.

A log sample from havp:

Oct 15 10:22:28 192.168.89.254 Oct 15 10:22:34 havp[2937]: 192.168.2.3 GET 200 http://www.eicar.org/download/eicar.com 380+68 VIRUS Clamd: Eicar-Test-Signature

How can I separate the logs to their respective source types based on the regex? Sorry, I'm pretty new to Splunk, so this may come across as a newbie question.

[martin_mueller · Accepted solution]

You could do something like this (untested, so double-check for typos before copypastaing):

props.conf:

[source::udp:50000]
sourcetype = havp
TRANSFORMS-set_pfsense_sourcetype = set_pfsense_sourcetype
...

transforms.conf:

[set_pfsense_sourcetype]
REGEX = [\r\n]
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::pfsense-firewall

...

The sourcetype will default to hvap as set in props.conf, and will be overwritten to pfsense in transforms.conf if a line break is present in the event.

Note, I'd personally prefer giving each source its own UDP port... but this should work as well.

[tayyujie]

Unfortunately, havp do not have the settings to send logs on a different port.

I'll give it a try!

[kml_uvce]

in props.conf make 2 stanzaz for pfsense and hvap

[pfsense_sourcetype]
.....
REPORT-pfsense-firewall = pfsense-firewall
[hvap_sourcetype]
....
REPORT-hvap-firewall = hvap-firewall

In transforms.conf

[pfsense-firewall]
 REGEX = .* (?pass|block) .* (?TCP|UDP|IGMP|ICMP) .* (?(\d+\.\d+\.\d+\.\d+))\.?(?(\d*)) [<|>] (?(\d+\.\d+\.\d+\.\d+))\.?(?(\d*)): (.*)
 CLEAN_KEYS = 1
 MV_ADD = 0
[ hvap-firewall]

[tayyujie]

I can make two stanzas even though my source is udp:50000?

I defined UDP port 50000 as pfsense-firewall in my data inputs.