All Apps and Add-ons

Can I use regex to extract the host field for cisco:ios events?

damode
Motivator

I applied the below settings to parse data from 47 equallogic hosts, but I stopped get any logs altogether. Whereas, prior to these settings, I was getting logs but they were getting parsed into cisco:ios sourcetype.

transforms.conf
[set_sourcetype_equal_log_for_Equallogic_hosts]
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?).\w+\s+)[?(195.235.128.16\d|195.165.203.\d\d|195.165.201.21|10.40.40.\d)[\w.-]]?\s
FORMAT = sourcetype::equal_log
DEST_KEY = MetaData:Sourcetype

props.conf
[source::udp:514]
TRANSFORMS-changesourcetype = set_sourcetype_equal_log_for_Equallogic_hosts

Additionally, I even successfully tested the regex from regex101.com as well as by using the below command to check in Splunk
[index=* | head 1 | eval testdata="123456a" | regex testdata="^.{5,6}(?

0 Karma

landen99
Motivator

The best way to set the sourcetype is in inputs.conf

The best way to do regex on the host field is a transforms with SOURCE_KEY = MetaData:Host

The best way to set the host field is in inputs with a reference to a directory where the host information is stored.

Complicated regex is never recommended. And when regex can't be avoided, it can usually be simplified a lot. If the first three points have received due diligence, a link to the regex101 test plus the primary objectives of the regex should be provided. Currently, it appears that regex is not necessary.

0 Karma

damode
Motivator

Hi @landen99,

Thanks for your input.

considering 1st option, would the inputs.conf be in \etc\system\local and look like below ?

[udp://X.X.X.X:514]
index = main
sourcetype = equal_log
disabled = 0

in above case, would specifying all 47 IPs work ? I cant use wildcards because the IP range is very discontiguous in nature.

Would the 2nd option look something like this ?
transform.conf
[set_sourcetype_equal_log_for_Equallogic_hosts]
REGEX = 199.299.129.19\d|199.169.205.\d\d\d|199.165.200.22|20.90.90.\d
SOURCE_KEY = MetaData:Host
FORMAT = sourcetype::equal_log
DEST_KEY = MetaData:Sourcetype

I am not sure how the 3rd option is implemented. Can you please provide an example ?

0 Karma

landen99
Motivator

Your inputs can be there, but it is better in an app for organization/management.

Your transform extracts nothing and tries to use a list of matches, which is always bad without a good anchor.

I don't understand what you mean about my 3rd option.  I was providing guidance not options.

0 Karma

damode
Motivator

Hi @landen99, can you please help clarify the above solution ?

Thanks

0 Karma

agarrison
Path Finder

So you want to match all of those events and set them as a different source type?

That regex does not match those events at all, I would say the best way to do that would be to send the syslogs on a different port and setup a new input to a new sourcetype.

rather than that you could make a different regex.
It's a little messy, but something like this should match everything starting with that format:
REGEX = \w+\s\d+\s\d+:\d+:\d+\s\d+.\d+.\d+.\d+\s\d+:\d+:\w+:\d+-\w+-\d+\s\d+:\d{2}:\d{2}.\d{6}:

Like i said, changing the UDP port for the syslog would be the best bet, like port 4500 or something and then make a new input for UDP 4500 that goes to the new sourcetype.

0 Karma

agarrison
Path Finder

Try
\w+\s\d+\s\d+\d+:\d+:\d+\s(23.34.35.80|27.34.35.80)\s\d+:\d+:\w+:\d+-\w+-\d+\s\d+:\d{2}:\d{2}.\d{6}:

you could replace the addresses.
Maybe there could be a better solution than that though.

0 Karma

damode
Motivator

Hi @agarrison,

I cannot set a new udp port as Equallogic has no provision to specify a particular port. It sends on 514 by default. No other option than sourctype overriding.

I tried your regex but it parsed unexpected event as well. Example below,

Oct 20 06:53:26 x.x.x.x 59368:8696:agent:20-Oct-2017 06:53:25.653704:agent.c:1866:AUDIT:Unknown:22.7.1:SNMP packet validation failed, request received from x.x.x.x
Oct 20 06:00:37 x.x.x.x 2130:127:VolExec:20-Oct-2017 06:00:36.190088:VE_VolSetWorker.hh:151:WARNING::43.3.5:Volume MAIL has used 1 percent of its local replication reserve. If the in-use space exceeds the local replication reserve (set to 5 percent of the volume reserve), the group will cancel any in-progress replication for the volume.

0 Karma

sowings
Splunk Employee
Splunk Employee

The easiest solution would be to call the built-in "syslog-host" TRANSFORM, in addition to your "changesourcetype" TRANSFORM.

0 Karma

damode
Motivator

Hi @sowings,

did you mean the below way ?

[set_sourcetype_syslog_for_Equallogic_hosts]
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?).\w+\s+)[?(host1|host2|host3|host4|host5|host6)[\w.-]]?\s
FORMAT = sourcetype::equal_log
DEST_KEY = MetaData:Sourcetype

[source::udp:514]
TRANSFORMS-changesourcetype = set_sourcetype_equal_log_for_Equallogic_hosts
TRANSFORMS-changesourcetype = syslog-host

0 Karma

sowings
Splunk Employee
Splunk Employee

No; by having two stanzas with the same LHS (left-hand side) you introduce a collision, and Splunk must choose one of them to pick. IIRC it picks the first one it sees. In my case I was referring to something like this:

[source::udp:514]
TRANSFORMS-changesourcetype = set_sourcetype_equal_log_for_Equallogic_hosts, syslog-host

Transforms listed on the RHS (right-hand side) of a props.conf stanza are called in the order they appear, left to right. This means that your custom transform would be called first, then syslog-host after. Depending upon the structure of your logs, you may wish to reverse the order.

 

 

0 Karma

sowings
Splunk Employee
Splunk Employee

Close: The syntax you've provided would actually overwrite the settings (because the "TRANSFORMS-changesourctype" string is the same). Try this instead:


[source::udp:514]
TRANSFORMS-changesourcetype = set_sourcetype_equal_log_for_Equallogic_hosts, syslog-host

0 Karma

damode
Motivator

I tested it, it started parsing even the cisco:ios logs into equal_log sourcetype, which I guess, is due to including the default syslog-host transforms.

0 Karma

sowings
Splunk Employee
Splunk Employee

No, the "syslog-host" transform only attempts to change the 'host' field to the value found from the syslog preamble. If the logs are changing to the equal_log sourcetype for Cisco events, too, then your regex might need some tweaking.

0 Karma

damode
Motivator

So I created a regex and successfully tested it on regex101.com.

Regex = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?).\w+\s+)[?(x.x.x.2\d|x.x.x.1\d)[\w.-]]?\s

I used the above regex, and raw logs from Splunk. However, I noticed 2 main things that
there were some logs that that had no host IP in it and Splunk split the same event into two.

Example:
10/19/17
12:36:05.000 PM Oct 19 12:36:04 x.x.x.x 344237:342551:MgmtExec:19-Oct-2017 12:36:02.821184:targetAttr.cc:582:ERROR::7.4.3:iSCSI login to target 'x.x.x.x:3260, abc.2000-05.com.equallogic:8-jhduyc76-cjdbhc-jhd6hjs-dj78-iscsi-raid50-datastore8' from initiator 'x.x.x.x:49171, aaa.1000-05.com.microsoft:abc-netvault.mail.com' failed for the following reason:

10/19/17 Requested target not found.
12:36:05.000 PM

0 Karma

agarrison
Path Finder

Do you have some sample events?

0 Karma

agarrison
Path Finder

Something like:

\w+\s\d+\s\d+\d+:\d+:\d+\s(23.34.35.80|27.34.35.80)\s\d+:\d+:\w+:\d+-\w+-\d+\s\d+:\d{2}:\d{2}.\d{6}:

with a list of addresses, 23.34.35.80 and 27.34.35.80 being placeholders, you could add more. This is really not a great solution however.

0 Karma

damode
Motivator

Oct 18 08:49:31 X.X.X.80 251188:63:netmgtd:18-Oct-2017 08:49:29.341224:rca_ocptcp.c:655:AUDIT:grpadmin:25.7.4:GUI: Account grpadmin from X.X.X.X to X.X.X.8 logged out.
Sourcetype =equal_log
Oct 18 07:43:01 X.X.X.80 251003:61:netmgtd:18-Oct-2017 07:42:59.604422:rca_ocp.c:1372:AUDIT:grpadmin:25.7.3:GUI: Account grpadmin logged in from X.X.X.X to X.X.X.8, using local authentication. User privilege is group-admin.
Sourcetype =equal_log
Oct 18 06:00:52 X.X.X.62 2109:125:VolExec:18-Oct-2017 06:00:50.382206:VE_VolSetWorker.hh:151:WARNING::43.3.5:Volume MAIL has used 1 percent of its local replication reserve. If the in-use space exceeds the local replication reserve (set to 5 percent of the volume reserve), the group will cancel any in-progress replication for the volume.
Sourcetype =cisco:ios
Above is the latest sample event.

From the above events, I have noticed that its parsing correctly the event from equallogic, however, weird thing is I had put a host which ends in x.x.x.8, however, this log from host x.x.x.80 also seems to have parsed in sourcetype :equal_log. How did this happen ? I didnt event mention any wildcard.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...