About mwk

mwk · ‎12-26-2018

This would be one host going down on two different SAN fabrics. 12/6/18 4:47:14.000 AM Dec 6 04:47:14 UNIQUE-SWITCH-NAME-1 : 2018 Dec 6 04:47:15 CST: %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 12%$ Interface fc1/35 is down (Link failure loss of signal) hostname-XXXXX-fabric2 12/6/18 4:47:13.000 AM Dec 6 04:47:13 UNIQUE-SWITCH-NAME-2 : 2018 Dec 6 04:47:14 CST: %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 11%$ Interface fc1/35 is down (Link failure loss of signal) hostname-XXXXX-fabric1 Hopefully after a few minutes you'd get: 12/6/18 4:50:04.000 AM Dec 6 04:50:04 UNIQUE-SWITCH-NAME-3 : 2018 Dec 6 04:50:05 CST: %PORT-5-IF_UP: %$VSAN 11%$ Interface fc1/35 is up in mode F hostname-ZZZZZ-fabric1 12/6/18 4:50:02.000 AM Dec 6 04:50:02 UNIQUE-SWITCH-NAME-4: 2018 Dec 6 04:50:03 CST: %PORT-5-IF_UP: %$VSAN 12%$ Interface fc1/35 is up in mode F hostname-ZZZZZ-fabric2 Hostnames are unique by fabric because we add the last 2 octets of the hosts wwn to the hostname on that fabric - so myhost-departmental-abbrivation-89f5 would be on fabric 1 - myhost-departmental-abbrivation-ff87 would be on fabric 2. Now they can do go down and come up in a seemingly random order from splunks perspective because each SAN fabric is logically and physically unique and they don't have any concept of the others existence. So they come into splunk whenever they are logged and forwarded from each device, through rsyslog/splunkforwarder to the splunk indexer.

mwk · ‎12-06-2018

I've been trying things to figure this out for a few months now off and on. I get close but . . . and since my log output is different that anyone else's I've seen here, I'll put mine up here. I even had a Splunk instructor try this and he couldn't get it to do what I want it to do. I'm looking for: when did it happen, and how long was it down, and at some point I'd like to extend that out to: "hey this not only went down 6 hours ago, it's still down, I'll send an alert out" Here's a chunk of the log output. I've regex'd every piece of it six ways to Sunday trying to find something to transaction or stats on, but I can't figure it out. Each line represents a hosts path on a SAN fabric so each line is actually unique but in a good SAN environment we have two of everything and there is overlap in things like switch port names, error messages etc. So, for every host going down on a fabric message, there should be a message logged about that host on that fabric coming back up. And the same for the other fabric and no the fabrics don't know about each other (failure domain seperation etc.) 99% of the time, I don't care about the activity at all unless it looks like the host fell away on one fabric and not the other and then never came back. That's what I want to know about. 12/6/18 4:50:04.000 AM Dec 6 04:50:04 UNIQUE-SWITCH-NAME-1 : 2018 Dec 6 04:50:05 CST: %PORT-5-IF_UP: %$VSAN 11%$ Interface fc1/35 is up in mode F hostname-XXXXX-fabric1 12/6/18 4:50:02.000 AM Dec 6 04:50:02 UNIQUE-SWITCH-NAME-2: 2018 Dec 6 04:50:03 CST: %PORT-5-IF_UP: %$VSAN 12%$ Interface fc1/35 is up in mode F hostname-XXXXX-fabric2 12/6/18 4:50:04.000 AM Dec 6 04:50:04 UNIQUE-SWITCH-NAME-3 : 2018 Dec 6 04:50:05 CST: %PORT-5-IF_UP: %$VSAN 11%$ Interface fc1/35 is up in mode F hostname-ZZZZZ-fabric1 12/6/18 4:50:02.000 AM Dec 6 04:50:02 UNIQUE-SWITCH-NAME-4: 2018 Dec 6 04:50:03 CST: %PORT-5-IF_UP: %$VSAN 12%$ Interface fc1/35 is up in mode F hostname-ZZZZZ-fabric2 12/6/18 4:47:14.000 AM Dec 6 04:47:14 UNIQUE-SWITCH-NAME-1 : 2018 Dec 6 04:47:15 CST: %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 12%$ Interface fc1/35 is down (Link failure loss of signal) hostname-XXXXX-fabric1 12/6/18 4:47:13.000 AM Dec 6 04:47:13 UNIQUE-SWITCH-NAME-2 : 2018 Dec 6 04:47:14 CST: %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 11%$ Interface fc1/35 is down (Link failure loss of signal) hostname-XXXXX-fabric2 12/6/18 4:47:13.000 AM Dec 6 04:47:14 UNIQUE-SWITCH-NAME-3 : 2018 Dec 6 04:47:13 CST: %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 12%$ Interface fc1/36 is down (Link failure loss of signal) hostname-ZZZZZ-fabric1 12/6/18 4:47:13.000 AM Dec 6 04:47:13 UNIQUE-SWITCH-NAME-4 : 2018 Dec 6 04:47:14 CST: %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 11%$ Interface fc1/36 is down (Link failure loss of signal) hostname-ZZZZZZ-fabric2

mwk · ‎12-30-2014

So I seem to have gotten it all to play nice. props.conf talking to transforms.conf correctly now and here's the regex that's killing my annoying messages (at least it seems to be working. I've had this issue for so long I'll believe it when I don't see them for a day or two.) in the transforms.conf this syntax REGEX = (?s)(last).+?(message).+?(repeated.).+(times) seems to be the winner.

mwk · ‎12-30-2014

outputs.conf [tcpout] defaultGroup = default-autolb-group [tcpout:default-autolb-group] server = indexer.server.name:9997 [tcpout-server://indexer.server.name:9997] FWIW the transforms.conf and props.conf seem to be working now. I can get it to stop specific one word messages that I hand feed the rsyslog server from the monitored server but now of course I'm stuck in regex hell trying to get the correct combination to have it get rid of: 2014-12-30T14:12:03-06:00 last message repeated 447 times Currently trying this but it doesn't seem to be working. [setnull-server] REGEX = (?s)(last).+?(message).+?(repeated.) DEST_KEY = queue FORMAT = nullQueue

mwk · ‎12-30-2014

Blacklists are on a file level right? I need everything else that's being logged to the file. Just want some part of splunk to filter out the noise. FWIW rsyslog has trouble getting rid of this as well or it wouldn't even be something would see. 2014-12-30T12:11:16-06:00 last message repeated 3 times 2014-12-30T12:12:01-06:00 last message repeated 3 times 2014-12-30T12:12:53-06:00 last message repeated 3 times 2014-12-30T12:13:44-06:00 last message repeated 11 times 2014-12-30T12:14:29-06:00 last message repeated 6 times

mwk · ‎12-30-2014

This message is fairly repetitive (10's of thousands a month) in one of our systems and it's just taking up space in splunk. I've spent the better part of the last two days trying to get this to work. Using a heavy forwarder on the syslog server and trying to get it to "Discard specific events and keep the rest" Route and filter data I've got a props.conf on the heavy forwarder [source::/var/log/rsyslog/iswb1/syslog] TRANSFORMS-null= setnull and a transforms.conf on the heavy forwarder [setnull] REGEX = \[sshd\] DEST_KEY = queue FORMAT = nullQueue in /opt/splunk/etc/system/local . I even tried to just emulate the example to see if all the parts are connected to each other and working correctly. I used logger to send 'sshd' through the from the system that rsyslog is monitoring and that dutifully ends up in my splunk indexer. Obviously doing something wrong, probably a forest for the trees problem at this point.

Posts	6
Solutions	1
Karma Given	0
Karma Received	0
Member Since	‎08-05-2014

Online Status	Offline
Date Last Visited	‎10-17-2025 09:06 AM

Can you help me find the time between events?

I want to black hole 'last message repeated xxx ti...

Re: Can you help me find the time between events?

Can you help me find the time between events?

Re: I want to black hole 'last message repeated xx...

Re: I want to black hole 'last message repeated xx...

Re: I want to black hole 'last message repeated xx...

I want to black hole 'last message repeated xxx ti...

Join the Conversation