Filtering logs for a string only based on date

[shenoyveer]

Hi All,

I have a requirement where I need to filter the virtual machine outage occurrence from the kernel logs.

I have sent kernel logs to splunk based on some pattern. Now I have a issue for filtering those values in splunk.

Here the requirement is, I need to filter the data only if one "string" has appeared in logs on same day.

example:

I have following logs in splunk

date1: hv_vmbus: registering driver hv_netvsc

date1:hv_netvsc 000d3 eth0: VF dot 1 added

date1:hv_netvsc 000d3 eth0: VF dot 2 added

date1:hv_netvsc 000d3 eth0: VF dot 2 removed

date1:hv_netvsc 000d3 eth0: VF dot 1 removed

date2:hv_netvsc 000d3 eth0: VF dot 1 added

date2:hv_netvsc 000d3 eth0: VF dot 2 added

date2:hv_netvsc 000d3 eth0: VF dot 2 removed

date2:hv_netvsc 000d3 eth0: VF dot 1 removed

I need to fetch  the data for "dot" only if "hv_vmbus" pattern occured on same date. here I need only data in date1

I tried following query but it isn't working for me.

"index="index0" | search "dot" | rex field=msg "VF\s+dot\s+(?<dot_number>\d+)" | dedup msg | sort _time,host | stats range(_time) as n1 by host,dum_number"

Requesting help for achieving this requirement.

Thanks,

Veeresh Shenoy

[ITWhisperer]

index="index0" 
``` Assuming you are actually searching _raw and that date has already been extracted ```
| rex "(?<vmbus>vmbus)"
| eventstats values(vmbus) as vmbus by date
| where vmbus="vmbus"
| search "dot" | rex field=msg "VF\s+dot\s+(?<dot_number>\d+)" | dedup msg | sort _time,host | stats range(_time) as n1 by host,dum_number"

If this doesn't work for you, please share some actual (anonymised) events so we can see what you are actually dealing with rather than a confusing set of pseudo events.

[shenoyveer]

Hi @ITWhisperer ,

I found your answer really helpful other day. now I am facing one small issue in it.

The query is adding the time(number of seconds) of previous occurrences in dashboard.

my requirement is, query should show the host name with date and number of seconds of downtime on that particular date.

current query is: 

index="index1" |search "slot" | rex field=msg "VF\s+slot\s+(?<slot_number>\d+)" | dedup msg | sort _time,host | stats range(_time) as downtime by host,slot_number

here basically I am calculating network card slot downtime which occured in servers with number of seconds

can you please help me with modifying the query?

[ITWhisperer]

Please share some actual (anonymised) events so we can see what you are actually dealing with.

Also, provide an example of the type of output you are looking for.

[shenoyveer]

Hi @ITWhisperer ,

Thanks for the reply .

Let me explain you my exact requirement.

Here I am trying to create a dashboard of visualizing and calculating downtime in VMs I manage.

I am trying to calculate based on log messages that are sending to splunk from servers.

Logs will have messages like 

<timestamp> <nic-card-id> slot 1 removed

<timestamp> <nic-card-id> slot 3 added

I am calculating difference between 2 timestamps as a downtime and visualizing it.

Output dashboard I am expecting

Hostname, date , slot and the difference in time(downtime)

Current query is calculating the difference, but its adding previous downtime as well.

my query is, I want it to show the downtime in host on 2 different dates instead of adding it.

Can you please help me with tihs?

[ITWhisperer]

What is you current query?

[shenoyveer]

Hi @ITWhisperer 

please find the current query:

index="index1" |search "slot" | rex field=msg "VF\s+slot\s+(?<slot_number>\d+)" | dedup msg | sort _time,host | stats range(_time) as downtime by host,slot_number

[ITWhisperer]

Please provide some sample events which demonstrate the issue you have with your search

[shenoyveer]

with the current query it is calculating the downtime between the slot removed and added but the real problem is, its calculating previous downtime and adding the time and making it as single event.

my point is, I need the seperate events for every downtime in servers so looking for dashboard which should show hostname, date, slot and the downtime

[ITWhisperer]

Please provide some sample events which demonstrate the issue you have with your search

[shenoyveer]

I am not getting it.

you want me to share dashboard output?

[ITWhisperer]

We can't see what might be wrong with your search if we can't see the actual events the search is running against. Please share some anonymised events which demonstrate the issue you are facing.

[shenoyveer]

when running index="index1" | search "slot" 

its giving below events. which has time, hostname as well.

events:

{"priority":6,"sequence":4704,"sec":695048,"usec":639227,"msg":"hv_netvsc 54243fd-13dc-6043-bddd-13dc6045bddd eth0: VF slot 1 added\n SUBSYSTEM=vmbus\n DEVICE=+vmbus:54243fd-13dc-6045-bddd-13dc6045bdda"}

{"priority":6,"sequence":4698,"sec":695037,"usec":497286,"msg":"hv_netvsc 54243fd-13dc-6043-bddd-13dc6045bddd eth0: VF slot 1 removed\n SUBSYSTEM=vmbus\n DEVICE=+vmbus:54243fd-13dc-6045-bddd-13dc6045bdda"}

my requirement is I need a difference of time between message removed and added for the particular day. i.e It should not add previous events.

[ITWhisperer]

Try something along these lines

| rex field=msg "(?<action>added|removed)"
| eval added_time=if(action="added",_time,null())
| eval removed_time=if(action="removed",_time,null())
| sort 0 _time
| streamstats max(added_time) as added_time latest(removed_time) as removed_time by host slot_number
| eval downtime=if(action="added",added_time-removed_time,null())

[shenoyveer]

I have tried this in following way

index="index1" | search "slot" | rex field=msg "(?<action>added|removed)"
| eval added_time=if(action="added",strftime(_time, "%H:%M:%S"),null())
| eval removed_time=if(action="removed",strftime(_time, "%H:%M:%S"),null())
| sort 0 _time
| streamstats max(added_time) as added_time latest(removed_time) as removed_time by host slot
| eval downtime=if(isnotnull(added_time) AND isnotnull(removed_time), strptime(removed_time, "%H:%M:%S") - strptime(added_time, "%H:%M:%S"), 0)

but the issue is, downtime is not getting calculated and its printing 0 always.

need help in fixing this.

[ITWhisperer]

Please share the event for which this is not working

[shenoyveer]

Hi @ITWhisperer ,

I am getting same events which has "slot" messages

events:

{"priority":6,"sequence":4704,"sec":695048,"usec":639227,"msg":"hv_netvsc 54243fd-13dc-6043-bddd-13dc6045bddd eth0: VF slot 1 added

{"priority":6,"sequence":4698,"sec":695037,"usec":497286,"msg":"hv_netvsc 54243fd-13dc-6043-bddd-13dc6045bddd eth0: VF slot 1 removed

 query used :

index="index1" | search "slot" | rex field=msg "(?<action>added|removed)"| eval added_time=if(action="added",strftime(_time, "%H:%M:%S"),null())| eval removed_time=if(action="removed",strftime(_time, "%H:%M:%S"),null())| sort 0 _time| streamstats max(added_time) as added_time latest(removed_time) as removed_time by host, slot| eval added_epoch=strptime(added_time, "%H:%M:%S")| eval removed_epoch=strptime(removed_time, "%H:%M:%S")| eval downtime=if(isnotnull(added_epoch) AND isnotnull(removed_epoch), removed_epoch - added_epoch, 0)

here I tried converting time to hour:min:sec and later into epoch to get the difference in seconds 

but its not working and downtime is always showing 0

[ITWhisperer]

You seem to have removed the parsing of the slot - also, try using epoch times and not converting them to strings (as this is unnecessary)

index="index1"
| search "slot" 
| rex field=msg "VF\s+slot\s+(?<slot_number>\d+)"
| rex field=msg "(?<action>added|removed)"
| eval added_epoch=if(action="added",_time,null())
| eval removed_epoch=if(action="removed",_time,null())
| sort 0 _time
| streamstats max(added_epoch) as added_epoch latest(removed_epoch) as removed_epoch by host, slot_number
| eval downtime=if(isnotnull(added_epoch) AND isnotnull(removed_epoch), removed_epoch - added_epoch, 0)

[shenoyveer]

Thanks @ITWhisperer  for the reply.

the downtime field is not getting populated only. I tried converting it to epoch time and still same.

can you please look into it once?

[shenoyveer]

Hi @ITWhisperer ,

I need small tweak in same query.

I am trying to filter the same data but it should give only data which shouldn't have "hv_vmbus" pattern in same day

[shenoyveer]

Thank you @ITWhisperer  for quick solution.

Its working for me and doing some more tweaks in it.