topic Re: Filtering logs for a string only based on date in Splunk Search

Filtering logs for a string only based on date

shenoyveer — Wed, 11 Dec 2024 17:04:28 GMT

Hi All,

I have a requirement where I need to filter the virtual machine outage occurrence from the kernel logs.

I have sent kernel logs to splunk based on some pattern. Now I have a issue for filtering those values in splunk.

Here the requirement is, I need to filter the data only if one "string" has appeared in logs on same day.

example:

I have following logs in splunk

date1: hv_vmbus: registering driver hv_netvsc date1:hv_netvsc 000d3 eth0: VF dot 1 added date1:hv_netvsc 000d3 eth0: VF dot 2 added date1:hv_netvsc 000d3 eth0: VF dot 2 removed date1:hv_netvsc 000d3 eth0: VF dot 1 removed date2:hv_netvsc 000d3 eth0: VF dot 1 added date2:hv_netvsc 000d3 eth0: VF dot 2 added date2:hv_netvsc 000d3 eth0: VF dot 2 removed date2:hv_netvsc 000d3 eth0: VF dot 1 removed

I need to fetch the data for "dot" only if "hv_vmbus" pattern occured on same date. here I need only data in date1

I tried following query but it isn't working for me.

Requesting help for achieving this requirement.

Thanks,

Veeresh Shenoy

Re: Filtering logs for a string only based on date

ITWhisperer — Sun, 03 Nov 2024 12:28:41 GMT

If this doesn't work for you, please share some actual (anonymised) events so we can see what you are actually dealing with rather than a confusing set of pseudo events.

Re: Filtering logs for a string only based on date

shenoyveer — Mon, 04 Nov 2024 09:49:20 GMT

Thank you @ITWhisperer for quick solution.

Its working for me and doing some more tweaks in it.

Re: Filtering logs for a string only based on date

shenoyveer — Wed, 11 Dec 2024 17:01:06 GMT

Hi @ITWhisperer ,

I need small tweak in same query.

I am trying to filter the same data but it should give only data which shouldn't have "hv_vmbus" pattern in same day

Re: Filtering logs for a string only based on date

shenoyveer — Mon, 20 Jan 2025 15:12:02 GMT

Hi @ITWhisperer ,

I found your answer really helpful other day. now I am facing one small issue in it.

The query is adding the time(number of seconds) of previous occurrences in dashboard.

my requirement is, query should show the host name with date and number of seconds of downtime on that particular date.

current query is:

here basically I am calculating network card slot downtime which occured in servers with number of seconds

can you please help me with modifying the query?

Re: Filtering logs for a string only based on date

ITWhisperer — Mon, 20 Jan 2025 10:14:55 GMT

Please share some actual (anonymised) events so we can see what you are actually dealing with.

Also, provide an example of the type of output you are looking for.

Re: Filtering logs for a string only based on date

shenoyveer — Tue, 21 Jan 2025 17:01:13 GMT

Hi @ITWhisperer ,

Thanks for the reply .

Let me explain you my exact requirement.

Here I am trying to create a dashboard of visualizing and calculating downtime in VMs I manage.

I am trying to calculate based on log messages that are sending to splunk from servers.

Logs will have messages like

<timestamp> <nic-card-id> slot 1 removed

<timestamp> <nic-card-id> slot 3 added

I am calculating difference between 2 timestamps as a downtime and visualizing it.

Output dashboard I am expecting

Hostname, date , slot and the difference in time(downtime)

Current query is calculating the difference, but its adding previous downtime as well.

my query is, I want it to show the downtime in host on 2 different dates instead of adding it.

Can you please help me with tihs?

Re: Filtering logs for a string only based on date

ITWhisperer — Tue, 21 Jan 2025 18:02:26 GMT

What is you current query?

Re: Filtering logs for a string only based on date

shenoyveer — Tue, 21 Jan 2025 18:08:38 GMT

Hi @ITWhisperer

please find the current query:

Re: Filtering logs for a string only based on date

ITWhisperer — Wed, 22 Jan 2025 11:13:55 GMT

Please provide some sample events which demonstrate the issue you have with your search

Re: Filtering logs for a string only based on date

shenoyveer — Wed, 22 Jan 2025 14:26:05 GMT

with the current query it is calculating the downtime between the slot removed and added but the real problem is, its calculating previous downtime and adding the time and making it as single event.

my point is, I need the seperate events for every downtime in servers so looking for dashboard which should show hostname, date, slot and the downtime

Re: Filtering logs for a string only based on date

ITWhisperer — Wed, 22 Jan 2025 15:57:21 GMT

Please provide some sample events which demonstrate the issue you have with your search

Re: Filtering logs for a string only based on date

shenoyveer — Thu, 23 Jan 2025 16:49:54 GMT

I am not getting it.

you want me to share dashboard output?

Re: Filtering logs for a string only based on date

ITWhisperer — Thu, 23 Jan 2025 18:41:05 GMT

We can't see what might be wrong with your search if we can't see the actual events the search is running against. Please share some anonymised events which demonstrate the issue you are facing.

Re: Filtering logs for a string only based on date

shenoyveer — Fri, 24 Jan 2025 04:09:09 GMT

when running index="index1" | search "slot"

its giving below events. which has time, hostname as well.

events:

{"priority":6,"sequence":4704,"sec":695048,"usec":639227,"msg":"hv_netvsc 54243fd-13dc-6043-bddd-13dc6045bddd eth0: VF slot 1 added\n SUBSYSTEM=vmbus\n DEVICE=+vmbus:54243fd-13dc-6045-bddd-13dc6045bdda"}

{"priority":6,"sequence":4698,"sec":695037,"usec":497286,"msg":"hv_netvsc 54243fd-13dc-6043-bddd-13dc6045bddd eth0: VF slot 1 removed\n SUBSYSTEM=vmbus\n DEVICE=+vmbus:54243fd-13dc-6045-bddd-13dc6045bdda"}

my requirement is I need a difference of time between message removed and added for the particular day. i.e It should not add previous events.

Re: Filtering logs for a string only based on date

ITWhisperer — Mon, 27 Jan 2025 09:24:02 GMT

Try something along these lines

| rex field=msg "(?<action>added|removed)" | eval added_time=if(action="added",_time,null()) | eval removed_time=if(action="removed",_time,null()) | sort 0 _time | streamstats max(added_time) as added_time latest(removed_time) as removed_time by host slot_number | eval downtime=if(action="added",added_time-removed_time,null())

Re: Filtering logs for a string only based on date

shenoyveer — Tue, 28 Jan 2025 05:12:59 GMT

Thanks @ITWhisperer for the reply.

the downtime field is not getting populated only. I tried converting it to epoch time and still same.

can you please look into it once?

Re: Filtering logs for a string only based on date

shenoyveer — Tue, 28 Jan 2025 05:51:41 GMT

I have tried this in following way

index="index1" | search "slot" | rex field=msg "(?<action>added|removed)"
| eval added_time=if(action="added",strftime(_time, "%H:%M:%S"),null())
| eval removed_time=if(action="removed",strftime(_time, "%H:%M:%S"),null())
| sort 0 _time
| streamstats max(added_time) as added_time latest(removed_time) as removed_time by host slot
| eval downtime=if(isnotnull(added_time) AND isnotnull(removed_time), strptime(removed_time, "%H:%M:%S") - strptime(added_time, "%H:%M:%S"), 0)

but the issue is, downtime is not getting calculated and its printing 0 always.

need help in fixing this.

Re: Filtering logs for a string only based on date

ITWhisperer — Tue, 28 Jan 2025 08:25:04 GMT

Please share the event for which this is not working

Re: Filtering logs for a string only based on date

shenoyveer — Tue, 28 Jan 2025 17:17:34 GMT

Hi @ITWhisperer ,

I am getting same events which has "slot" messages

events:

{"priority":6,"sequence":4704,"sec":695048,"usec":639227,"msg":"hv_netvsc 54243fd-13dc-6043-bddd-13dc6045bddd eth0: VF slot 1 added

{"priority":6,"sequence":4698,"sec":695037,"usec":497286,"msg":"hv_netvsc 54243fd-13dc-6043-bddd-13dc6045bddd eth0: VF slot 1 removed

query used :

index="index1" | search "slot" | rex field=msg "(?<action>added|removed)"| eval added_time=if(action="added",strftime(_time, "%H:%M:%S"),null())| eval removed_time=if(action="removed",strftime(_time, "%H:%M:%S"),null())| sort 0 _time| streamstats max(added_time) as added_time latest(removed_time) as removed_time by host, slot| eval added_epoch=strptime(added_time, "%H:%M:%S")| eval removed_epoch=strptime(removed_time, "%H:%M:%S")| eval downtime=if(isnotnull(added_epoch) AND isnotnull(removed_epoch), removed_epoch - added_epoch, 0)

here I tried converting time to hour:min:sec and later into epoch to get the difference in seconds

but its not working and downtime is always showing 0