topic Re: Comparing value of same fields derrived from Rex command in Splunk Search

How to compare the value of same fields derived from rex command?

thaghost99 — Tue, 08 Mar 2022 21:46:37 GMT

hi i am hoping for some help regarding this. basically i would like to compare (subtract current to previous) the value of REX command on the latest data versus previous events REX command data.

today

Counters: Reset Uptime Lifetime
Messages Received 13,524,598 13,524,585 13,524,598

Yesterday

Counters: Reset Uptime Lifetime
Messages Received 12,524,598 12,524,585 12,524,598

current filter

| rex field="status detail" "(?<message_received_name>Messages Received)\\s*[0-9,]*\s*[0-9,]*\s*(?<message_received>[0-9,]*)"
| rex field="status detail" "(?<current_time_text>Status as of:)\s*(?<query_time>.*)GMT"
| stats latest(message_received_name) as Counter_Name latest(message_received) as Messages_Received latest(query_time) as Query_Time by Hostname

how can i use the same search on the previous event, so i can find the difference of "message_received"

thanks,

Comparing value of same fields derrived from Rex command

bowesmana — Tue, 08 Mar 2022 21:53:21 GMT

If you are searching over the entire date range, than when you do your stats you can bin by _time and then calculate the difference using delta, streamstats or autoregress, e.g. this example generates numbers of 4 days and then calculates the deltas of those numbers.

The same search, but replace the last line with

| streamstats window=1 current=f values(*) as prev_* | foreach prev_* [ eval Diff_<<MATCHSTR>>='<<MATCHSTR>>'-<<FIELD>> ]

shows how to do it with streamstats

Not sure how that fits into your dataset though, but hopefully the example helps

Re: Comparing value of same fields derrived from Rex command

thaghost99 — Tue, 08 Mar 2022 22:05:54 GMT

hi @bowesmana 🙂

thanks for the swift response, but i think i got a little more confused. hehe

i am trying to extract the value of "messages received" for "lifetime" using the REX command.

but i would like to compare this LIFETIME value from the previous event (lets assume last event is an hour ago), using the same REX command. i would then like to subtract the value of 'LIFETIME' on the current event and the Previous event.

this would be so much easier if i could make message receive, lifetime as its own field, but this is from a CSV file which is mix with a ton of other data on the same field.

Counters: Reset Uptime Lifetime
Messages Received 12,524,598 12,524,585 12,524,598

Counters: Reset Uptime Lifetime
Messages Received 13,524,598 13,524,585 13,524,598

so "Message Received" 13,524,598 - 12,524,598 = 1,000,000

so i want to get the 1 million as the final value of message received

Re: Comparing value of same fields derrived from Rex command

bowesmana — Tue, 08 Mar 2022 22:23:54 GMT

The rex is not important - it's just extracting a field from your data.

I assumed when you posted "today" and "yesterday", you had some existing daily data.

So, this CSV - are you saying that the 'previous' event could be ANY time interval prior to the current?

Can you show an example of your data and any existing query. If you say it's a CSV - are you doing inputlookup to get the data in the first place?

Doing the calculation is easy with the streamstats example I gave - but what comes before is important as to whether it works for you.

Either delta or streamstats will be the solution, but I'd like to better understand what your output looks like, as the stats command you show would not result in the 'Counters:' example, so I'm not sure what will work for your data

Re: Comparing value of same fields derrived from Rex command

thaghost99 — Tue, 08 Mar 2022 22:36:43 GMT

@bowesmana

thanks,

splunk is currently monitoring a directory and ingesting as data gets put in there. so at the moment its daily schedule, but we can trigger the same script anytime during the day which will put the new csv in the same directory.

my goal is to capture based on previous event from the current one. so the previous event can be the previous day, or if script is triggered manually, the event can be in the previous hour. i just need to compare it with the previous event (dont matter how long it is assuming my time range would show at least 2 events).

data looks something like this.

currently this is what my output looks like. which basically just spits out the current number.

i want to show the Delta

Re: Comparing value of same fields derrived from Rex command

bowesmana — Wed, 09 Mar 2022 21:59:14 GMT

Try this

your rex_statements | reverse | streamstats window=1 current=f global=f values(message_received) as prev_message_received by Hostname | stats latest(message_received_name) as Counter_Name latest(message_received) as Messages_Received latest(query_time) as Query_Time latest(prev_message_received) as prev_Messages_Received by Hostname | eval diff=Messages_Received-prev_Messages_Received

The reverse/streamstats will copy the previous message_received into a new field in the current event called prev_message_received. Then the stats gets both message_received and prev_message_received and finally the diff calculates the difference between the two.

Hope this helps