Powermax: Why does search bring in log entries from last year instead of only last 24hrs?

[Dalidavinci_Wor]

We have data coming in and we are still searching for a best practice on what alerts to monitor, however, my question is on the query below: 

index="storage_vmax" sourcetype="dellemc:vmax:rest" type=ARRAY severity = FATAL
|search (severity!=NORMAL AND severity!=INFORMATION)
| stats count by _time,created_date,source,reporting_level,severity,asset_id,array_id,type, state,description

Where I would like to bring in only what was created in the last 24 hours.. The problem with the existing query is that it is bring in created log entries from a year ago which are stale. If we are going to have SNOW open tickets we do not want it to so on stale data only new.

Thanks,

Dali

[Dalidavinci_Wor]

I am getting better results from the below query, but I would think there is a better way of excluding all except recent 1 day

ndex="storage_vmax" sourcetype="dellemc:vmax:rest" type=ARRAY severity = * | search (severity!=NORMAL AND severity!=INFORMATION) created_date!="*-*-2021 *:*:*.*" | stats count by _time,source,created_date,reporting_level,severity,asset_id,array_id,type, state,description

[Dalidavinci_Wor]

I will take this up with SPLUNK Admin to check for 

" data's sourcetype in the props.conf"

Thanks!

[Dalidavinci_Wor]

Hi, Yes, I am running this in the following timestamp "

Complete 2,058 events (4/21/22 12:00:00.000 AM to 4/22/22 1:28:17.000 PM)" and it still shows data from last year

 

created_date source reporting_level severity asset_id array_id type state description count
2022-04-21 02:00:00 Jul-18-2021 20:37:33.000

[VatsalJagani]

That does mean incorrect timestamp parsing. Please review the timestamp parsing related parameters (refer to the document that I mentioned in my answer) for your data's sourcetype in the props.conf

[PickleRick]

It does not have to mean incorrect time parsing. As far as I can see, there is more than one time-related field whereas _time can be parsed obviously from only one from them. It's up to @Dalidavinci_Wor 's team to decide which field is most important and should indeed be used as _time value in the events.

With the other fields - you can filter on them (they might need strptime()-ing if they're not properly calculated from the event as they are) but you have to do it explicitly using "| where" condition and have to remember that the timepicker or earliest/latest parameters for the search only limit _time field, not any other field that might contain a timestamp value. So the limits might be completely unrelated. You might get - with a search for last 15 minutes - events pertaining to something that - for example started long ago (like a created_date for a ticket which was created a year ago but changes status 5 minutes ago so you have an event from 5 minutes ago but the ticket's timestamp included in the event would be from a year ago).

[Dalidavinci_Wor]

So does this  still go back to  changing the below?

" data's sourcetype in the props.conf"

Or can I run a query where created date or created_date is less than 2 days? 

Presntly I have the below:

index="storage_vmax" sourcetype="dellemc:vmax:rest" type=ARRAY severity = FATAL
|search (severity!=NORMAL AND severity!=INFORMATION)
| stats count by _time,source,created_date,reporting_level,severity,asset_id,array_id,type, state,description

[PickleRick]

Depends whether you do have multiple timestamps in your logs or not. If your events contain only a single timestamp, it should be parsed into the _time field. This way you can easily limit your search using earliest/latest conditions or timerange picker.

But if your events do have another field which is parsed into _time and additionally have this created_date field, you must limit that created_date in a different way.

The first part of your search stays mostly the same (the question is if your severity field is multivalue or single valued because you're already searching only for severity=FATAL so there's not much point of filtering out the NORMALs and INFORMATIONs unless you have multivalued field).

index="storage_vmax" sourcetype="dellemc:vmax:rest" type=ARRAY severity = FATAL

Then you have to parse a timestamp from your created_date field

| eval cdate=strptime(created_date,"appropriate format for your date")

See the https://docs.splunk.com/Documentation/SplunkCloud/latest/SearchReference/Commontimeformatvariables article for the format variables to use depending on your date format in the created_date field

Then you can limit your search only to those events where this timestamp is bigger (not earlier than) now() - your desired interval. For example, for getting only last 24h of events you'd use

| where cdate>= now()-86400

And now you can do your stats

| stats count by _time,source,created_date,reporting_level,severity,asset_id,array_id,type, state,description

The caveat in this solution is that since your created_date is unrelated to the _time field, you might have to look from a bigger timerange to make sure to get all events containing your desired created_date values.

[VatsalJagani]

@Dalidavinci_Wor  - Are you running the query in a time range of the last 24 hours?

• If the answer is yes and still you are seeing the old data that means your data you have not extracted the timestamp appropriately.
• How will you verify - See _raw (raw data) vs _time field to see what is extracted as a value for _time vs what is present in the raw data.
• See this document for proper timestamp parsing - https://docs.splunk.com/Documentation/StreamProcessor/standard/FunctionReference/Timestamp

------
I hope this helps!!!