Splunk Search

Why are my timechart results getting skewed?

bstreber
Path Finder

I have come across an issue with my timecharts.

When I do a search for all day on Feb 26th and check 9AM, I see 127 results. However, when I run a search for Feb 26th 8-10AM, the 9AM spot has over 2400 results.

I have confirmed that the value changes depending on the timeslot selected. For example, if I search for 00-10AM, I get a different result than the previous two. I have confirmed that the jobs are completing successfully and that multiple different searches are doing the same thing.

0 Karma
1 Solution

bstreber
Path Finder

I have solved the issue as it was caused by the dedups in my search. Since Splunk does not take time into consideration when removing duplicate results. Once I removed the dedups in my search it was working correctly.

View solution in original post

0 Karma

woodcock
Esteemed Legend

The dedup command will work with time if you add _time to the list of fields in your dedup arguments.

bstreber
Path Finder

This is amazing, thank you!!!

woodcock
Esteemed Legend

Don't forget to UpVote!

bstreber
Path Finder

I have solved the issue as it was caused by the dedups in my search. Since Splunk does not take time into consideration when removing duplicate results. Once I removed the dedups in my search it was working correctly.

0 Karma

woodcock
Esteemed Legend

Try this:

index="ngv" device.firmwareVersion="*" tuneStatus=FAILURE device.accountSourceId!=NULL assetClass!=DVR assetClass!=IPDVR
|dedup device.accountSourceId device.deviceId
|rename device.ecmMacAddress as DEVICE_MAC
| lookup xb6_accountid_mac.csv device.accountSourceId OUTPUT XB6_MAC 
| replace 00:00:00:00:00:00 with NULL 
| eval DEVICE_MAC=coalesce(XB6_MAC, BLUESKY_MAC, DEVICE_MAC)
| lookup FDB_BlueSky_Device_Details.csv DEVICE_MAC OUTPUT CITY, CMTS_DEVICE, LOCATION_ID, MANUFACTURER, MODEL, OPTICAL_RECEIVER_NAME 
| lookup CityToDACRegionMapping.csv CITY OUTPUT DAC_REGION
| search DAC_REGION=*
| timechart usenull=f useother=f span=10min count by DAC_REGION

But really, the problem is surely that you are getting a red exclamation-point triangle icon that warns you that your results are truncated because you have too many elements to plot. That means you need ti increase your span=10min to something higher/longer.

0 Karma

bstreber
Path Finder

Thanks for the reply,
I have confirmed that my searches are not getting truncated due to a high number of results. The only notifications I receive is about the implied implicit lookup of the tables I am using.

0 Karma

woodcock
Esteemed Legend

OK, so what do those messages say?

0 Karma

bstreber
Path Finder

Assuming implicit lookup table with filename 'CityToDACRegionMapping.csv'.

Assuming implicit lookup table with filename 'FDB_BlueSky_Device_Details.csv'.

Assuming implicit lookup table with filename 'xb6_accountid_mac.csv'.

Nothing out of the ordinary.

0 Karma

woodcock
Esteemed Legend

Yes, those are warnings for violations of best-practices but the configurations do work.

0 Karma

efavreau
Motivator

The span interval matters. Are you specifying the same span for each of these searches? If not, refer to the documentation on default time spans:
https://docs.splunk.com/Documentation/Splunk/7.2.4/SearchReference/Timechart#Default_time_spans

Try putting span=15m after your timechart commands in each of your searches. For example:

| timechart span=15m
###

If this reply helps you, an upvote would be appreciated.
0 Karma

bstreber
Path Finder

Hey I have set the span to 10 minutes. See the search attached below

index="ngv"
|fields device.accountSourceId, device.ecmMacAddress, device.firmwareVersion, tuneStatus, device.deviceType assetClass device.deviceId
|search device.firmwareVersion="*" tuneStatus=FAILURE device.accountSourceId!=NULL assetClass!=DVR assetClass!=IPDVR |dedup device.accountSourceId
|search |dedup device.deviceId
|rename device.ecmMacAddress as DEVICE_MAC
| lookup xb6_accountid_mac.csv device.accountSourceId OUTPUT XB6_MAC 
| replace 00:00:00:00:00:00 with NULL 
| eval DEVICE_MAC=coalesce(XB6_MAC, BLUESKY_MAC, DEVICE_MAC)
| lookup FDB_BlueSky_Device_Details.csv DEVICE_MAC OUTPUT CITY, CMTS_DEVICE, LOCATION_ID, MANUFACTURER, MODEL, OPTICAL_RECEIVER_NAME 
| lookup CityToDACRegionMapping.csv CITY OUTPUT DAC_REGION
| search DAC_REGION=*
| timechart usenull=f useother=f span=10min count by DAC_REGION
0 Karma

efavreau
Motivator

To confirm, the same search is being used both times, you have accounted for the span in the search, and all you are doing is changing the time in the GUI time picker? If so, then let's rule out the time picker.

Calculate time ranges that mean something to you. The documentation for that is here:
https://docs.splunk.com/Documentation/Splunk/7.2.4/SearchReference/SearchTimeModifiers

Example to isolate all of Feb 26:

index="ngv" earliest="02/26/2019:00:00:00" latest="02/27/2019:00:00:00"
###

If this reply helps you, an upvote would be appreciated.
0 Karma

bstreber
Path Finder

So for 8-10AM on the 26th I get 43 results at 9AM

index="ngv" earliest="02/26/2019:08:00:00" latest="02/26/2019:10:00:00"
|fields device.accountSourceId, device.ecmMacAddress, device.firmwareVersion, tuneStatus, device.deviceType assetClass device.deviceId
|search device.firmwareVersion="*" tuneStatus=FAILURE device.accountSourceId!=NULL assetClass!=DVR assetClass!=IPDVR |dedup device.accountSourceId
|search |dedup device.deviceId
|rename device.ecmMacAddress as DEVICE_MAC
| lookup xb6_accountid_mac.csv device.accountSourceId OUTPUT XB6_MAC 
| replace 00:00:00:00:00:00 with NULL 
| eval DEVICE_MAC=coalesce(XB6_MAC, BLUESKY_MAC, DEVICE_MAC)
| lookup FDB_BlueSky_Device_Details.csv DEVICE_MAC OUTPUT CITY, CMTS_DEVICE, LOCATION_ID, MANUFACTURER, MODEL, OPTICAL_RECEIVER_NAME 
| lookup CityToDACRegionMapping.csv CITY OUTPUT DAC_REGION
| search DAC_REGION=*
| timechart usenull=f useother=f span=10min count by DAC_REGION

And with all day I get 30 at 9AM

index="ngv" earliest="02/26/2019:00:00:00" latest="02/27/2019:00:00:00"
|fields device.accountSourceId, device.ecmMacAddress, device.firmwareVersion, tuneStatus, device.deviceType assetClass device.deviceId
|search device.firmwareVersion="*" tuneStatus=FAILURE device.accountSourceId!=NULL assetClass!=DVR assetClass!=IPDVR |dedup device.accountSourceId
|search |dedup device.deviceId
|rename device.ecmMacAddress as DEVICE_MAC
| lookup xb6_accountid_mac.csv device.accountSourceId OUTPUT XB6_MAC 
| replace 00:00:00:00:00:00 with NULL 
| eval DEVICE_MAC=coalesce(XB6_MAC, BLUESKY_MAC, DEVICE_MAC)
| lookup FDB_BlueSky_Device_Details.csv DEVICE_MAC OUTPUT CITY, CMTS_DEVICE, LOCATION_ID, MANUFACTURER, MODEL, OPTICAL_RECEIVER_NAME 
| lookup CityToDACRegionMapping.csv CITY OUTPUT DAC_REGION
| search DAC_REGION=*
| timechart usenull=f useother=f span=10min count by DAC_REGION
0 Karma

somesoni2
Revered Legend

What's the search you're using in each case (where you see different result)? Also share the selected time range for each search.

bstreber
Path Finder

Search is

index="ngv"
|fields device.accountSourceId, device.ecmMacAddress, device.firmwareVersion, tuneStatus, device.deviceType assetClass device.deviceId
|search device.firmwareVersion="*" tuneStatus=FAILURE device.accountSourceId!=NULL assetClass!=DVR assetClass!=IPDVR |dedup device.accountSourceId
|search |dedup device.deviceId
|rename device.ecmMacAddress as DEVICE_MAC
| lookup xb6_accountid_mac.csv device.accountSourceId OUTPUT XB6_MAC 
| replace 00:00:00:00:00:00 with NULL 
| eval DEVICE_MAC=coalesce(XB6_MAC, BLUESKY_MAC, DEVICE_MAC)
| lookup FDB_BlueSky_Device_Details.csv DEVICE_MAC OUTPUT CITY, CMTS_DEVICE, LOCATION_ID, MANUFACTURER, MODEL, OPTICAL_RECEIVER_NAME 
| lookup CityToDACRegionMapping.csv CITY OUTPUT DAC_REGION
| search DAC_REGION=*
| timechart usenull=f useother=f span=10min count by DAC_REGION

Time pick

Feb 26th to Feb 26th (at 9 AM 30 results) and then Feb 26th 08:00:00 to Feb 26th 10:00:00 (at 9 AM 43 results)

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...