Lately my teammates have been searching for email logs and not finding them until several hours or even a day later.   I checked for latency issues by comparing log record times to _indextime but never found latency more than 20 minutes    index=proofpoint source=proofpoint_message_log | eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S") | eval delay=_indextime-_time | timechart span=1m max(delay)     so ingestion seems to have happened within 20 minutes of all records.  But my teammates cannot find some of the logs until hours later.   Has anyone here seen this sort of thing?  In the past I've only seen this as the result of delayed ingestion.  What else could cause it? We did recently move to SmartStore.  Is it possible that this is adding additional delay in Splunk determining which buckets to search?   ... View more

I have begun to accumulate some reference information about my company's AWS environment based on a bunch of queries.  Things like what accounts and VPCs, we have and when they were first seen (among other info).  Been happily accumulating this data into lookup tables, but now I realize that users on another Search Head Cluster would benefit from what I am doing on my SHC (which is reserved for Splunk ES) Lookup tables don't cut it anymore since they are maintained on the SHC so their data is not available to the other SHC. Is there a best practice on how to maintain such data so that it can be accessed from 2+ SHCs?  Some solutions that I can think of: Use a Summary Index.   Seems less than ideal because I am shooting for current state including some past info.  So using  a summary index would probably involve rewriting the current state of objects tracked -would not be the worst thing in the world to rewrite a few thousand entries daily, but I feel like an updatable source is more sensible.      Just build all of the KOs in each environment.  This incurs the cost of maintaining all KOs in each environment. are there other ways to approach what I want? (I'm really hoping that there is an answer like "you can make a KV store on the indexer") ... View more

Trying to create a sparkline from data in a lookup table monitor_user_traffic.csv has fields -user -traffic_dest_ip -app -bytes_out -time when I run | inputlookup monitor_user_traffic.csv | eval _time=time | stats sum(bytes_out) sparkline(sum(bytes_out),1d) as data_trend by user traffic_dest_ip app I get a value for "sum(bytes_out)" but nothing under "sparkline(sum(bytes_out),1d) as data_trend" Is there some sort of magical way that I need to alert my data for Splunk to be able to create a sparkline? ... View more

Lots of custom commands come with Splunk. 31 in the search app alone. I often see all of those commands and wonder if there is anything that I could be using. Since I am not an admin on the system, I can't just look at the code to find out. Is there a place that documents what these commands do and what sort of arguments they expect? ... View more

I have enabled the Network_Traffic data model with acceleration going back 32 days. After a recent Splunk upgrade to Splunk: 7.2.6 Splunk ES: 5.2.2 I noticed that my Network Traffic data model count volumes dropped off after going back about 2 weeks, despite an acceleration status of 100% complete. Found that by setting allowoldsummaries=t I could get all of the data. Tried re-indexing the data model. It seemed to pick up data further back, but still not all of it. I can check the model with this: | tstats summariesonly=t allow_old_summaries=t count as DMcountWithOld from datamodel=Network_Traffic.All_Traffic by span=1d _time | append [tstats summariesonly=t count as DMcountNoOld from datamodel=Network_Traffic.All_Traffic by span=1d _time ] | timechart span=1d sum(DMcountWithOld) as DMcountWithOld sum(DMcountNoOld) as DMcountNoOld and I still see results diverge starting about 2 weeks back Any ideas on why the data model acceleration fizzled out? If it matters, ES is on a 6 search head cluster ... View more