We've been trying to set up the CIM datamodels in our environment.   One that seems particularly useful is Network_Resolution (DNS).  Network Resolution has just one dataset for DNS and the events look like they intend to capture request and response data answer dest message_type query query_type reply code   src of these some are clearly related to the request and some are clearly related to an answer.   Is the idea that the datamodel should be populated with just query data (no answer) for message type=query? and just response data (no query/query type) for message_type=response? Or is there some effort that should be made to correlate the request and responses? so that each record contains everything where possible?   We did set up Splunk Stream to handle DNS.  Splunk stream tries to pair up request and response but seems to miss some pairings: I find that about 30% of DNS events from Splunk Stream have both request and response, ~40% have only query, and ~30% have only response so it would be possible to add 30% of DNS events to Network Resolution with both query and response, but then the rest would have to be just query or just response events -- missing the other fields.  Or I could just split out the query and response from the joint records.  Does anyone who uses Network Resolution (especially in ES) have a recommendation on how to do this? ... View more

Lately my teammates have been searching for email logs and not finding them until several hours or even a day later.   I checked for latency issues by comparing log record times to _indextime but never found latency more than 20 minutes      index=proofpoint source=proofpoint_message_log | eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S") | eval delay=_indextime-_time | timechart span=1m max(delay)       so ingestion seems to have happened within 20 minutes of all records.  But my teammates cannot find some of the logs until hours later.   Has anyone here seen this sort of thing?  In the past I've only seen this as the result of delayed ingestion.  What else could cause it? We did recently move to SmartStore.  Is it possible that this is adding additional delay in Splunk determining which buckets to search?   ... View more

I have begun to accumulate some reference information about my company's AWS environment based on a bunch of queries.  Things like what accounts and VPCs, we have and when they were first seen (among other info).  Been happily accumulating this data into lookup tables, but now I realize that users on another Search Head Cluster would benefit from what I am doing on my SHC (which is reserved for Splunk ES) Lookup tables don't cut it anymore since they are maintained on the SHC so their data is not available to the other SHC. Is there a best practice on how to maintain such data so that it can be accessed from 2+ SHCs?  Some solutions that I can think of: Use a Summary Index.   Seems less than ideal because I am shooting for current state including some past info.  So using  a summary index would probably involve rewriting the current state of objects tracked -would not be the worst thing in the world to rewrite a few thousand entries daily, but I feel like an updatable source is more sensible.      Just build all of the KOs in each environment.  This incurs the cost of maintaining all KOs in each environment. are there other ways to approach what I want? (I'm really hoping that there is an answer like "you can make a KV store on the indexer") ... View more

Trying to create a sparkline from data in a lookup table monitor_user_traffic.csv has fields -user -traffic_dest_ip -app -bytes_out -time when I run | inputlookup monitor_user_traffic.csv | eval _time=time | stats sum(bytes_out) sparkline(sum(bytes_out),1d) as data_trend by user traffic_dest_ip app I get a value for "sum(bytes_out)" but nothing under "sparkline(sum(bytes_out),1d) as data_trend" Is there some sort of magical way that I need to alert my data for Splunk to be able to create a sparkline? ... View more