I am looking to show I/O latency on our indexers specific to reads and/or writes? The Monitoring Console shows total IOPS but we'd like to go a little more granular than that. We want to know if our disk latency is because of reads or writes on our hot/warm and cold mounts. I'm looking at the introspection logs, at the fields indicated below, and it's not clear to me if reads_kb_ps and writes_kb_ps are the fields that will provide this data , based on their descriptions. avg_service_ms: Average time requests caused the CPU to be in use, in milliseconds. avg_total_ms: Average queue + execution time for requests to be completed, in milliseconds. cpu_pct: Percentage of time the CPU was servicing requests. device: Device name (e.g., as listed under /dev on UNIX). fs_type: Mounted device file system type. interval: Interval over which sampling occurred, in seconds. mount_point: Mount point(s) of the underlying device. reads_kb_ps: Total number of kb read per second. reads_ps: Number of read requests per second. writes_kb_ps: Total number of kb written per second. writes_ps: Number of write requests per second. I've looked all over and haven't been able to find anything helpful. I feel like someone else has to be doing this type of performance metric. This is what the Monitoring Console has as the IOPS search. How can I pick this apart to give me what I'm looking for? index=_introspection sourcetype=splunk_resource_usage component=IOStats host=<myhost> | eval mount_point = 'data.mount_point' | eval reads_ps = 'data.reads_ps' | eval writes_ps = 'data.writes_ps' | eval interval = 'data.interval' | eval op_count = (reads_ps + writes_ps) * interval | eval avg_service_ms = 'data.avg_service_ms' | eval avg_wait_ms = 'data.avg_total_ms' | eval cpu_pct = 'data.cpu_pct' | eval network_pct = 'data.network_pct' | timechart minspan=60s partial=f per_second(op_count) as iops, avg(data.cpu_pct) as avg_cpu_pct, avg(data.avg_service_ms) as avg_service_ms, avg(data.avg_total_ms) as avg_wait_ms, avg(data.network_pct) as avg_network_pct | eval iops = round(iops) | eval avg_cpu_pct = round(avg_cpu_pct) | eval avg_service_ms = round(avg_service_ms) | eval avg_wait_ms = round(avg_wait_ms) | eval avg_network_pct = round(avg_network_pct) | fields _time, iops avg_wait_ms | rename avg_wait_ms as "Wait Time (ms)" ... View more

I am currently planning on upgrading our Splunk Enterprise to version 6.5.2. I know I need to upgrade the Search Heads prior to the Indexers but I'm not sure what order everything else belongs in and am looking for a recommendation. We have 18 indexers, running version 6.4.1. We have 8 search heads in a cluster, running version 6.4.1. We have a deployer (Cluster Master), running version 6.4.1. We have a deployment server, running version 6.3.1. We have 4 heavy forwarders that we use as syslog-ng and snmptrapd servers, running versions 6.3.1 We have several standalone search heads, not in the cluster, that do our alerting and run Splunk DB Connect and/or the Splunk App for CEF, running in either 6.3.1 or 6.4.1. We have a mixed bag of Universal Forwarders running 5.x and 6.x versions. ... View more

We have the following logs coming into Splunk: {"log":"\u0009at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:113)\n","stream":"stdout","time":"2016-10-07T10:10:38.971217557Z"} {"log":"\u0009at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:683)\n","stream":"stdout","time":"2016-10-07T10:10:38.971213862Z"} {"log":"\u0009at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)\n","stream":"stdout","time":"2016-10-07T10:10:38.971206809Z"} Assume there are lots more log lines in that stack, how do we end up with the following in Splunk as one final log line? … at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80) at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:683) at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:113) .. Note how the final log line has been put into chronological order. Here is my Props.conf that I have set up: [my_sourcetype] #TIME_PREFIX = ^ #MAX_TIMESTAMP_LOOKAHEAD = 25 #TZ = GMT #TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N LINE_BREAKER = ([\r\n]+) SHOULD_LINEMERGE = True # 10000 is default, should be set on a case by case basis TRUNCATE = 0 KV_MODE = JSON # comment this line if they need to use PUNCT #ANNOTATE_PUNCT = false ... View more