Here is an example using the splunkd_access.log <dashboard> <label>API Calls</label> <row> <panel> <chart> <search> <query>index=_internal source=*splunkd_access.log | timechart count by method</query> <earliest></earliest> <latest></latest> </search> <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option> <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option> <option name="charting.axisTitleX.visibility">visible</option> <option name="charting.axisTitleY.visibility">visible</option> <option name="charting.axisTitleY2.visibility">visible</option> <option name="charting.axisX.scale">linear</option> <option name="charting.axisY.scale">linear</option> <option name="charting.axisY2.enabled">0</option> <option name="charting.axisY2.scale">inherit</option> <option name="charting.chart">line</option> <option name="charting.chart.bubbleMaximumSize">50</option> <option name="charting.chart.bubbleMinimumSize">10</option> <option name="charting.chart.bubbleSizeBy">area</option> <option name="charting.chart.nullValueMode">gaps</option> <option name="charting.chart.sliceCollapsingThreshold">0.01</option> <option name="charting.chart.stackMode">default</option> <option name="charting.chart.style">shiny</option> <option name="charting.drilldown">all</option> <option name="charting.layout.splitSeries">0</option> <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option> <option name="charting.legend.placement">right</option> </chart> </panel> </row> </dashboard> ... View more

also reports of customers hitting this issue while trying to upgrade from 6.2.2 to 6.2.3 ... View more

it sounds like you want to extract a portion of the URL into its own field correct? /v0.1/first ... View more

Can you indicate what device this is from and provide similar output for all downstream splunk instances from the UF? If the downstream nodes have blocked queues that would be the place to start troubleshooting. ... View more

So you are seeing blocked typing queues on your heavy forwarder or indexers? This is typically an indication of heavy regex replacement / field extraction / routing (via props & transforms). I would look in props & transforms.conf on these splunk instances and see if you can offload some of that processing of regex & field extractions to the search heads. Increasing the queue size isn't the right direction to go in for resolving this. If there are other queues blocked the troubleshooting approach would be different. What is your output of this search: index=_internal host=hosttoinvestigate source=*metrics.log group=queue blocked=true | stats count by name ... View more

It sounds like your downstream heavy forwarder (or further HF / indexers downstream) queues could be full / blocked which is why it is refusing additional connections, which would also explain the intermittent nature of the problem as congestion could come and go. Try running this search on each of the downstream splunk instances from the UF and see which queues are blocked. index=_internal host=hosttoinvestigate source=*metrics.log group=queue blocked=true | timechart count limit=0 by name ... View more

You need some method of differentiating the devices from one another, tagging is one way that could work. I have seen companies implement a global naming standard to indicate the type of the device in the host & dns name, which could then be extracted from the host field into its own field. Then your search would just include the new field (i.e.: type=switch). Are all of your events syslog? If the format of the events are different between device types you could use props & transforms to change the sourcetype based on a regex you match in the event. I would bet the load balancer syslog looks different from the router/switch syslog however if the router & switch syslog look similar in format your best bet would be the tagging or renaming of the device. ... View more

you should know the value of the index field and sourcetype field to narrow down your search. If you did not specify either of these in inputs.conf on the forwarder the events will default into the main index (index=main). a basic search with the "key words" you are after would look like this: index=x sourcetype=y "failed logins"| stats count by host index=main "failed logins"| stats count by host use the time picker in the flash timeline to define the time period to search over. ... View more

@dfronck, did this configuration above on your HF work out for you? ... View more

you must have the index already configured on the indexers in indexes.conf before sending events to the new index. ... View more

running this search will give you disk consumption by index & splunk_server: | rest /services/data/indexes | eval indexSizeGB = if(currentDBSizeMB > 1, round(currentDBSizeMB / 1024, 2), null()) | rename title AS index | stats first(indexSizeGB) AS "Disk Usage (GB)" by index, splunk_server NOTE: This Information is exposed in the "Indexes & Volumes" views of the Distributed Management Console as of 6.3. ... View more

Is your data being sourcetyped correctly? i.e.: does the sourcetype field return a value of hadoop for these events? If so I would add a field extraction definition to the hadoop sourcetype stanza in props.conf on your search head: $SPLUNK_HOME/etc/system/local/props.conf (or props.conf in the app of your choice) $SPLUNK_HOME/etc/apps/appofyourchoice/local/props.conf [hadoop] EXTRACT-client_host = (?m)client_host":"(?<client_host>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"} then restart splunk on your SH: $SPLUNK_HOME/bin ./splunk restart to validate, you can run a search like this: index=* sourcetype=hadoop | stats count by client_host If not and you want to do the extraction on the source field, this should work: on your search head: $SPLUNK_HOME/etc/system/local/props.conf (or props.conf in the app of your choice) $SPLUNK_HOME/etc/apps/appofyourchoice/local/props.conf [source::/myprovider/syslogs/...] EXTRACT-client_host = (?m)client_host":"(?<client_host>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"} ... View more

Try setting the TIME_PREFIX since there is a , in front of the date. This should work: $SPLUNK_HOME/etc/system/local/props.conf on indexers: [your_sourcetype] TIME_FORMAT=%d/%m/%Y TIME_PREFIX=, MAX_TIMESTAMP_LOOKAHEAD=10 SHOULD_LINEMERGE=false NO_BINARY_CHECK=true ... View more

http://answers.splunk.com/answers/234717/how-to-get-list-of-buckets-which-are-having-issues.html#answer-237983 ... View more

This search comes courtesy of my co-worker @Masa Clustering Multi-site enabled Simple version of bucket state by site | rest /services/cluster/master/buckets | rex field=title "^(?[^\~]+)" | search repl_index="*" standalone=0 frozen=0 | rename title AS bucketID | fields bucketID *origin_site* *_by_site* | untable bucketID siteState value | rex mode=sed field=siteState "s/\./__/" | rex mode=sed field=siteState "s/_count_/_/" | search NOT siteState=primaries_* | xyseries bucketID siteState value | fields - search_by_site | fillnull | eval rep_total= rep_by_site__site1 + rep_by_site__site2 + rep_by_site__site3 | eval srch_total = search_by_site__site1 + search_by_site__site2 + search_by_site__site3 | rename constrain_to_origin_site AS constrain | rename origin_site AS origin | rename rep_by_site__site1 AS rep_site1 | rename rep_by_site__site2 AS rep_site2 | rename rep_by_site__site3 AS rep_site3 | rename search_by_site__site1 AS srch_site1 | rename search_by_site__site2 AS srch_site2 | rename search_by_site__site3 AS srch_site3 table output: bucketID constrain origin rep_site1 rep_site2 rep_site3 rep_total srch_site1 srch_site2 srch_site3 srch_total _audit~118~FF782A13-8AFB-4617-BCB4-15ED11928DD7 0 site1 2 1 1 4 2 1 1 4 _audit~119~FF782A13-8AFB-4617-BCB4-15ED11928DD7 0 site1 2 1 2 5 2 1 1 4 You can further filter out for buckets where rep or search factor is not met (assuming your rep factor=4 and search factor=3) by appending this to the end of the search: | search rep_total<4 OR srch_total<3 Note: remove references to site3 in the search if you only have 2 sites in the multi-site cluster Clustering Multi-site enabled | rest /services/cluster/master/buckets | rex field=title "^(?[^\~]+)" | search repl_index="*" standalone=0 frozen=0 | rename title AS bucketID | fields bucketID peers.*.search_state *site* | untable bucketID siteState value | rex field=siteState "peers\.(?[^\.]*?)\.(?search_state)" | rex field=siteState "(?primaries_by_site)\.(?\S+)" | rex field=siteState "(?rep_count_by_site)\.(?\S+)" | rex field=siteState "(?search_count_by_site)\.(?\S+)" | eval peerGUID=if(siteState=="primaries_by_site", value, peerGUID) | eval site=if(siteState=="origin_site", value, site) | eval value=if(siteState=="search_count_by_site", site + ":" + value, value) | eval value=if(siteState=="rep_count_by_site", site + ":" + value, value) | join type=outer peerGUID [ rest /services/cluster/master/peers | fields active_* host* label title status site | eval PeerName= site + ":" + label + ":" + host_port_pair | rename title AS peerGUID | rename site AS peerSite | table peerGUID PeerName peerSite ] | eval site=if(siteState=="search_state", peerSite, site) | eval value=if(siteState=="primaries_by_site", PeerName + ":For_" + site, value) | eval value=if(siteState=="search_state", PeerName + ":" + value, value) | fields - PeerName peerGUID peerSite | chart values(value) over bucketID by siteState table output: bucketID constrain origin primaries_by_site rep_by_site srch_by_site search_state _audit~118~FF782A13-8AFB-4617-BCB4-15ED11928DD7 0 site1 site1:centos58-64sup01-620CP:10.140.48.137:55591:For_site1 site1:2 site1:2 site1:centos58-64sup01-620CP:10.140.48.137:55591:Searchable site2:centos65-64sup14-620CP:10.140.48.150:55591:For_site2 site2:1 site2:1 site1:centos65-64sup06-620CP:10.140.48.142:55591:Searchable site3:centos62-64sup13-620CP:10.140.48.149:55591:For_site3 site3:1 site3:1 site2:centos65-64sup14-620CP:10.140.48.150:55591:Searchable site3:centos62-64sup13-620CP:10.140.48.149:55591:Searchable _audit~119~FF782A13-8AFB-4617-BCB4-15ED11928DD7 0 site1 site1:centos58-64sup01-620CP:10.140.48.137:55591:For_site1 site1:2 site1:2 site1:centos58-64sup01-620CP:10.140.48.137:55591:Searchable site2:centos65-64sup14-620CP:10.140.48.150:55591:For_site2 site2:1 site2:1 site1:centos65-64sup06-620CP:10.140.48.142:55591:Searchable site3:centos62-64sup13-620CP:10.140.48.149:55591:For_site3 site3:2 site3:1 site2:centos65-64sup14-620CP:10.140.48.150:55591:Searchable site3:centos62-64sup12-620CP:10.140.48.148:55591:Unsearchable site3:centos62-64sup13-620CP:10.140.48.149:55591:Searchable ... View more

@tdiestel You should try using INDEXED_EXTRACTIONS = JSON in props.conf of your forwarder. see: http://answers.splunk.com/answers/237933/how-to-configure-timestamp-recognition-for-json-da.html#answer-237936 ... View more