I'm at a .conf2017 session on Splunk NLP and the demo'ed app is "App:Babel Fish" in a test environment - that converts the language queries into SPL and presents visualizations. This can integrate really well with Alexa. Anyone knows - where to get this? ... View more

Hi, I have a single site cluster right now with the below configuration. 1. One License Server - also deployment server (001) 2. One Cluster Master (002) 3. Two Indexers (003, 004) 4. Two Heavy Forwarders (005, 006) 5. Two Search Servers (007, 008) All the above servers are now opened for TCP 8089 between each other. There are about 100 Splunk Forwarders forwarding data to this 8 server cluster in Site1. Here is the output of cluster-config on license master. config access_logging_for_heartbeats:1 cxn_timeout:60 disabled:0 forwarderdata_rcv_port:? forwarderdata_use_ssl:0 guid:xxxxxxxx heartbeat_period:4222054400 heartbeat_timeout:60 master_uri:https://002:8089 max_auto_service_interval:30 max_peer_build_load:5 max_peer_rep_load:5 max_peer_sum_rep_load:5 mode:searchhead multisite:false notify_scan_period:10 percent_peers_to_restart:10 ping_flag:1 quiet_period:60 rcv_timeout:60 rep_cxn_timeout:60 rep_max_rcv_timeout:600 rep_max_send_timeout:600 rep_rcv_timeout:60 rep_send_timeout:60 replication_factor:3 replication_use_ssl:0 restart_timeout:60 search_factor:2 search_files_retry_timeout:600 secret:******* send_timeout:60 service_interval:1 site:default* Now the plan is to setup another site - that'll also have about another 50 Splunk Forwarder that need to forward logs to the same set of indices. I have the below questions now. 1. I still need the 8 servers in the new Site2 - but the deployment server will be acting as a license slave. is that correct? I've already installed Splunk Enterprise same version (6.5.2) on about 7 servers and didn't do any configurations further. 2. Assuming servers 101 to 108 will be available in Site2 and the above configuration - what commands should I be executing to configure multisite clustering on all these 16 servers? 3. What values of search_factor and replication_factor should I consider? 4. For these servers to form a multisite cluster opening firewall ports between Site1 and Site2 - for splunk management and replication? How do I configure these replication ports? 5. I'd like to setup replication first and make sure that the logs of Site1 are searchable in Site2 search servers. Is that the right approach? 6. The Site2 Deployment Server is going to be a backup for Site1 Deployment Server or is going to be a new Deployment Server? The end goal is to make the 4 search servers (2 from Site1 and 2 from Site2) be able to serve the same data - with 100 forwarders to Site1 and 50 forwarders to Site2. ... View more

I've deployed the deployment-app on the deployment client from deployment server. The server appeared on the phoned list. But is not indexing the logs. The splunkforwarder logs don't show any error and the splunk user has the permissions to read the logs. Where can I check what is not causing the indexing? ... View more

My Sample event every minute looks like this: 03/06/2017 15:19:00 -0500, app01:JVM1=12, app01:JVM2=6, app01:JVM3=9, app01:JVM4=3, app01:JVM6=7, app02:JVM1=14, app02:JVM2=0, app02:JVM3=0, app02:JVM4=25, app02:JVM5=11, app02:JVM6=4, app03:JVM1=5, app03:JVM2=9, app03:JVM3=5, app03:JVM4=26, app03:JVM5=10, app03:JVM6=2, app04:JVM1=13, app04:JVM2=4, app04:JVM3=10, app04:JVM4=16, app04:JVM5=13, app04:JVM6=0, app05:JVM1=10, app05:JVM2=47, app05:JVM3=21, app05:JVM4=15, app05:JVM5=13, app05:JVM6=2, app06:JVM1=8, app06:JVM2=8, app06:JVM3=10, app06:JVM4=8, app06:JVM5=8, app06:JVM6=6 I am able to plot the timechart per field - by using | timechart max("app*:JVM*") span=1m or avg function. But, now I need help in writing the queries to: 1. Sort and show the top 10 key value pairs in a table - of the sum of values (off the 5 events) every 5 mins in a dashboard 2. Show the outliers off the timeframe's average in a table 3. Sort and show the top 10 key value pairs with highest delta between timeframes (every 5 mins) In verbose mode, these keys are showing up in "Interesting Fields" but was not able to use them by field. ... View more

I have a clustered Splunk env with an index="myjavaapp". I need to collect the logs from multiple environments - Dev/QA/Stress/Pre-Prod/Prod - where each environment has about 2 to 15 servers. The logs on all servers fall in the same path /opt/appserver/logs/server.log. What is the best practice in this case to maintain logs from different envs, but coming from the same path. Also, I have other applications coming from the same environments, but different servers. I need to implement the strategy for all the indexes. (ex: index="mywebapp") ... View more

Hi, I have a Splunk Heavy Forwarder, Indexer master and two Indexer slaves, two search heads in the current architecture. I have few rest inputs within the network (a firewall away) to get the values in. Now, I need to choose the machine on which the rest_ta app can be installed and define the rest inputs, to an existing index. By practice/convention, where should the rest_ta app be installed and the inputs defined? ... View more

Hi, We have a cluster setup - where we have 1. Heavy Forwarders 2. Indexer servers and an indexer master 3. Search heads and master I have a MAX_EVENTS configuration need for log4j source type and I made a change in indexer master $SPLUNK_HOME/etc/master-apps/my-app-props/props.conf and pushed the change to to the indexers in the cluster. I see the configuration effective in the indexer servers (ran $SPLUNK_HOME/bin/splunk cmd btool props list log4j ) as I see the below output. Still, my events are breaking at 257 lines of the event. I have at least 3 sourcetypes - for which I need to make props.conf changes. Where should these changes go? Heavy Forwarder or Indexer master/slaves? [log4j] ANNOTATE_PUNCT = True AUTO_KV_JSON = true BREAK_ONLY_BEFORE = \d\d?:\d\d:\d\d BREAK_ONLY_BEFORE_DATE = True CHARSET = UTF-8 DATETIME_CONFIG = /etc/datetime.xml HEADER_MODE = LEARN_MODEL = true LEARN_SOURCETYPE = true LINE_BREAKER_LOOKBEHIND = 100 MAX_DAYS_AGO = 2000 MAX_DAYS_HENCE = 2 MAX_DIFF_SECS_AGO = 3600 MAX_DIFF_SECS_HENCE = 604800 MAX_EVENTS = 10000 MAX_TIMESTAMP_LOOKAHEAD = 128 MUST_BREAK_AFTER = MUST_NOT_BREAK_AFTER = MUST_NOT_BREAK_BEFORE = SEGMENTATION = indexing SEGMENTATION-all = full SEGMENTATION-inner = inner SEGMENTATION-outer = outer SEGMENTATION-raw = none SEGMENTATION-standard = standard SHOULD_LINEMERGE = True TRANSFORMS = TRUNCATE = 10000 category = Application description = Output produced by any Java 2 Enterprise Edition (J2EE) application server using log4j detect_trailing_nulls = false maxDist = 75 priority = pulldown_type = true sourcetype = ... View more