Staging Server for Search Head cluster. Build all your apps/TAs in here. (/etc/apps/) Deployer to push the app to search head members (/etc/shcluster/apps). Both above can be same server http://docs.splunk.com/Documentation/Splunk/6.5.0/DistSearch/SHCdeploymentoverview We centrally manage the code in "git" repo, push it to Staging Server for consistency. For rest of your cluster - Forwarders using deployment server - Indexers using cluster master ... View more

Try this Enable a user (generic user with simple password) eg: noc_user password=changeme in web.conf (edit and restart splunk) enable_insecure_login = true then ensure the URL is: hostname:8000/en-GB/account/insecurelogin?username=noc_user&password=changeme ... View more

The key thing to understand is "its all about data" !! Once you have all the data, then its just an app to make it for any functionality which could be alert/event-management. In my opinion, you could replace - event management tools (like Tivoli suite/netcool/scom) - other agents like pinger, almost everything.. opportunities are endless once you have access to data. I really wish I could create an app with Ansible + splunk alongside.. 🙂 ... View more

Just from expierence, the competition I've worked/considered are ( I hate when people call as log management, but I consider them as big data tools) - Splunk - ELK (ElasticSearch - Logstash - kibana) - SumoLogic - Hadoop Some key things from my experience - Splunk is highly enterprise-class with very easy to install. Hadoop is a real pain unless you have good java developers. ELK is really good and true competition to Splunk. SumoLogic is good too, but I've limited experience - Splunk is both on-premises + cloud. Sumologic is cloudOnly which may not be good for every organisation. ELK and Hadoop variants i've worked only on premises. - Agents/Datashippers. None of the agents out there can be compared to Splunk Universal Forwarders. This makes a huge difference in enterprise level. ELK Beats sounds promising though, but no-where compared to Splunk UF in areas of load balancing/consistency/secure transfer/cross platform/run-as functionalities - Though Hadoop is very extensible, literally Hadoop takes 2-3 weeks to write a simple Demo/POC (when you can do in minutes for Splunk/ELK). So time to market is huge in Hadoop/Spark and requires good java developers in your team. if your organisation is huge, using hadoop i can bet the project to last 2 years. - Cost. Splunk costs approx approx $1000 for 1GB (one time cost) + support cost. Hadoop is free, but dont ever think of going live without Cloudera/Hortonworks & development time which is a costly one. ELK seems to be the winner here, easy to setup but scaling is quite tricky. - Learning Curve - Splunk is easy to learn and pretty consistent. ELK changes every year to new ideology which could be tricky for employees. Hadoop is wild wild west with layers/development going too fast for enterprise level and a huge learning curve. Options you could consider - Truly Enterprise Level data analytics = Splunk Enterprise - Low cost for SME = ELK or Splunk light - Highly flexible, complete new application development = Hadoop/Spark and variants or Splunk Enterprise with SDK's - Completely SAAS option - Splunk or SumoLogic - Mix options = Use ELK/Hadoop for dumping data which are less relevant, but Splunk for real core and valuable datasets ... View more

Also, i've requested for Heavy Forwarders to be part of DMC: https://answers.splunk.com/answers/377028/how-to-configure-dmc-for-heavy-forwarder-monitorin.html ... View more

We faced the same issue. Assuming "splunk" have read access to the OS logs, what we have done is using Splunk_TA_nix. Put into the "local" of this app, with what files you want to collect by adding the paragraph and putting disable = false (Most of things are already part of TA_nix) For different layers, enable Splunk_TA_nix in below fashion. - For Splunk Forwarders push using deployment-server. It goes into $SPLUNK_HOME/etc/apps of forwarders. - Copy and restart Splunk_TA_nix into $SPLUNK_HOME/etc/apps for deployment-server, - Copy and restart Splunk_TA_nix into $SPLUNK_HOME/etc/apps for cluster master, - For clustered Search Heads, package into $SPLUNK_HOME/shcluster/etc/apps and push to Search Members. In search members, it will be merged into "default", but works. - For clustered Indexers, copy Splunk_TA_nix using cluster master via master-apps. This goes into "slave-apps" of Indexer slaves and works perfectly. If you enable Splunk_TA_nix, then you can start colllecting every information about your Splunk Infrastructure/OS ... View more

I use github repo instead of blogging and i've created a project with all learnings/experiences/scripts together. (You can make it to .io if you want to make it a blog) This way, you can change your experience, if you find something better and it can be followed by other people, open source etc. ... View more

cheers. agree with you. But if I put "syslog", the CIM extractions are lost as it cannot identify if it is a cron or audit file etc. ... View more

the host segment will be overridden if it is a "syslog" . So if I specify "linux_messages_syslog" it is OK, but host_segment is lost and Splunk_TA_nix CIM is lost ... View more

Any error in logs? ... View more

How would you then associate sourcetype? As each wineventlog have specific sourcetype ... View more

is your Cluster master 6.3x or 6.4x by any chance? we had same issue from Search heads when indexer discovery is enabled. if yes, this is a bug which is fixed in 6.4.4 ownwards ... View more

why not upgrading to ES 4.1.x? ... View more

tstatsHomePath is Location where datamodel acceleration TSIDX data for this index should be stored and is mandatory. to fix it, - go to indexes.conf (in your custom app or any app which you configured indexes) - on each individual index, put a new tstatsHomePath entry - Put the same value as your hotPath (hope it has enough diskspace) - distribute and restart ... View more

Best way to query a database (local or remote) is using Splunk DBconnect (v2). DBconnectv2 will handle pooling and caching etc. It can import the table in block by block basis, so you can test plan before you load whole of the system. (You can operate the database like a lookup if you don't want to index it.) ... View more

http://stackoverflow.com/questions/4742746/jquery-open-new-window-on-page-load http://stackoverflow.com/questions/33547885/load-function-after-whole-page-has-been-loaded can embed same logic of jquery in splunk & also beware of any pop-up blockers ... View more

could you do sourcetype=WinEventLog:Security EventCode=4624 Logon_Type=3 src_ip=xxx| stats count by _time,user src_ip Workstation_Name | join src_ip [ search <your Second search>] ... View more

Instead of "stats" , use "eventstats". Then you can have original event with stats data coming as well Please check this article: http://blogs.splunk.com/2014/04/01/search-command-stats-eventstats-and-streamstats-2/ ... View more

The data still can be in u specified indexes. All you need to make is to use CIM standards and accurate sourcetype and ES will take them automatically ... View more

I wouldn't choose any default indexes. You need to plan your index names inline with your organisation structure. Eg if my company is Amazon, then my index names would be like amz_os_windows, amz_os_linux, amz_network_cisco, amz_network_juniper etc So in future if u want to give access for network team/role u can give amz_network* to that roke ... View more