Thanks David, unfortunately I'm unable to run unapproved applications on my network. If it was an app within splunk I could though. I appreciate your help on this and will continue to see what's out there. It's crazy I have to do all this because the virtualization team won't be honest smh ... View more

For example, I'm pulling huge logs. And I read that due to the large log set event processing can be clogged on the forwarders. I saw this bit in an article I was reading For optimal performance of your data, you can set the following settings for your sourcetype in props.conf: DATETIME_CONFIG MAX_TIMESTAMP_LOOKAHEAD TIME_PREFIX TIME_FORMAT SHOULD_LINEMERGE ANNOTATE_PUNCT Should I be doing this on the inexers? ... View more

Thanks David, I have these currently set on my search heads but curious as to what makes sense to add explicitly to time date / parsing that may improve indexing. Generally I don't want the entire app on my indexer as that will add to index time (slow resources) Is setting the time zone all I need to do for these source types? There's a lot of options for time date field parsing ... View more

You may have missed the part where I mentioned this particular environment is on Windows so iostat and Bonnie++ won't work here. I was thinking maybe there was a scientific way by knowing the amount of data /events pouring in vs the index time. Perhaps with all the pieces I could figure it out. For example if we had 200 iops storage vs. 2000 we should be able to estimate the difference some kind of way with the index data? Of course ballpark not exact estimates Thanks for your input! ... View more

Thanks for that information about increasing the pipeline. I believe the issue is with the forwarders trying to process too much information. The indexer don't show queues between the various stages of the piplines. I'm also curious now if our virtual team screwed us on our storage (gave us the slow stuff) ... View more

You may have missed me stating this in my op "It's not an issue with the parsing queue on the indexer as there's no lag between the various stages" as I've checked the monitoring console first. It's 0 across-the-board of each stage. ... View more

I meant normalize the data in respect to the timestamp, I should of been clearer. Generally I do my Field extractions at search time on the search heads only. You may have missed that I have 2 indexers currently so one indxer is getting half this amount so I don't think it's the indexers.. no issues with the data pipeline.. i'm thinking limit.conf is probably where I need to concentrate. Today I evaluated my actual logs and see someone doing something crazy with web Api calls that have more than quadrupled the log size. So I'll have them stop what they are doing first and look into the limit.conf at the same time Thanks for your help ... View more

Maybe I'm ignorant to the idea of me hitting any limits as I'm only ingesting 250gb daily and I know of plenty who pull TB's of data a day. Perhaps they've adjusted their limits.conf to allow the data to flow or perhaps they are pulling from 1,000 devices to = that 1tb and no individual node is reaching the default limit in limits.conf where I'm only pulling from 84 devices = 250gb's? I definitely need to fix this problem asap! ... View more

Yes, the timestamp on all the IIS servers look fine. They are in UTC and as stated in the OP I've added a props.conf entry for that sourcetype that normalized the data. If I do a search on future logs nothing is returned so I'm not of the impression it's a timestamp issue. One thing I meant to mention i discovered leaving out work, another log source is also delayed. Both of these logs are the biggest logs source I'm pulling However smaller logs and sources still come through I'm starting to think I'm hitting my limit in limits.conf. ... View more

Change what setting and did it fix your problem? ... View more

So in my outputs.conf for my UFs [Indexer_discovery:master1] [tcpoutput:group1] IndexerDiscovery=master1 Don't know if this works in combination with cluster label in clusters server.conf But I made the label master1 (to match) and everything just worked afterward. i also commented out multisite references and rekeyed pass4 on cM and indexers. ... View more

Change the pass4 key to whatever you'd like. You'd have to change it in both cluster master and indexer nodes if you don't remember It, and then restart the nodes starting with the Master node first (you can do it from the gui). ... View more

I got it working, My problem wasn't ports or the security key. It was the fact the my server.conf cluster master label was set to: my_cluster and the outputs.conf referenced the example set under the indexer discover article here: https://docs.splunk.com/Documentation/Splunk/7.3.0/Indexer/indexerdiscovery [indexer_discovery:master1] pass4SymmKey = my_secret master_uri = https://10.152.31.202:8089 [tcpout:group1] autoLBFrequency = 30 forceTimebasedAutoLB = true indexerDiscovery = master1 useACK=true [tcpout] defaultGroup = group1 Once i made both outputs.conf and server.conf files match the same information for cluster label everything worked as expected "ingesting, replication". If you can edit your original post to include this information i'll accept it as the answer. Thank you! ... View more

Ok, I just added the site ID's as you specified still no luck. I still don't see the newly created index showing up on the master nodes clustering screen. It only shows _audit and _internal. splunk|splunk is owner of splunk directory files, I believe root is the owner of the /opt/indexes/myindex could that be the problem? I'm new to linux and clustering so any help is appreciated So my problems are 1) though the cluster master was able to create the new index with index.conf and distribute the index , the index still doesn't show up under list of index via the index clustering link on the cluster master 2) though the server is set up as you described above something doesn't seem to be configured correctly. what are some good troubleshooting steps to try and pinpoint the exact issue? ... View more

Hi Naresh, Thank you for your response and your assistance is appreciated. so i don't see that it's sending data to the index cluster i created. Via cluster master i deployed index.conf via master-apps, _cluster, local It then created indexes in the specified location of the index.conf file (everything looks good so far) however on the cluster master page it doesn't show the newly created index, so i'm thinking that's problem #1 Why isn't it showing the new index that the cluster master just created on the peers? Moving on.. Index cluster is up and running, healthy and replicating _internal indexes. I've added an outputs.conf to one of my web server's universal forwarders "etcs\system\local\" directory with the information below and then I restarted the forwarder [indexer_discovery:clustermaster] pass4SymmKey = mypassword master_uri = https://clustermaster:8089 [tcpout:clustermastergroup] indexerDiscovery = clustermaster useACK = true [tcpout] defaultGroup = clustermastergroup My server.conf file for my index cluster master: [general] serverName = clustermaster pass4SymmKey = $7$VDinTNOJp0GCcK0jj8fYCQoxQW6+p3exc2PtgRIEek5OTErTR9+q5g== sessionTimeout = 1000d [sslConfig] sslPassword = $7$6o4579kYGK8VotDH9I5VFy0ly48OdYWJ3jnmvv8tKTFPIUdUebd38w== [lmpool:auto_generated_pool_download-trial] description = auto_generated_pool_download-trial quota = MAX slaves = * stack_id = download-trial [lmpool:auto_generated_pool_forwarder] description = auto_generated_pool_forwarder quota = MAX slaves = * stack_id = forwarder [lmpool:auto_generated_pool_free] description = auto_generated_pool_free quota = MAX slaves = * stack_id = free [license] master_uri = https://LicenseMaster:8089 [clustering] cluster_label = my_cluster mode = master pass4SymmKey = $7$497Zb7a04lOvgYxtdzmIiTdcmHomDYYA7TRypAx+LcFwcUXOKz+ovFMHmeA= replication_factor = 2 search_factor = 1 [indexer_discovery] pass4SymmKey = $7$5o6HjfUbtuiigSL4yEcVGs6CT8zSCtin+4l+NyTCkWTKF2hLCV7WfZMEVKg= indexerWeightByDiskCapacity = true A few things to note. I have a standalone test environment and only used one index called "myindex" Using the deployment server I distributed my apps and all forwarders would send it's data to that stand-alone server to myindex. On the index cluster nodes, i deployed an index.conf file that created the raw index db's for myindex. When I deleted the original outputs.conf file from the webserver and replaced it with the new one specified above i noticed the forwarders stop sending data to the stand-alone server (GOOD that part is what i wanted) however no data has been sent to the index cluster I restarted the universal forwarder and I expected to see it forward data to my index cluster but no data has returned. Can you tell me what I've done wrong? did I miss a step somewhere? . ... View more

How would you set this up in an index cluster? What would the outputs.conf file look like? ... View more

I downvoted this post because wrong! ... View more

Unfortunately mines isn't on a public subnet. Are you using your domain name as the ldap server name? Some people put their local domain controller host name or IP. I use the domain name root that way if they change out a domain controller or switch the IP I'm always good. For example: Mydomain.com (whatever your company's logical domain name is) vs servername. You can test your ldap strategy accounts rights by going to start...run... Type in dsa.msc and run as the ldap strategy binding name. If that account can't view AD objects them that could be your problem. You could try with your own personal admin account (not recommend in the long) but good way to rule out it being the account ... View more