If a sourcetype is not in props.conf then it doesn't exist.  Mentioning it in inputs.conf alone is not enough.  Props.conf is where the properties of the sourcetype are specified.  Without them, Splunk has to guess about the sourcetype and often guesses wrong. ... View more

2nd such post with no resolution 😞 ... View more

I had to change the first regular expression to the following. \<Data\>(?<zzz_sql_audit_data>[^*]*)\<\/Data\> ... View more

Hi @hectorvp. at first I don't understand whay you say "We are not allowed to use HF in between": a Splunk server that forwards log s  to an Indexer (eventually locally indexing some or all of them) is usually called Heavy Forwarder, not Indexer! The second thing I don't understand is what you mean with "validate": if you mean filter data before indexing, you can do this also on Indexers. Then, "edited version for us & different edit for them": you cannot edit data, you can index a data on your HFs and send to other systems elaborated data, but in this way you haven't the original data. Maybe the best approach is to index original data on customer Indexers and extract the "validated data" in a summary index to use for searches (in this way you haven't a twice license consuption). Anyway, an indexer can index around150-200 GB/day, but Splunk best practices say that it's better to have a max value of 100 GB/day, especially when the Indexer has also to answer to the searches. So, two Indexers are few to manage 400 GB/day. About your Indexers specs, they are correct, maybe also excessive, but "melius abundare quam deficere! Ciao. Giuseppe ... View more

Thanks @gcusello , we've decided to keep all on different boxes. ... View more

Thanks @richgalloway  Yes, these are other complexities as well as we are going to route these events to other third party indexers   ... View more

I think the sampling done by the per_source_thruput group will even out over the course of a day.  If that doesn't work well enough then try this alternative. index=_internal component=LicenseUsage earliest=-7d@d latest=-1d | stats count as old_count by s | append [ search index=_internal component=LicenseUsage earliest=@d | stats count as new_count by s] | stats values(*) as * by s | fillnull value=0 new_count | where new_count=0 ... View more

Hi @gcusello , First of all thanks for the suggested approaches. Apologies, but it would be a great help if you can explain the 2nd approach in detail, I mean how can I configure HF to send single event of a particular source per host to our indexer in a day + all the events to customer indexers and then  we can have validation to logs we are sending. Question is raised on community with the link, https://community.splunk.com/t5/Monitoring-Splunk/Forward-single-event-from-HF-to-Indexer/td-p/527781   ... View more

Thanks @richgalloway , this would help us. ... View more

Thanks a lot @gcusello  ... View more

Average number of events per min per UF will be well within the range you selected, so with that buffer of 50%, you should be good.  ... View more

I think that for this use case, you can use the Splunk Enterprise trial license available in a variety of deployment factors to satisfy your requirements. This includes a simple virtual machine, AWS AMI, docker images, etc. The trial license supports all features that you describe and past versions are available as well.   https://www.splunk.com/en_us/download/splunk-enterprise.html https://splunk.github.io/docker-splunk/ https://aws.amazon.com/marketplace/pp/Splunk-Inc-Splunk-Enterprise/B00PUXWXNE ... View more

Thanks @gcusello , Since I'm storing only internal logs in my standalone indexer, I may use raid0 or 15k rpm sas HDD....still would think over it. ... View more

Hi @isoutamo , I did the following to make rough estimate that is avgerae size of internal logs = 300 bytes  average events per seconds by internal index with 1 UF = 10 total internal data at  1 day in MB by 1UF= 10 * 300 * 60 *60 *24 / (1000*1000) = 260 MB So for   1 day with 500UFs = 26 * 500 = 130000= 130GB in  day Compression ratio of 50% so total 1 day data =  65GB so 60 days of retention capacity then = 65*60 = approx 4TB Am I going in right direction???  Or missing any factors? ... View more

There is no such thing as a "forwarder license owning issue".  All universal forwarders have the same license, which has no ownership (except perhaps by Splunk). Changing the admin password is a good idea.  Good instructions for resetting the password can be found at https://www.hurricanelabs.com/splunk-tutorials/splunk-7-1-performing-a-splunk-password-reset ... View more

Yes, there are scenarios where events can be dropped, but Indexer Ack prevents them. ... View more

Hi @hectorvp .. if you use the indexer acknowledgement feature, as per my understanding, there will be no data loss in-flight(dropped data), as it uses the "handshake" logic's. (with this handshake logic, if the UF got crashed, when its resolved and comes up again, indexer will know from where it should read. so the events inside the queues will be re-read and re-processed again). I have worked on few financial projects in which the client managers will ask these questions, i have explained them as well. In practical scenarios, 1) the critical data, the clients wont agree to be sent to splunk altogether, thats good.  2). so, the lesser-critical data, that can be sent to splunk. 3). the data loss when UF down / indexer down is very rare case. lets say around 5 out of 100 cases. and when it happens, those 5 cases, the logs can be re-read. 4) the value of the logs is not that much critical, so we need not worry much on this "worst case" scenarios.   (PS - i have given around 350+ karma points so far, received badge for that, if an answer helped you, a karma point would be nice!. we all should start "Learn, Give Back, Have Fun") ... View more

Hi UF starts reading where it has left before output queue is full. It's own situation has stored on fishbucket index. But if you don't use persistent queues and UF will go down before it can send events to IDX then you will lost those events which are in output queue as those are just memory queues with normal configuration. r. Ismo ... View more

Hi @hectorvp ..  Q - Is there any way to find how many events were dropped by UF in a day?  A - nope. if you analyse this situation little more, you can understand that,..  1. UF sends the data to indexer. lets say indexer is down. then the data is not dropped, it is still waiting to be read at UF. when indexer comes up again, it will start reading from where it left before going down.  2. the persistent queues are one solution to look for.  By default, forwarders and indexers have an in-memory input queue of 500KB. you can configure and increase the size of this queue, so that there will be no concerns about data dropped. https://docs.splunk.com/Documentation/Splunk/latest/Data/Usepersistentqueues   3. the indexer acknowledgement feature is another good solution. it adds little more load on indexer, but it is worth the load. so, with indexer acknowledgement feature, the indexer and UF will have an extra layer of "handshakes", so that UF and indexer always knows that the data is not dropped.  clear documentation on this indexer acknowledgement feature: https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Protectagainstlossofin-flightdata   (PS - i have given around 350+ karma points so far, received badge for that, if an answer helped you, a karma point would be nice!. we all should start "Learn, Give Back, Have Fun") ... View more

The UF will start from the last file position it saved. ... View more