Possibly your _internal index size is not big enough to hold all the days data and it is rolling off? ... View more

You can use this search to look at your queues size per pipeline, and see where it gets full. ( we are using perc90 of the queue filled size) index=_internal host=* source=*metrics.log sourcetype=splunkd group=queue (name=parsingqueue OR name=aggqueue OR name=typingqueue OR name=indexqueue) | eval name=case(name=="aggqueue","2 - Aggregation Queue",name=="indexqueue","4 - Indexing Queue",name=="parsingqueue","1 - Parsing Queue",name=="typingqueue","3 - Typing Queue") | eval ingest_pipe = if(isnotnull(ingest_pipe), ingest_pipe, "none") | search ingest_pipe=* | eval max=if(isnotnull(max_size_kb),max_size_kb,max_size) | eval curr=if(isnotnull(current_size_kb),current_size_kb,current_size) | eval fill_perc=round((curr/max)*100,2) | eval name=name."-".ingest_pipe | timechart Perc90(fill_perc) by name useother=false limit=15 We will order the processor in the order they run, so that way you can see if one is full and is backing up the previous ones. 1- parsing -> 2 -aggregation -> 3 -typing -> 4- indexing ... View more

If you don't normally see your indexing queues blocked, you are probably fine. Start with two parallelIngestion pipelines and monitor behavior. ... View more

In props.conf you've got the wrong key: TRANSFORM-null = setnull Should be TRANSFORMS-null = setnull I did the same thing about a month ago and I lost 4 hours of my life or more... vowed never to forget it again and so I spotted it right away on your post. ... View more

Can you post the code? ... View more

In indexers cluster, indexes are created on the master node, otherwise they aren't replicated. Bye. Giuseppe ... View more

You must use search heads and/or heavy forwarders: http://docs.splunk.com/Documentation/DBX/3.0.3/DeployDBX/Distributeddeployment Forwarders can handle the load balancing without adding hardware: https://docs.splunk.com/Documentation/SplunkCloud/6.6.0/Forwarding/Setuploadbalancingd ... View more

If one data center went down, would the "downed data center" still produce logs? If so then the logs will sit on the file system until the data center is brought back up and sent to Splunk without missing a beat. I don't think you need an intermediate forwarding layer. You can install universal forwards to send directly to your indexer layer ... View more

Creating copy of data and ingesting twice (on two data center's indexers) will cause license duplication. For Fail-over use case like this, Splunk provides Indexer clustering where the multisite cluster will ingest data once and indexed data will get replicated across sites, as configured. Here is the starting point about clusters: http://docs.splunk.com/Documentation/Splunk/6.5.1/Indexer/Clusterdeploymentoverview ... View more

there are specific instructions on how to install. read here all the way: http://docs.splunk.com/Documentation/MSApp/1.4.1/MSInfra/ConfigureActiveDirectoryauditpolicy data will be good for MS Exchange mS Infra and yes also for ES. you can look at the add-ons details and verify they are CIM compliant if you dont have deployment server, either install manually or via simple script. since its a POC, why don't you start with just 1 or 2 severs? lastly, you replied in a new answer, if the answer above satisfies the question, kindly mark it as answered. hope it helps ... View more

Thanks for the response . we are going to collect all the logs with syslog-ng and UF and sending it to intermediate forwarder which in turn would send data to indexer . ... View more

@MuS I used the same results,dummyresults,settings = splunk.Intersplunk.getOrganizedResults() to grab the events from search results but it doesnt work ? any ideas ? ... View more

@khourihan_splunk I checked my logs it has the space as shown below Jul 7 11:21:56 180.189.15.13 CEF: 0|Imperva Inc.|... so I followed the steps mentioned above to modify the regex in cefkv.py at line 11 . here is my modification CEF_DATA_RE = re.compile('CEF:\s\d+|[^|]|[^|]|[^|]|[^|]|[^|]|[^|]|(.*)') I still don't get the expected key value pair can you recommend . ... View more