Glad to help! Keep in mind that this may come back if the offending cookie is recreated. Also, if this worked, can you mark the answer as accepted so others can find it should they run into the same thing? ... View more

Short answer: Don't tail them. Don't even try to ingest them until they're done being written. (So if NetApp is rolling the files, only monitor the rolled files, not the active ones) Then you need to figure out appropriate parsing for it to be split into events correctly. (If it doesn't do so already) If there are other logging formats available, they may be worth investigating as well. ... View more

There is a way to use more cores by adding parallel indexing pipelines. But if your queues are empty, that won't make any difference. I typically see indexers saturate 5 cores when fully loaded on indexing. (Processing about 20MB/sec) If the problem were with your indexer, you'd be seeing one of those queues as a bottleneck for the rest of the pipeline. I would suspect something going wrong at the input layer. Check queues and look for ERROR/WARN messages on your forwarders. (The queues you care about there are parsingqueue and tcpout_* ) ... View more

First thing I would look at is the state of the indexing queues. index=_internal host=YOUR_INDEXER sourcetype=splunkd component=Metrics group=queue (name=aggqueue OR name=splunktcpin OR name=parsingqueue OR name=typingqueue OR name=indexqueue) | eval max=if(isnotnull(max_size_kb),max_size_kb,max_size) | eval curr=if(isnotnull(current_size_kb),current_size_kb,current_size) | eval fill_perc=round((curr/max)*100,2) | timechart p90(fill_perc) by name Change up metrics (median, max, p90) to see where things are falling for each queue. If any of them are consistently high, you've got a bottleneck in the indexing pipeline. For details of what each pipeline does, check the community wiki. That can help you dig further into root cause. Splunk also reports if a queue is blocked in those events ( blocked=true ) so you can just search for that to see if you have any. index=_internal host=YOUR_INDEXER sourcetype=splunkd component=Metrics blocked=true Since you're talking about JSON files, I'd wonder how big they are, and if the bottleneck is actually on the forwarders. Parsing configs can make a BIG difference in indexing performance, especially for structured data. If props.conf on your forwarder has INDEXED_EXTRACTIONS=JSON set, then a majority of the legwork to index that data is actually happening on the forwarder, not the indexer, meaning the forwarder could be the bottleneck. (If you're forwarding _internal data from your forwarders, you can check their queues using a search similar to the above) ... View more

Check out MuS's answer. It's the best one for this, and works just fine. https://answers.splunk.com/answers/39370/is-it-possibl-to-get-a-list-of-available-indices.html#answer-125545 ... View more

Aha, there's a distsearch.conf file on the forwarder for some reason. It's got an old modtime, so assuming it's something leftover from botched automation. I'll get it fixed. Thank you!! ... View more

It looks like you have to explicitly configure the forwarder to tell it where to go for indexer discovery. Since I haven't set that up, (There is no [indexer_discovery:$foo] stanza on the forwarder) I would be very surprised if it was just trying to reach out to an indexer for that info. ... View more

Not sure what you're referring to by saying the "bundle replicated" as that can refer to a couple of things, not all of which affect indexing. On your indexers, run the following: $SPLUNK_HOME/bin/splunk btool props list $STANZA_NAME Check the output and make sure it reflects the current state of your props.conf file. I suspect it doesn't. The delete command simply makes items invisible to search. (Doesn't free up disk space or anything) However, it doesn't prevent the data from being re-indexed. So if you clean your fishbucket on your forwarder, that'll re-ingest the data. My hunch is that your props changes are happening on the search head, which isn't involved in the indexing process at all. When you run searches, a 'search bundle' is pushed to the indexers, but that is just for searching, it isn't involved in the indexing pipeline. What I would really recommend is to stop testing this live. You're eating up license and disk space to do so. Spin up a local install of splunk, put your props there, restart, and one-shot the file. If it doesn't work, clean the index and try again. Once you have a working stanza, then you put it on your real indexers and start ingesting data. ... View more

Couple more questions then: What is your environment? Is this a single instance serving as indexer and search head? If not, is this props.conf on the indexer? (That is where event-breaking happens) When you tried different things, how did you 'purge' the index? If you're doing ...| delete in a search, then you're not actually deleting anything, just making it invisible. Actually cleaning an index happens on the command line while splunk is stopped. ( $SPLUNK_HOME/bin/splunk clean eventdata -index $INDEX ) | extract reload=t is for search time extractions and does not apply to index-time configurations. Did you restart Splunk after updating your props.conf? In short, to test this you need to do the following: On your indexer: - Update your props.conf - stop splunk - splunk clean eventdata from the command line. (This drops EVERYTHING in an index, mind) - start splunk On your forwarder: - stop splunk - clean fish bucket - start splunk ... View more

Where is your props.conf file? ... View more

One more thing to try: add TRUNCATE=0 to your props. ... View more

Okay, so you've got some config mismatch. BREAK_ONLY_BEFORE is only applicable if SHOULD_LINEMERGE=true , which it isn't. When you have that set to false you want to use LINE_BREAKER to tell it how to break events. Also, both BREAK_ONLY_BEFORE and LINE_BREAKER take regex expressions, so what you have would not match what you intend. I would start with something like this: [DumbApp:ActLogs] KV_MODE = xml SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+)(?:<\?xml) MAX_EVENTS = 100000 If these XML segments are one per file, then another option is to set your LINE_BREAKER to something that will never appear. You will also want to tell splunk where to find the timestamp in your event. I'm not sure which of the different time values you want to use, so I'll just say you want to set the following: TIME_PREFIX , TIME_FORMAT , and MAX_TIMESTAMP_LOOKAHEAD Finally, depending on how these logs are written, you may run into other issues: If the xml is not written in one shot, Splunk may get confused as it will see the modtime update, read the file, and grab the partially written logs. If blocks are injected into the XML schema, it will wreak havoc. (Think a XML schema where events are inserted into it as time goes on, so the end of the block keeps getting pushed down instead of appending) Splunk will keep re-reading the entire file when it encounters this. If you have one file per event, and a lot of events are generated, this will result in Splunk tracking a LOT of files unnecessarily. (They will never be appended once written, so tracking them is unneccesary) In the event this becomes a problem, you may want to investigate the [BATCH://...] inputs. This will ingest the file and delete it when done. Keep in mind it is destructive, so may not fit your use case. ... View more

Right, but if the OS presenting that data and running the forwarder goes down... then what? Any redundancy you have in place for that contingency will also apply to the forwarder. What scenarios are left that we're trying to cover for? Short answer is that there isn't a mechanism to have multiple forwarder instances watching the same data and sharing state on what has/has not been forwarded. But there are very few circumstances where that would be useful, as generally whatever takes out the forwarder, takes out whatever is presenting the data as well. ... View more

Where is your data coming from in the first place? Is that redundant/HA? If so, how? ... View more

You're probably going to need to figure out some orchestration here. (And might already have some) One thing that comes to mind is that you shouldn't be configuring your forwarders with IP addresses for each indexer. Instead, create a DNS listing with all of your indexer IP's as A records within it. Then you just point your forwarder at the DNS record, and it'll load-balance across all the IP's found. When you need to add/remove indexers, you simply update the DNS listing. The forwarders will pick up on that change and forward to the new indexers automatically. Relevant documentation ... View more

This may not be helpful, but I would strongly advise against doing this. Having the bars potentially change order with every time-slice would make the overall presentation confusing and difficult to parse. ... View more

Any way to filter out that noise? It might be a good idea to shape the initial search to only grab the lines you really care about from these transactions. ... View more

If the logging is single-threaded (meaning you won't have processes interleaved with each other) then you can actually omit the 'startswith' and get what you want. Or, I do in testing at least. If you may have multiple hosts running concurrently, then you want to include that in the transaction: | transaction host endswith="(value=0)" ... View more

Extract out the function name as a field, and use that in your transaction: | rex "function\s(?<function>[^\s]+)\s" | transaction function startswith="Executing" endswith="(value=0)" That should pull the whole shebang as a single transaction. However, this will omit any lines which do not have a 'function' field. (which may be context you need) ... View more

Try this: $ splunk btool authorize list role_user [role_user] accelerate_search = enabled change_own_password = enabled cumulativeRTSrchJobsQuota = 100 cumulativeSrchJobsQuota = 50 get_metadata = enabled get_typeahead = enabled input_file = enabled list_inputs = enabled output_file = enabled pattern_detect = enabled request_remote_tok = enabled rest_apps_view = enabled rest_properties_get = enabled rest_properties_set = enabled rtSrchJobsQuota = 6 search = enabled srchDiskQuota = 100 srchFilterSelecting = true srchIndexesAllowed = * srchIndexesDefault = main srchJobsQuota = 3 srchMaxTime = 100days You can append --debug to see what conf files each line comes from. Note this is by role, not user. But as long as you know what role(s) the user has, you can use this. Another option is to use the REST endpoints to enumerate a role and it's consolidated capabilities. (In case it inherits from another role) This can be accessed via the search UI. | rest /services/authorization/roles ... View more