Alright so it sounds like we've cleared up the internal logging question (yay btool!) I appreciate this discussion as well and thank you for bringing it up as some people may overlook this sort of thing. And while it's most of the time not an issue, I can see where some people may have this question and seek this type of specificity. I read through your test case and I think it's good but let me explain a little bit more of what you may have experienced. Since the in-memory queue is so small (500KB by default) when you're dealing with a large amount of data, that 500KB can be somewhat negligible and hard to see. In your case when you turned off the UF in step 3 you only lost 500KB of data. When you turned it back on in step 5 and did the search again, that UF had started back up and began indexing the data that never made it into the queue in the first place. The reason you probably seemingly saw all of your data is because that 500KB is so tiny and Splunk handles failures quite well. Take a look at the following test case: I just performed the following: Added a monitor stanza for a file (test.log) on my UF but did NOT restart so the input did not take affect yet. Wrote 20 lines of data to this file that were just "value=1". I had a props.conf setup so lines do not merge, and I set the time to index time. This way when and if the events come in, they will be new lines and the time will be the current time they were indexed. I then stopped my UF. I then shutdown my Indexer so that when I restart my UF, queues are blocked. I restarted my Universal Forwarder so it began monitoring the test.log file. Since this file is so small, and since my Universal Forwarder is doing nothing else, that log file along with any internal splunkd.logs should fit into that initial memory queue of 500KB. I shut down my Universal Forwarder so that the inputs in my memory queue are destroyed. I started my indexer back up so queues open back up. I started my Universal Forwarder to see if anything would be indexed from test.log. I searched for the source test.log and found no data. This confirms that anything in the test.log file was in memory and waiting to be output to my indexers when it was destroyed. I then began all-time real-time search for test.log. I went and added one new line to test.log. That new line was simply value=2. The new line showed up in my search results immediately. What this tells us is that anything in memory when the Universal Forwarder fails may be lost. The fact that it is 500KB and so small is generally negligible when we're talking about gigabytes, terabytes, and even petabytes of data per day. Additionally, 500KB is the maximum amount of data you could lose if the queues are completely blocked for file monitoring. Since Splunk processes data so fast, typically you won't even see anywhere near that amount be lost unless you're monitoring a LOT of information in which case you may need to modify other options like your maxKBps or switch to a Heavy Forwarder. ... View more

Do you have Single Sign-On enabled? ... View more

Alright I modified my original response so that it is a bit more clear, but let me also expand as I think there are some other things we can clear up here for anyone else that might be reading this. Regarding the disabled inputs, what did you run to obtain that output? Can you try the following since there are internal logs coming in, there is some monitoring going on somewhere: $SPLUNK_HOME/bin/splunk cmd btool inputs monitor As for the data you saw even though your indexer was down for a period of time, Splunk Universal Forwarders also have queues (http://docs.splunk.com/Documentation/Splunk/6.4.1/Deploy/Componentsofadistributedenvironment). The queueSize that you brought up is an in-memory queue. Since your indexer was down, the Universal Forwarder's Input queue was blocked up because data couldn't get to the indexer. When the indexer came back up (since your Universal Forwarder never went down) the data in-memory was able to get out and you never lost any data. In addition, new data that had never been sent started to flow into the input queue and played catch up for the data that had been missed during the indexer outage. This is why Splunk has multiple time fields. One is an _indextime (the time the data was actually indexed) and the _time (the time of the event). When you look at the timeline in your image, you will notice that your data is making sense in chronological order. However the time field you don't typically look at is the index time. To visually see an example of this, the following search will show latency (difference) between indexing time and event time for your splunkd.log events. A high latency means that a specific event had a distinct difference between the actual event time and when it got indexed. I bet if you run the following search over that time frame, you'll see some high max latency indicating that your system was playing "catch up" for events it had missed. index=_internal source=*splunkd.log host=**<universal_forwarder_hostname>** | eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S") | eval time=_time | eval itime=_indextime | eval latency=(itime - time) |stats count, avg(latency), min(latency), max(latency) by host Also as an added bonus, here is a nice conf presentation on some of the data pipeline stuff, https://conf.splunk.com/session/2014/conf2014_AmritBathJagKerai_Splunk_WhatsNew.pdf ... View more

If possible, can you clean that index and re-index the file after you've made that change? Can you also post the updated version of the csv file once they've changed it so we can confirm it looks correct? Thank you, Ryan ... View more

Can you also modify the first couple lines? the CSV indexed extractions work best if the first line is a header for the CSV file. Back to Index, HOST INFORMATION, Should be software_build, count So your CSV would look more like software_build, count Software build-2718055,10 Software build-3116895,15 Software build-2583090,35 Software 5.5.0 build-1746974,22 ... View more

The biggest issue I see with this file is that it's poorly formatted and not truly a well-formatted CSV file. See the screenshot below of when I saved the sample text you sent, saved it as software_summary.csv and opened it in Numbers on my mac. If you did have it as a CSV then your props.conf would also include INDEXED_EXTRACTIONS = CSV and would handle this file much easier. ... View more

This seems network related and not as much about the add_orphan_field parameter. Are you using a load balancer for your Search Head Cluster? ... View more

You might benefit from using the following add-on: https://splunkbase.splunk.com/app/2648/#/overview This should give you plenty of information regarding performance of your MSSQL database. ... View more

The following line can be removed since "SHOULD_LINEMERGE" is set to false. BREAK_ONLY_BEFORE = ^{ Can you post a sample JSON event that you're seeing in Splunk? This appears to be valid JSON so it should be extracting. ... View more

In order to ingest Cisco ASA logs you'll want to start by sending data via syslog from your Cisco devices. This section of the document covers it but may be a little vague if you've never configured Splunk to receive syslog before, or configured a syslog receiver such as syslog-ng. http://docs.splunk.com/Documentation/AddOns/latest/CiscoASA/Inputs I would consider reviewing the below documents if you've never done this before. http://wiki.splunk.com/Community:Best_Practice_For_Configuring_Syslog_Input http://www.function1.com/2012/05/syslog-collection-with-splunk Essentially you'll either setup a syslog receiver such as syslog-ng to drop the syslog events into a file. Then Splunk can monitor the file and place it in the proper sourcetype defined here: http://docs.splunk.com/Documentation/AddOns/latest/CiscoASA/DataTypes You can also configure Splunk to accept the input directly, but best practices suggest you use syslog-ng. Let me know if you have any other questions. ... View more

Is there a reason you're still running on Splunk 5.0.4? Typically it's not recommend to start altering those defaults. There have been several improvements between the current version of Splunk and the one you're running. ... View more

Do you have any control over the output of this script? Is this something custom you are running? ... View more

1 thing I do know about this app is that you need to have a "default" domain so you likely can't successfully add a new domain until the default one is configured. You should however be able to click the + button. What version of Splunk are you running? Have you tried launching the setup page in another browser? Are you able to restart Splunk and try again? ... View more

What REST API are you trying to query? Are you sure the response type is CSV? Can you send a sample response? Typically if Splunk is monitoring a CSV file there are all types of configurations you can use to parse that data in props.conf http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf The following settings seem like they might help you out if you are in-fact monitoring CSV Data. INDEXED_EXTRACTIONS = < CSV|W3C|TSV|PSV|JSON > * Tells Splunk the type of file and the extraction and/or parsing method Splunk should use on the file. CSV - Comma separated value format TSV - Tab-separated value format PSV - pipe "|" separated value format W3C - W3C Extended Extended Log File Format JSON - JavaScript Object Notation format * These settings default the values of the remaining settings to the appropriate values for these known formats. * Defaults to unset. FIELD_NAMES = [ ,..., ] * Some CSV and structured files might have missing headers. This attribute tells Splunk to specify the header field names directly. TIMESTAMP_FIELDS = [ ,..., ] * Some CSV and structured files have their timestamp encompass multiple fields in the event separated by delimiters. This attribue tells Splunk to specify all such fields which constitute the timestamp in a comma-separated fashion. * If not specified, Splunk tries to automatically extract the timestamp of the event. ... View more

I don't believe the Cisco ASA add-on has any sort of setup page. What exactly are you trying to do? Have you taken a look at the documentation for that add-on? http://docs.splunk.com/Documentation/AddOns/latest/CiscoASA/Description ... View more

Splunk may re-index entire files like this if something changes at the beginning or middle of a file. If something changes, it may assume the file is new and re-index the entire thing. Is it possible something like is happening? Maybe older events are being removed from the file or "rolling" out? ... View more

What are your events looking like? Have you tried setting the following in props.conf? props.conf DATETIME_CONFIG = CURRENT SHOULD_LINEMERGE = false ... View more

Ah sorry about that! During the initial install you should be able to set a number of different parameters like SPLUNKD_PORT. Take a look at the following doc: http://docs.splunk.com/Documentation/Splunk/6.4.1/Installation/InstallonWindowsviathecommandline ... View more

The search I provided you should be able to run directly from your license master (if you have one on prem) or if each Heavy Forwarder has it's own license installed, you could run it on each one. I would highly recommend setting up a license master if you don't have one already. ... View more

Typically apps that haven't been certified in Splunk Cloud yet will have to undergo an app vetting process which can take some time. This is for a number of reasons related to the security and overall performance of your Splunk Cloud infrastructure. Do you have a sales rep at this point that you can reach out to to discuss? If not I would recommend contacting sales@splunk.com and explain your situation. ... View more