It sounds like there may be one of two problems. The pdfserver may not be able to render the pdf The Splunk server is not delivering the search result to the pdf server For issue 1, I would double check the error to make sure it is truly a timeout issue and not a local firefox process issue. Does firefox have proper resources to render the pdf? For issue 2, it is possible that the search is timing out internally. I would check to see how long the search takes to run and see if it is hitting some sort of limit internally. Also, sending a 10k event result might be hitting another limit depending on your search. The following debugging page might be helpful: http://www.splunk.com/wiki/Community:TroubleshootingPDFServer ... View more

The dedup command can be used to create results that do not include duplicates: http://www.splunk.com/base/Documentation/latest/SearchReference/Dedup ... View more

Per another thread: You can check the HTTP auth tokens endpoint to see the session keys that are valid and can be used to access splunkd. https://splunk-server:8089/services/admin/httpauth-tokens http://answers.splunk.com/questions/3768/how-do-you-find-out-who-is-logged-onto-splunk-right-now ... View more

Index wise, you should be able to restrict the metadata. I'm not completely sure about your particular application and would like to see the specific role filters. ... View more

Splunk has the concept of a line and event breaker. The line breaker will separate the lines within a source, whereas the event breaker will dictate when an event is multi-line or not. For your scenario, it sounds as thought you need to tune the event breaker to properly separate the events. In some situations, Splunk will try to combine events if it does not see a timestamp on each line. Again, this depends on how you have set the event breaker. For your scenario, you have said that Splunk shows these events have hundreds of lines implying that the line breaker is working correctly. To turn off line merging, you can set the SHOULD_LINEMERGE to False under your specific sourcetype within your $SPLUNK_HOME/etc/apps/search/local/props.conf file: [your_sourcetype] SHOULD_LINEMERGE = False For more detail on settings, specifically for what dictates multi-line events: http://www.splunk.com/base/Documentation/latest/Admin/Indexmulti-lineevents ... View more

There are various reasons why a search may not return, but typically they can stem from the following: the search query does not span the right data or timerange the data does not exist (it's surprising how often this happens) there is a problem with the indexes in some way If you have run wild card searches that span all time ranges for all indexes (including _internal) and receive no results, then I suspect you have a problem with the indexes. For this situation you should log a case with Splunk support. ... View more

There is a setting that dictates the maximum number of results that will be sent with any alert. This is the maxresults parameter that resides in the alert_actions.conf file. By default this is set to 100. For reference, you could set it to 2000 by adding this line to a $SPLUNK_HOME/etc/system/local/alert_actions.conf file: maxresults=2000 http://www.splunk.com/base/Documentation/latest/Admin/Alertactionsconf ... View more

There is a setting that dictates how long firefox will wait to receive the completed search from the Splunk server. You can increase this timeout by setting the firefox_timeout value on the pdf_server side. This setting would reside under the [settings] stanza in $SPLUNK_HOME/etc/apps/pdfserver/local/pdf_server.conf. [settings] screenshot_enabled = True restrict_to_splunkweb = False firefox_timeout = 340 I believe the default is a four minutes (240), so increase this to something greater than the search job duration. ... View more

You can perform conditional routing where the destination key gets set to your specific index. ... View more

This looks like a form search of some sort. What is the actual search you are running? ... View more

Let's assume you have multiple fields values in your "puppet-facts" including duration and status. To chart any of these in combination with host, you could do the following search: sourcetype="puppet-facts" | dedup host | chart count by host,duration,status The above search uses the chart command to create a listing of the vent count with the host, duration, and status fields. ... View more

You have too many questions lumped together. You should break them apart. ... View more

I recall using these values simply to track what version of an app I have built. I have not seen it used otherwise. ... View more

Sounds like your Splunk indexer does not have Receiving turned on, or there is a problem with network connectivity from your forwarder to the indexer. First, verify Splunk receiving is enabled on the indexer. Two commands can verify this: splunk cmd btool inputs list | grep 9997 A listing for you splunk tcp input stanza should be returned. netstat -an | grep 9997 A "Listen" line should be shown with port 9997. Second, verify the Splunk forwarder is not blocked from contacting the Splunk indexer: telnet <splunk_indexer_host> 9997 If you do not get a connection established, then there is likely a deeper problem which will require help from support. ... View more

There are many reasons why indexing can be paused, but it typically stems from a blocked queue or max disk space is reached. If searching is indeed functional, then you can try the following search to see what queue is blocked: index=_internal source=*metrics.log* blocked If your system is unable to search, then you likely have a corrupt index that needs repair. You should inspect your $SPLUNK_HOME/var/log/splunk/splunkd.log and look for FATAL or ERROR messages. Those messages should hint at what is causing the problem. ... View more

I would not suspect this to be a performance problem in any way. The line breaking/merging functionality should not be affected by poor performance, as the system will just index or search at a slower rate when encountering resource issues. Are you positive you loaded the exact same set of data, in the exact same scenario as it gets input to your system? If you did that AND you set the correct line breaking (syslog settings) as suggested, then you should probably create a support case. ... View more

You have two questions here which you should separate. ... View more

First you need to verify that the Splunk instance on the Windows Server is forwarding to the Linux box. Once you verify that, then you can modify your inputs.conf on the Windows Server to accept the new file input. Once you restart the Forwarder, it should begin streaming data to the Splunk indexer. There are multiple questions and answers regarding troubleshooting of Splunk Forwarders, which you should review. Some common remedies for your symptoms include setting the correct index (existing on the Splunk indexer) and setting up the forwarder topology properly. ... View more