We are using a Cloud Foundry for an Internal Cloud Implementation. We are migrating applications and hence using TCP streaming versus the Splunk Forwarding Agent to stream logs. The Cloud Support Team is observing errors logged because the TCP stream is backing up. I am using a vanilla TCP configuration. Are there extra configurations I can add to the stanza to increase the buffer to prevent messages from dropping? Sample Vanilla TCP configuration: [tcp://3301] connection_host = dns index = index_name sourcetype = rfc5424_syslog Logs from the Cloud Foundry Doppler Service: {"timestamp":1474911116.219854593,"process_id":17528,"source":"doppler","log_level":"warn","message":"TB: Output channel too full","data":{"appId":"d41c5c78-955a-4148-b575-cf868dc0b6fe","destination":"syslog://tlaloga1.dev.prod.travp.net:3303","dropped":99,"total_dropped":496},"file":"/var/vcap/data/compile/doppler/loggregator/src/truncatingbuffer/truncating_buffer.go","line":112,"method":"truncatingbuffer.(*TruncatingBuffer).forwardMessage"} {"timestamp":1474911142.510904074,"process_id":17528,"source":"doppler","log_level":"warn","message":"TB: Output channel too full","data":{"appId":"d41c5c78-955a-4148-b575-cf868dc0b6fe","destination":"syslog://tlaloga1.dev.prod.travp.net:3303","dropped":99,"total_dropped":595},"file":"/var/vcap/data/compile/doppler/loggregator/src/truncatingbuffer/truncating_buffer.go","line":112,"method":"truncatingbuffer.(*TruncatingBuffer).forwardMessage"} {"timestamp":1474911161.138316393,"process_id":17528,"source":"doppler","log_level":"warn","message":"TB: Output channel too ... View more

What is needed to change Splunk to only index using the System Date/Time? I have data indexed today with a date of 2030 and 2001. ... View more

I am setting up Splunk 6.3.2 to run on a new Red Hat 6 server and migrated from a Red Hat 5 server. I installed Splunk using the 6.3.2 rpm and Splunk works fine. I tar'd the contents of /etc on the old RH5 server and untar'd them on the new RH6 server. I have done this in the past with no issues (RH5 to RH5). Email is the only thing not working, and in the ~splunk_home/var/log/python.log I see a bunch of Connection Errors when trying to send email. What did I do wrong here when converting? 16-02-25 09:09:13,810 -0500 ERROR sendemail:378 - [Errno 111] Connection refused while sending mail to: [email protected] 2016-02-25 09:10:46,042 -0500 INFO sendemail:985 - sendemail pdfService = pdfgen 2016-02-25 09:10:46,043 -0500 INFO sendemail:1117 - sendemail:mail effectiveTime=1456409400 2016-02-25 09:10:47,330 -0500 INFO sendemail:1137 - Generated PDF for email 2016-02-25 09:10:47,407 -0500 ERROR sendemail:115 - Sending email. subject="Splunk Alert: Distributed_Alert_MI_(Disk Percent Free)", results_link="http://tlpsplu1:8000/app/search/@go?sid=scheduler__userid__search__RMD5ac90a8a41fbc3d92_at_1456409400_33993", recipients="[u'[email protected]']", server="tlpsplu1.domain.net" 2016-02-25 09:10:47,407 -0500 ERROR sendemail:378 - [Errno 111] Connection refused while sending mail to: [email protected] 2016-02-25 09:11:15,807 -0500 INFO sendemail:985 - sendemail pdfService = pdfgen 2016-02-25 09:11:15,808 -0500 INFO sendemail:1117 - sendemail:mail effectiveTime=1456409460 2016-02-25 09:11:17,099 -0500 INFO sendemail:1137 - Generated PDF for email 2016-02-25 09:11:17,214 -0500 ERROR sendemail:115 - Sending email. subject="Splunk Alert: Distributed_Alert_MI_(Disk Percent Free)", results_link="http://tlpsplu1:8000/app/search/@go?sid=scheduler__userid__search__RMD5ac90a8a41fbc3d92_at_1456409460_34020", recipients="[u'[email protected]']", server="tlpsplu1.domain.net" ... View more

We are planning a Private Cloud implementation of Java applications using a Cloud Vendor. We use Splunk Forwarders on our existing static servers. I have started to research different ways to stream the logs to Splunk once we starting using the Paas Solution. The suggestions I have read are to stream the logs to syslog and use a Data Sink to push the logs to the Splunk Indexers. Then we would need to set up a new UDP listener on each of our Indexers. This seems like a really complicated way to do this because we would have to set up a UDP port for each sourcetype for each index we want to index. The other thought was to use a custom log4j appender to stream the logs from the container to the Splunk Indexers. This seems like a better way, but it doesn't look like you set up the logs to to separate the logs by index and sourcetype. Is there an easier way to stream the logs into Splunk like the Splunk Forwarder provides? ... View more

I have a timechart for the last 7 days and I want to add a percentage of the two results returned from the timechart data. I know I need an eval to add a total of the two results and then divide the result of the value that I want the percentage by total. I think I would need a table to format the data into a column format. index=ui_prod sourcetype=instrumentation processCode=NB (val1 OR val2) earliest=-7d latest=-1m | timechart span=1d count as Number_Clicks by dataValue I want the end result to look like this: _time val1 val2 Val2PercentofTotal 2015-07-20 7 3 30% 2015-07-19 160 40 20% ........... ... View more

I've got a SQL Table that contains XML Blobs that I want to index in Splunk. The XML Blob contains carriage returns. When indexing, Splunk is creating a new row for each line in the XML Blob. I'm sure this can be worked around by modifying one of the .conf files in the Splunk DB Connect App. I would like one entry in Splunk for each row of data in the database (not one entry per line of XML in all the rows in the database). Desired: Entry 1: "date id=123 xml_results=contentn body /body n /content " Entry 2: "date id=124 xml_results= content n body /body n /content " What is happening now: Entry 1: "date id=123 xml_results= content " Entry 2: body /body Entry 3: /content Entry 4: "date id=124 xml= content " ... View more

How can I create a field extraction to modify a key in a key value pair? I have a new file that I am indexing. The key is using "source" and this is conflicting with the built in "source" in Splunk. Making this change when the file is getting indexed will not work in this situation. I am looking to do this at Search Time. 2014-07-23 09:59:56,996 || Thread=9 || channel=CONTACTVIEW || endTimeRaw=1406123991420 || duration=3786 || startTimeRaw=1406123987634 || source=Portfolio || endTime=2014-07-23T13:59:51.420Z ... View more

I have just upgraded to Splunk 6.1. I have daily reports that are sent from Splunk. Before the upgrade the report would get sent and the content of the report would be in the header as well as a record count. Now there is a link to the report in Splunk where you can get the details. How can I get back at a bare minimum the total count of rows returned in the report without clicking on the link? This was suggested: If you are talking about email alerts, in Splunk Enterprise 6.1 you can customize the email using tokens. See the Set up alert actions topic in the Alerting Manual for the specifics. Would this work? To add Row Count, would savedsearches.conf have this configuration now? Where $job.result.Count$ is new? [Issue Policy Numbers from States] action.email = 1 action.email.inline = 1 action.email.reportServerEnabled = 0 action.email.sendresults = 1 action.email.subject = Splunk Alert: ReportName action.email.to = email addresses alert.digest_mode = True more configurations here vsid = hmm5pnnl $job.resultCount$ ... View more

I have just upgraded to Splunk 6.1. I have daily reports that are sent from Splunk. Before the upgrade the report would get sent and the content of the report would be in the header as well as a record count. Now there is a link to the report in Splunk where you can get the details. How can I get back at a bare minimum the total count of rows returned in the report without clicking on the link? ... View more

I would like to trim down a field to 5 characters using an Excel Left Logic. I have read some suggestions to use LEN, but the length of the field is not fixed. In Python I would do something like this: newvariable = 'blahblahblah' newvariable[0:4] blahb ... View more

We are looking to use Log4J to stream log files to Splunk Indexers. In my Splunk environment I have three Splunk Indexers. I found configurations that can be used for non clustered Splunk Environment. I have other servers that use a forwarder with a Splunk Indexing Group configured. What modifications are needed Log4J Appender to write to multiple indexers the way that the Splunk Forwarders stream log files? ... View more

I have a log file that is writing session data for users using an application in a csv format. The session data provides information about each transaction using "Meta" events and action information (Page Loads, Application Initialization). The sessionId is common to both types of events and is used to link the events. There is a one to many relationship between the action events and the meta events. (Each Action Event will have many Meta Events associated with it.) I want to write a search that will add the information provided from the Meta information to the action information. I started using a join, but after running the search it looks like the search is only pulling back the meta information from the first meta tag. I have been able to push this data to Hadoop and run Ruby to "Sessionize" the data, but I want to be able to do this directly in Splunk. Below is the join search. Is "Join" the right search to create a one to many relationship? Can I create a Subtable of Meta Data to each action event? index=prod_ui sourcetype=ui_instrumentation Type=init OR Type=view | rename Type AS Type1 | rename SubjectName AS SubjectName1 | rename DataValue AS DataValue1 | table _time, SessionId, Type1, SubjectName1, DataValue1, Duration | join SessionId [search index=prod_ui sourcetype=ui_instrumentation Type=meta | table SessionId, DataValue, SubjectName, Type] ... View more

I have a clustered environment and using a Master Node to handle replication. One of my peers crashed and I had to restore Splunk on that server. That is all complete and I have rejoined the peer to the cluster. The host name is different but the IP has been assigned to the rebuilt server. How do I remove the rogue server from the Master Node? ... View more

A developer made a change to the logging that they were managing. They added a new Key Value Pair and the results now have spaces ie Operation=Web Service Call: callABCService. Splunk Search is classifying this as Operation=Web. Is there a quick fix that I can make in Transforms or Props to work around this? Thank you ... View more

My company recently has built a Hadoop Cloudera cluster build: 2.0.0-cdh4.1.2. I am using Splunk 5.0 on a Windows Server 2003. I have installed the Hadoop Connect app and Java 6u27. Has anyone had success in finding the install for the CLI for Hadoop Cloudera that will work on a Splunk Hadoop Connect that runs on a Windows platform? ... View more

I haven’t set this type of input before. I have logs available over http from a URL like below. The typical user views the log in their browser. What is the easiest way to configure my inputs.conf file Index and Different Sourcetypes. http://servername/sta4j_logs/JVMName/ServiceName.log ... View more

I am looking for a way to deploy Splunk Forwarders to a Server Farm. I would like to package the installation and then copy an inputs.conf file to the server after the deployment. Is there a way to configure an Environment Variable for the Host Name so the computer name is used like %COMPUTERNAME%? I don't want to have to make a manual change for every server in the server farm. [default] host = %COMPUTERNAME% I tried this earlier and the Indexer used %COMPUTERNAME% as the Host Name. ... View more

How does the Splunk Universal Forwarder handle the condition when SPLUNK TCP is used as the communication method and the Splunk Indexer is down? (maintenance, someone disconnects the server, ...) ... View more

I have a new log that I need to define a sourcetype for. There are several different ways the fields are delimited. The formatting is defined in a Log4Net.config file is below. Is there an easy way to configure this in the transforms.conf file? 2012-10-17 12:06:43,546 [53] INFO ServiceCalls [(null)] - ServiceCall CRDBServiceHook.GetPolicyData 203.125 2012-10-17 12:06:43,656 [6] INFO ServiceCalls [(null)] - ServiceCall CRDBServiceHook.GetPolicyData 203.125 2012-10-17 12:06:43,765 [53] INFO ServiceCalls [(null)] - ServiceCall CRDBServiceHook.GetPolicyNameAddress 156.25 2012-10-17 12:06:43,828 [53] INFO HTTPPages [(null)] - PROD\AOSHOGAY HTTPPage http://plcontactview.prodlb.travp.net/plcontactview/PLContactSearch.aspx?txtClientFunction=InsuredSearch&ixn_subtype=null&ixnID=02420216ac9589cd&ixn_type=null&txtBAOrPol=9872514641011 671.875 GET 2012-10-17 12:06:43,875 [6] INFO ServiceCalls [(null)] - ServiceCall CRDBServiceHook.GetPolicyNameAddress 203.125 2012-10-17 12:06:43,921 [6] INFO HTTPPages [(null)] - PROD\BROBERT4 HTTPPage http://plcontactview.prodlb.travp.net/plcontactview/PLContactSearch.aspx?txtClientFunction=PhoneSearch&ixn_subtype=null&ixnID=01d00216ab58c5e0&ixn_type=null&txtPhone=8047526988 640.625 398 POST 2012-10-17 12:06:44,234 [6] INFO ServiceCalls [(null)] - ServiceCall CRDBServiceHook.GetPolicyData 250 2012-10-17 12:06:44,265 [41] INFO ServiceCalls [(null)] - ServiceCall PreferenceManagementServiceHook.GetPreferences 218.75 2012-10-17 12:06:44,328 [16] INFO ServiceCalls [(null)] - ServiceCall CRDBServiceHook.GetPolicyData 234.375 2012-10-17 12:06:44,328 [53] INFO ServiceCalls [(null)] - ServiceCall PreferenceManagementServiceHook.GetPreferences 156.25 2012-10-17 12:06:44,421 [41] INFO ServiceCalls [(null)] - ServiceCall CRDBServiceHook.GetClaimData 156.25 2012-10-17 12:06:44,437 [41] INFO HTTPPages [(null)] - PROD\BROBERT4 HTTPPage http://plcontactview.prodlb.travp.net/plcontactview/PLBillingTreeView.aspx 421.875 GET 2012-10-17 12:06:44,437 [6] INFO ServiceCalls [(null)] - ServiceCall CRDBServiceHook.GetPolicyNameAddress 171.875 2012-10-17 12:06:44,531 [41] INFO ServiceCalls [(null)] - ServiceCall CRDBServiceHook.GetProducerData 46.875 2012-10-17 12:06:44,531 [53] INFO ServiceCalls [(null)] - ServiceCall CRDBServiceHook.GetClaimData 203.125 2012-10-17 12:06:44,546 [16] INFO ServiceCalls [(null)] - ServiceCall CRDBServiceHook.GetPolicyNameAddress 187.5 ... View more

I have an index to consolidate the IIS logs for an application. I have 6 servers that handle load balancing. The IIS logs are in Splunk. Is there a search parameters can I use to determine the peak concurrent users at a point in time? ... View more