Hello, I have a setup that consists of a Search Head and 2 indexers in a cluster. I also use a self signed SSL certificate between the indexers and my universal forwarders. For some reason, my UF is able to connect to the indexers, but no data is sent. 07-09-2016 00:21:15.670 +0000 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997 My test logs directly on the indexers were sent to the search head without issue. running splunk list monitor on the UF lists all the logs I want to monitor No errors in splunkd.log on the UF that I can see, just the usual warnings I get in my duplicate setup in another enviornment that IS working. No errors in metrics.log on the UF either. On the Indexer is this warning: WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Fri Jul 8 20:30:38 2016). Context: source::\ \s$\Logs\service.log|host:: |Service Logs|174315 What else can I test to pinpoint my issue? ... View more

Hello, I am trying to add two more indexers to our current Splunk setup. Our current setup is a search head and two indexers using replication. These are all Linux machines. I have attempted to add the first one, but it is not working. The error I get in Distributed Search >> Search Peers on the search head is: Error [00000080] Failed 12 out of 11 times.REST interface to peer is taking longer than 5 seconds to respond on https. Peer may be over subscribed or misconfigured. Check var/log/splunk/splunkd_access.log on the peer The splunkd_access.log does not have much useful info that I can see. Another strange thing is the warning on the search head lists an IP value that does not exist for this new indexer, and I have no idea where it got that IP from (server IP should match what is in name. These are just random IP values I replaced the real ones with for sample purposes): Failed to add peer 'guid=CA7694EA-40EE-4B40-8506-DAFD18BCAB2E server name=ip-99-8-321-101 ip=99.0.4.23:8089' to the master. Error=http client error=Connection refused, while trying to reach https://99.0.4.23:8089/services/cluster/config Here is how I set up the new indexer: 1. Created the linux Machine and installed 6.4 enterprise on it 2. Copied over the following conf files from a previously running Indexer: server, alert_actions, authentication, authorize, props, transforms, web 3. Copied over server.pem from etc/auth 4. When trying to restart splunk, The Waiting for web server at http://127.0.0.1:8000 to be available.... would never connect 5. Thought it might be because I copied over the server.pem, so I restored the backup. 6. Commented out the sslKeysfilePassword in server.conf 7. Tried restarting again, and splunk came up However, now I am getting the two errors I mentioned above. Questions 1. Where is that IP value coming from in the search head error? 2. Are there other conf files I should not have copied over? 3. Did I mess the server up as soon as I copied over the other indexer's server.pem and tried to restart splunk? 4. Do I need to change the pass4SymmKey in server.conf? Thank you! ... View more

Hello, I don't quite understand the difference between the current_size_kb value and current_size value in the metrics.log for a Universal Forwarder. This is for the parsingqueue, as I am getting the "Could not send data to output queue" error in splunkd.log sometimes. Metrics.log example line: 05-05-2016 19:03:06.209 +0000 INFO Metrics - group=queue, name=parsingqueue, max_size_kb=1228800, current_size_kb=1226982, current_size=19790, largest_size=19815, smallest_size=19692 So current_size_kb is almost the same size as my max_size_kb, but then current_size is only ~20MB. I started my maxSize value for parsingQueue in the server.conf at like 200MB, and every time I increased it, current_size_kb would increase as well. Do I need to increase maxSize even more? Or should I just pay attention to current_size? Thank you ... View more

Hello, Say I wanted to create a table with the fields State, City, City Count, and Total. If I try to use |stats values(city) as city, count by State I get a chart that only lists cities, but does not give their individual total, only the total for State. Is it possible to get my chart to look like this: | STATE | CITY | city_count| Total | |CA | San Jose | 5 | 15 | | | Santa Clara | 10 | | the last table in the answer here shows it better: https://answers.splunk.com/answers/97676/getting-counts-on-multiple-fields-while-grouping-by-one-of-those-fields.html I don't have an "mcount" field, so is this still possible to get the count of each city there? My table would have srcip as State, malware_type as city, and mcount as the total count of each city. Also, I tried using the solution using chart shown here: https://answers.splunk.com/answers/215808/how-to-get-stats-count-results-broken-down-per-fie.html?utm_source=typeahead&utm_medium=newquestion&utm_campaign=no_votes_sort_relev however that table is a lot harder to follow(it lists each city as its own column). Thank you. ... View more

Hello, I have a few servers with UniversalForwarder installed, and Splunk4JMX app on them. I have two servers that appear to have the forwarders setup the same, however; One server Splunk4JMX works great and jmx data gets sent to the indexers The other server nothing is sent over(no errors in the Splunk4JMX\log folder either). The only way I can get data to splunk indexers is by manually running: C:\Program Files\SplunkUniversalForwarder\bin>splunk cmd "C:\Program Files\SplunkUniversalForwarder\etc\apps\SPLUNK4JMX\bin\poll_jmx.bat" config.xml Both servers Spunk4JMX inputs.conf file is the same: [script://./bin/poll_jmx.sh config.xml] interval = 60 sourcetype = jmx index = jmx disabled = 1 [script://$SPLUNK_HOME\etc\apps\SPLUNK4JMX\bin\poll_jmx.bat config.xml ] interval = 60 index = jmx disabled = 0 sourcetype = jmx [monitor://$SPLUNK_HOME/etc/apps/SPLUNK4JMX/logs] disabled = false followTail = 0 index = jmx sourcetype = jmx_errors Both servers config.xml is basically the same, except different server values. According to the README file, Splunk4JMX is version 1.6 Question Is there another setting I am missing to check between these two servers? I assume that since I can manually run it on the broken server, it is not a port/network issue between the server and the indexer. A permissions issue with the user that splunk was installed on maybe? Update From the splunkd log: + 4 12-06-2015 14:18:29.139 +0000 ERROR ExecProcessor - message from ""C:Program FilesSplunkUniversalForwarderetcappsSPLUNK4JMXbinpoll_jmx.bat" config.xml" Access is denied. ... View more

Hello, We would like to be able to have our universal forwarders that are installed on AWS instances, to forward logs/indexes to an S3 bucket(instead of an indexer). Our client will then use their own splunk to read them. Is this possible? We want to avoid just sending all our logs for this client to S3, as we have a lot of different logs, and maintaining an in house script to this would be high. So using splunk is our first choice, but I have been unable to find this feature specifically. ... View more

Hello, I am trying to set up WMI on a universal forwarder, however, I am only getting WMI:CPUTime. The WMI:WinEventLog:Security is not working though. I tried following http://docs.splunk.com/Documentation/Splunk/6.2.4/Data/MonitorWMIdata but that is for all Windows servers, and not Linux. My setup Search head and main UI on Linux 2 distributed indexers also on Linux Servers to monitor are on Windows My wmi.conf file is on a Windows server that has universal forwarder installed. (All other logs being sent from this server are coming in) [WMI:CPUTime] interval = 10 disabled = 0 server = localhost wql = SELECT PercentProcessorTime, PercentUserTime FROM Win32_PerfFormattedData_PerfOS_Processor WHERE Name = "_Total" [WMI:WinEventLog:Security] interval = 10 disabled = 0 server = localhost event_log_file = Security Do I need to set something else up for security to work? What can I check to verify the event_log_file is being created? Is there a way I can use the wql parameter with security instead, since that works for the CPUTime? Thank you ... View more

Is it possible to directly put cold and frozen data on s3 in AWS? The only solution I found was through shuttl, from this question: https://answers.splunk.com/answers/56522/frozen-archives-into-amazon-s3.html?utm_source=typeahead&utm_medium=newquestion&utm_campaign=no_votes_sort_relev However, we would like to find a solution that does not use a script. Is this possible? Thank you ... View more

Hello, I am trying to add another search peer to my search head, through the Settings > Distributed Search menu. I do not have a username or password to connect to the search peer I want to add however, as I connect to it using putty and a ppk file. A week ago I was somehow able to add my first search peer, which has the same issue above(ppk file only for logging in), but I can't remember how I did it. Any ideas how I might have done it? thank you ... View more