First, you need to be using a Heavy Forwarder, and not a Universal Forwarder. Which do you have in your environment? Make sure you read the following : http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad It outlines requirements to do what you are describing. Aside from the, run btool against your outputs on that forwarder and make sure your configuration is applied. ... View more

I hope youre not running your 24x7 SOC off your laptop. If you have a enablement license, as in you have purchased Splunk Enterprise and Enterprise Security, you will need to install the provided license files (that your sales representative sends to you) on your License Master Instance. Your best place to start is at the documentation for Enterpise Security. Read from here : https://docs.splunk.com/Documentation/ES/5.0.0/Install/InstallEnterpriseSecurity ... View more

Can you elaborate more on this, and additionally, provide the commands you are running with the exact syntax. ... View more

Not sure on that, havent used the nexus in a few years. But by default in IOS, it is. HEre's a link to start : https://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/5_x/nx-os/system_management/configuration/guide/sm_nx_os_cg/sm_5syslog.html Against, run tcpdump / wireshark on the host. If its being sent over TCP, or a different UDP, it will catch it. ... View more

Im going under the assumption that you have already configured Splunk to receive UDP data on port 514, via this : https://docs.splunk.com/Documentation/Splunk/7.0.3/Data/Monitornetworkports#Configure_a_UDP_input The next step would be to make sure that your network can receive syslog over UDP from your nexus. Use a tool like tcpdump or wireshark on your Splunk box to see if the UDP/Syslog traffic is actually going to the box. If it is, then you need to re-read the above link and recreate the UDP input in a inputs.conf file on your Splunk instance. Otherwise, if TCPdump/wireshark doesnt see your syslog stream, then most likely you have a network configuration issue on the Nexus. ... View more

You need to look at your indexers, under your defined index volume ( default is $splunk_home/var/lib/splunk/ ) and look at free disk space. You should also see a searchpeer name associated to this error message. Thats where you should look. Additionally, you may be having disk issues in regards to I/o. Make sure you indexers meet the minimum requirements (800 iops) for disk volumes. Otherwise you will see backed up indexing queues and have associated search issues with the peers. ... View more

It is not. When you have a SHC, all members of the SHC have a GUID that they share. (If you look in server.conf under the SHClustering stanza, you will see this.) All datamodel acceleration is bound to the GUID, so all searcheads share the accelerated data. In a SHC, there would be no benefit to having just one member hold this. So, again, the most direct answer is 'no, not possible.' ... View more

If you notice, the app called travelers_all_app_props isnt included in the ES App Import Naming Space. So ES will ignore this app, and its knowledge objects. You either need to follow the docs on how to import the app into ES, or rename it to TA-travelers_all_props Regarding best practices, this doesnt cause an issue per say, its just that Splunk recommends naming apps in a certain way in order to standardize and make troubleshooting deployments easier. ... View more

Currently there is no direct fix for this. Splunk Cloud customers can currently run AR on other cloud based services, or you can create your own AR that would connect back to your onpremise deployment. The later option does require a vetting process, and does have to adhere to guidelines as outlined here : http://dev.splunk.com/view/app-cert/SP-CAAAE85 One of the key take aways here, is that any type of outbound communication from Splunk Cloud has to be over SSL, and any credentials use for authentication cannot be stored in clear text. I'd recommend building your integration via the Splunk Addon Builder (TA Builder) : https://splunkbase.splunk.com/app/2962/ Cheers ... View more

There is no way to control onboarding of data sources from the indexer / cluster side. If the forwarder has the indexer splunktcp port, ssl cert, or autodiscovery password, they are trusted and data is accepted. ... View more

There is no mechanism internal to Splunk that would prevent a 3rd party from editing the inputs in a forwarder and adding additional data sources. There are a few ways you can remediate this kind of scenario though.. 1) Restrict access to indexers for only authorized forwarders. But if you have thousands of forwards, this is not easy, and additionally if someone has access to the forwarder, they can still add inputs/datasources 2) Update permissions on the forwarders themselves so that only authorized users can add inputs / data sources. This is not done within Splunk, but at the OS level. This also means that you someone on the other end should have access / permissions to Splunk and that you trust them not to add more sources. 3) Use Splunk to monitor your data sources. E.g., set an alert when there is an increase in data sources over the previous day. And then you can remediate by contacting the owner / deleting the data etc. Those are just a few ideas i have based on experience. There are other ways this could be done, im sure there will be more feedback on this. ... View more

First, do you have THP disabled? There are some pretty major bugs with Redhat and CentOS regarding Transparent Huge Pages and Splunk memory consumption. Disable this.. More reading here : https://docs.splunk.com/Documentation/Splunk/7.0.2/ReleaseNotes/SplunkandTHP Aside from that, have you opened a support case with Splunk? There is some tuning you can do in limits.conf on both the SH and the IDX, however, its best if you get those parameters recommended via official support. They can provide a more comprehensive view of how these may effect your system. ... View more

By the latest log file, do you mean the most recent log file? Splunk data is indexed based on time series data. This means that as you search data, and as long as you have it indexed based on the proper event time stamps, then the results you get will be displayed in the time from latest to oldest. (In your results view.) Drilling down a bit more without understanding your use case fully, you can also you the latest Event Order command in your stats pipeline in order to get the latest events based on the fields of your choosing : http://docs.splunk.com/Documentation/Splunk/7.0.2/SearchReference/Eventorderfunctions Those are a few options there... ... View more

You need to run this as root, or sudo run this command, because to modify /etc/init.d, you need admin / root privileges. Additionally you need to specify the run-as user: ./splunk enable boot-start --accept-license -user splunk So if you run the above as root, and substitute splunk with the username of the service account you want to Splunk to run as, then you should be ok. Additionally, before finalizing this, you should change the ownership of /opt/splunk to the user designated... Do a service splunk stop and then : chown -R splunk:splunk /opt/splunk After that, you can start splunk via the service command, or sudo to the user you specified and start it that way. Do not start Splunk as root, or any other user. It will screw with permissions and throw errors. ... View more

What time zone are you in, and what time zone is set in your GUI for Splunk. Additionally you can set the timezone in the props on your host and help alleviate this kind of issue. ... View more

I downvoted this post because while technically true, this doesnt present the solution from the splunk search point of view. ... View more

Do you mean this? ** http://docs.splunk.com/Documentation/Splunk/latest/Search/Addcommentstosearches ** Been around for a bit... ... View more

General rule of thumb is that UF doesn’t do parsing. There are some exceptions to this with structured data sources like CSV and indexed extractions however. So for you, these props and transforms do need to go on your indexers. ... View more

One thought on this, you first result, which you say is correct, is over a span of 6 hours with 434 events. Whereas the second graph is over the similar timespan, but with only 5 events. As an educated guess, I would say that this sparkline is accurate based on 5 events being loaded and aggregated. Look at those raw events, and most likely they are very similar in nature, as in 100%.. Whereas your first sampling is more varied as there are 434 results... ... View more