The documentation clearly states that "[...] forwarder management creates the serverclass.conf file under $SPLUNK_HOME/etc/system/local on the deployment server." I have the same problem as you in 7.0.2, so apparently this bug hasn't been fixed yet, or if it isn't a bug, the documentation isn't clear on this matter. ... View more

Take a look at these: https://www.digitalocean.com/community/tutorials/how-to-use-journalctl-to-view-and-manipulate-systemd-logs https://www.splunk.com/blog/2015/04/30/integrating-splunk-with-docker-coreos-and-journald.html The latter will do what you want, works great (just omit the Docker lines for the systemd unit). You basically write the journal to a json file. Splunk loves json and it will map the fields perfectly. If you don't restart the service much, make sure you rotate or clean the file periodically, or it will continue to grow. ... View more

Thanks for your answer! So, if we use one of these storage types, is the data still accessible from Splunk Cloud? I mean, Splunk can set it up for me so that when data is rolled to e.g. cold it is moved from the indexer in the cloud to some data storing service that can store bigger amounts of data? Also, I see that it's possible to buy storage increments of 500 GB for Splunk Cloud, but I'm guessing this is rather expensive compared to moving the cold/frozen data to e.g. a local storage. ... View more

You are absolutely right! The app I was looking at was from a former Splunk Cloud Trial instance. There is another app for the new prod-instance of Splunk Cloud which has all the servers listed. That surely clears things up. Thanks! ... View more

Have a similar issue using Insights for Infrastructure. SMTP settings correctly set up. Alerts set up, but nothing showing up in python.log (empty 0kb) and no maillog.log file found. Are there other tests I can run from within Insights For Infrastructure to ensure splunk can send mail out and then we can focus on alerting? any help is appreciated ... View more

HF is not A SH. It does index-time parsing except for indexing (should not). So it’s basically an indexer. Hence, give it IDX role in DMC/MC. ... View more

Thanks for coming back with the solution! ... View more

Thanks! That explains it. 🙂 ... View more

I woud add that, instead of using: | fields * it is better to extract only the fields you need later on in all the other dashboard panels, this will improve the performance of the entire dashboard, here below the example: | fields field1, field2, field3 etc.. ... View more

Thanks a lot, that made things clearer! I've used Cisco systems before, and I'm used to access lists where you read from the top down and use the first rule that applies to your current file. Of course this wouldn't make sense here, where the first default rule is an "accept all"-rule. ... View more

Indeed, found the reason: Seems that some older forwarder versions have indeed a different input configuration. ... View more

Hi, I have a follow-up question. Whenever I create an index in Splunk Cloud I can set the retention time to whatever I want, so where does the "default" retention on 90 days take effect? Is it so that if I set a retention time to 150 days, data will still be deleted after 90 days? ... View more

That worked perfectly, thanks! I noticed that the addinfo command add fields to all events in the search. Isn't this a bit resource demanding considering the information I want to extract? I'm not complaining though, this solution is fine by me. ... View more

I just checked, the build number in app.conf is 274527, the same number as you. Though, now the "update to 1.2.2" link is gone, as pointed out in my comment below. Is it possible that this build number was indeed wrong (not 274527), but got updated to the right build number (274527) at some point if e.g. Splunk was restarted? If not, my best guess is that the "update to 1.2.2" link on the Splunk DB Connect v1 app already in version 1.2.2 was a bug. ... View more

Thanks a lot! I'd started looking at the append command before your answer as well, and it seems like the way to go. The way you described the use of the append command in your example worked flawlessly for the alert I'm creating. I encountered a new problem when appending a third search to the alarm. The searches are so long that my server can handle them when all three are combined into one three times as long search. Fortunately it turned out that the third search wasn't needed, just the other two, and now everything works just fine. Still, the two searches are very similar, and it seems kind of unnecessary to run both of them separately. I'm sure there must be a better and less resource demanding way. I'll look more into optimizing the alarm when I got more times on my hand. Best regards! ... View more

Maybe I'm missing something here, but I can't see that this app has a tags.conf, which I understand is needed to map logs to CIM. My events looks like the samples presented by Splunk in the documentation. http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/MonitorActiveDirectory#Sample_AD_monitoring_output ... View more

Clustered indexers have a 'cluster master' (CM) that manage their configuration files. So a hybrid set-up is going to complicate things with respect to configuration file precedence and overall management. Ideally, you'd migrate the existing configurations to the CM. If you're going to convert, I would suggest setting up a set of small test VMs with IDXes having existing data, and convert them to familiarize yourself with the process. Testing is going to be especially important if you want to migrate your existing configurations to the cluster, since I doubt that the two IDXes have consistent configurations right now, given the partitioning scheme. As for the partitioning aspect, do you have a UNIX system administrator in your organization? If so, you might want to sit down with him/her and discuss your requirements. Since you are in a VM situation, you could get a small-ish partition (20GB or so), and mount that at a temporary mount point, move /opt/splunk excluding /opt/splunk/var/lib to the new, smaller partition (or copy to new and then delete old). Then change /etc/fstab to mount the new partition at /opt ; mount the existing partition at /opt/splunk/var/lib . Essentially, *NIX doesn't care where a partition is mounted; you just need to make sure when you set the fsck order that partitions that are closer to root are have lower fsck numbers, since the system can't mount the partition until the fsck is complete. Is it possible to build new systems with consistent partitioning and migrate the data over to them? That might be cleaner, overall. It would consume temporary resources but allow you to release the existing resources when the migration is completed. ... View more

Thanks! I did read this in the doc, but I was wondering about the strictness of this requirement. As per today we're running search heads on Windows and indexers on Linux, and it's working perfectly fine, though this is not a clustered environment. It makes sense that the indexers would have to be running the same OS when clustered, but does the search heads really have to be running the same OS as well? Or is his "only" highly recommended? ... View more

We found the cause of the error. The datastream for the index wasn't being parsed correctly; the milliseconds was not being extracted. This meant that if say 100 events all occured in the same second, as Splunk wasn't extracting the milliseconds, they would have all appeared to have happened simultaneously from a Splunk perspective. This apparently confused Splunk, and performance went bad. Fixed the parsing, and now the we can search on the index just fine. ... View more