As has been hinted at, missing log data is often a symptom of source, indexer, and/or search head not agreeing on the time standard. If your data is consistently missing for the most recent hour, but then appears to back fill, you might find that there is a discrepancy causing the data you search for to be forward-dated by an hour. That happens when (for instance) the indexer is working to GMT and the source is working to BST and the indexer is taking its time reference from the log entry, not the current clock. (In that case the logs will be forward-dated by an hour, and any search up to the present moment will not find it. If, however, you consistently lose data during one specific hour of the day I would look to some daily task or other interrupting the log generation or forwarding on the host. (Premature deletion of logs by a daily process causing the data to go missing until the log service is restarted for instance, or something actually stalling the log generator and restarting it an hour later.) ... View more

syslog-ng is a superior implementation of a syslog server (replacing things like rsyslog and the like) under Linux, which allows for much more comprehensive formatting and sieving of the data into a structured hierarchical file tree. It is completely compatible with the syslog service protocol, and any device logging to it won't know the difference (although they wouldn't anyway, by default, because syslog is usually UDP not TCP). I would suggest you simply look up syslog-ng (it's by a company called Balabit), and if you don't know the distinction between TCP and UDP (I make no assumptions either way) you'd do yourself a favour to understand their differences. ... View more

Provided the proxy has routing to the indexer, knows to proxy on the port specified, and firewalling is not intercepting the traffic as being to a non-standard port, it should be fine. But you need to bear in mind that just because the search head can use the proxy for http/https traffic on the standard ports (80/443) doesn't mean anything with regard to its ability to relay the Splunk traffic (default destination ports 8089 and 9997 depending, on the nature of the traffic). ... View more

I have to disagree with this. Although you can assign it a password, you should not. Since splunk services run as root on bootup, the splunk account should not be provided with a login password, and limited su access should be through su-enabled accounts (with sudo ). ... View more

The splunk account will be automatically created during the install. Ideally it should not be a login account, and thus should not have a valid password. Anyone with superuser privileges can su to another account without password access. The best way to achieve that for a normal user is sudo (.i.e. as you sudo su - splunk ). If depending on how sudo has been set up for your account you may be prompted for a password (which will explicitly be your own current password), or you may be granted unauthenticated rights to perform specific limited or all commands through sudo. The downside is that it will not be setup for users by default, but it is the accepted "correct" way to perform administrative tasks. Logging in as "root" should only be a last resort. ... View more

I don't fully recall, but the UF's were configured by script, initially, and I think the ssl configuration was quite simply just missing in its totality. ~splunk/etc/system/local/server.conf [sslConfig] enableSplunkdSSL = true useClientSSLCompression = true useSplunkdClientSSLCompression = true ... View more

A variation of this question comes up periodically, and the basic answer is "no". A sourcetype is tied to the source in a one-to-many relationship. If your source is a set of zip files, then the sourcetype will apply to the zip files in their entirety, not their contents. A possible solution is a triggered script, which unpacks the zip and allows you to then ingest the component files individually as defined sources in their own right. ... View more

It's not a feature of the CSV standard, or of the file generation. CSV is not a complex data format, and the Euro symbol is not a simple character. CSV was designed for the era of pure ASCII (pre-dating even the extended code-page sets of the IBM PCs in the '80s), and only defines a mechanism for delimiting data. It has no concept of metadata, which would allow you to embed an encoding. As far as basic CSV encoding is concerned, the two-byte extended Euro symbol is just two bytes of data. Unfortunately, your problem is a feature of the target application not understanding (or being configured for) the correct extended character set. The very fact that your notepad (which is part of the base installation) is recognising the symbol demonstrates it is both present and correctly encoded. Your only valid solution is to ensure that the recipient applications are correctly configured in such a way that they are pre-disposed to recognise the correct language-encoding as the default (presumably ISO 8859-15). ... View more

That depends on what platform you are running on. (You don't specify.) On a *ix (e.g. Linux) it is as simple as disabling the service and then stopping id. sudo /etc/init.d/splunk stop sudo ~splunk/bin/splunk disable boot-start I imagine on Windows it is similar (stop the service, with the Windows interface, then get Splunk to do the boot-start disable). ... View more

I just had another thought, it could also be specified as @d-12h to @d-11h. ... View more

Ideally, you need to sanitise your incoming data at the point of production, rather than attempting to get Splunk to unravel its inconsistencies. Splunk is (perfectly reasonably) interpreting and recording reason=" " as being a field consisting of one space, at index time. When called upon, later, to regurgitate the value of that field that is all it produces. The fact that your data contains multiples of the same delimiter coupled with the inconsistency means it will be very difficult to frame a regex to filter it correctly. It would help if we could see a sample of a source log, to see the "reason" field in the context of its surrounding data, and you could then provide for it in your indexing configurations. ... View more

Missing serverclass is not going to cause the handshake to fail, surely? Unless things have changed in V6 it will just result in no class matches and hence an empty configuration deployment. The handshake will still complete. A tcpdump is a very quick and direct method of answering a whole bundle of fundamental network questions by direct observation and without the need for any circumstantial inference, before tinkering with configurations. You will know whether the packets are getting through, whether they are complete, the exact nature of the response if they are. There is absolutely no point in refining configurations if the fault lies on one end not talking correctly to the other. ... View more

They already mentioned that they have connectivity with a quick Telnet test. Admittedly I am assuming that the phrase "to the correct port" means what it says. ... View more

In that case - and assuming that your previously mentioned telnet test was correctly framed to the right port - I fall back to the suggestion of a tcpdump to analyse the actual network traffic at the deployment server. ... View more

You set up one LDAP "strategy" per LDAP tree you want to search (which might be one per LDAP server, or might be multiple per LDAP server to encompass separate sub-trees). But to simplify let's assume one server and one search tree. One of the big gotchas of LDAP is that by default most LDAP servers will not return more than 1000 records, and will truncate at that point, so you have to create your filters to ensure the required record set is within the data returned. For each strategy you have to provide a BindDN (and if necessary a password) because the BindDN is the authentication/authorisation for your search. If that BindDN does not have query access to the segments of the tree that you need to search, you won't see any results, and that's down to authentication/autorisations set up by the LDAP admin. Ideally you will have a BindDN specifically for your Splunk authentication, and nothing else, which has query access only to the data it needs (i.e. password hashes and group memberships relevant to Splunk role definition). In practice it will probably have access to a broader range of data than it needs, because granular access control across LDAP is a hairy subject. ... View more

So what you are saying is that you will always want the hour 12:00..13:00 of the previous day. OK, so that should be -d@d+12h to -d@d+13h . You can add and subtract offsets after the snap. ... View more

You have touched on my biggest gripe with Splunk's system architecture. /opt/splunk is a perfectly valid place to install Splunk, owned by the splunk user. However you should be a superuser to perform the installation. Furthermore - and this is the basis of my objection - you also have to run the entire Splunk instance with root privileges if you want inputs from system logs (unless, of course, you open up the file permissions). Kind of rock/hard place situation. (You'd think by now Splunk would have broken it down into the main engine and indexes dropping itself down to unprivileged status, and running a micro-service talking through the socket stack purely for accessing privileged logs.) Here's a typical install drwxr-xr-x 5 root root 4096 May 9 2014 /opt drwxr-xr-x 10 splunk splunk 4096 Jul 13 13:22 /opt/splunk 755 permission, and root ownership on /opt is perfectly normal. ... View more

Another question occurs to me, but not being Windows-centric I have no idea of the likely answer or how to find it: are there Windows access restrictions in force preventing the forwarder from obtaining the WMI security records? Clearly if you are seeing other Splunk entries from the machine in question, there is no network fault in play. ... View more

You could distribute the licencing, but first you would need to aggregate it on a master licence server serving both environments, and then redistribute it by pools. Also, this pre-supposes that the different environments don't represent different legal entities, and that by doing so you are contravening the terms of both licences. Of course that's a legal restriction, not a technological one. ... View more

There are so many different reasons why logs might not be appearing, that no, a simple search access would not be the full route to diagnosis. It might supply the answers, if they are the sort of issues logged in the audit logs. But it could simply be that the data is not being logged at all, and that could be down to any or all of misconfigured endpoint forwarder or indexer, SSL faults, broken network, local file permissions (all of which could be absent from the indexer because of the problem itself). ... View more

[Edited to preface with the caveat that I am assuming your initial telnet test is correctly framed, and that your idea of "the correct port" is 8089.] Are you using SSL? Is it correctly configured? The only real way to judge here is to run comparative tcpdump s from a working machine (preferably one in the same routed network zone - assuming there is firewalling and segragation going on here) and for the one which is failing (which since it is a 'Doze box would require a Linux installation receiving duplicate packets from the switch). You could also tcpdump on/for the deployment server, to see if something hooky is going on there. ... View more