Thanks. Good to know that I’m not veering off in the wrong direction. I am a bit unclear about the _time stuff though. You’re saying i should really write out ... | table _time, user Then? For some reason i was thinking that the time range that the search ran in would be captured in the summary index on its own and i wouldnt have to explicitly write it out. I am also unclear about the first and last time thing you suggested. How would i use that? Mark ... View more

I ran this and got 583 results again. I looked at the individual search components until I realized what the issue was. Some hosts may report themselves to Splunk with a name that's in all caps rather than what I've got in the lookup. So with just a little tweak to the above: | tstats count WHERE index=_internal sourcetype=splunkd BY host | search [| inputlookup my_hosts_lookup | fields host | format ] | eval host=lower(host) | eval found_in_logs="true" | append [| inputlookup my_hosts_lookup] | stats values(found_in_logs) AS found_in_logs BY host | fillnull value="false" found_in_logs I get 375 results returned. Thanks very much for you help! ... View more

Thank for your reply. As I noted in the comment above, I was looking to get only the status of the hosts in my lookup table which his only 375. This search returns 1190. We currently get results from 815 UF's. So... "bonus hosts" 🙂 As above, this approach is also education for me for future SPL's. ... View more

Thanks very much for your reply. My actual list of servers of my lookup is actually only 375. This search returns 583 results. I was really just looking for the status of those 375, not a larger portion of my overall hosts. 583 seems to be 208 (number of hosts that are reporting in) + 375 (total number of hosts from the lookup table). But, I will say this search is education for me and helpful to understand new approaches with SPL. ... View more

The major issue here (size of emailed scheduled search jobs) is resolved due to your limits.conf suggestions. I consider my question answered. But as I noted in another comment, I would like to figure out what needs to be fixed to make the 'sendemail' command work from within the search bar as well. That's on me, I think. My concern is that the descriptions of items in limits.conf don't lead to an obvious solution here. There are many keys with 'max' in them, but none of them were obviously about this issue and at this point, I don't quite know which would apply to the sendemail command. I need a decoder ring. Thanks ... View more

I figured it was something along these lines. However, I tried these settings (bumped them up enough to handle 250,000 results) and I still get the same message -- at least when I don't schedule it and pipe it through 'sendemail'. Perhaps that involves some other magical settings in limits.conf. I believe this job will run on its own tonight so we'll see what happens via the scheduler tonight. Thanks ... View more

This is interesting. For the sendemail commands I'd run to this point, I never got any error output. But using 'sendemail' and telling it to not use csv, I got command="sendemail", (552, 'size limit exceeded', u'me@mydomain.com') while sending mail to: me@mydomain.com which I'd never seen before. I saw another Splunk Answers post indicating to check the mail system (postfix in this case) configuration for maximum message size. I doubled that but it had no effect. Doesn't seem like that would be applicable here though. From some other digging into that error message, at least the 552 part of it, it does seem like something needs to be changed in limits.conf, although the docs don't make it at all clear what. ... View more

Understood. I tried this and got essentially the same result. Only 50K results inline and a message at the top of the email that said that the results were truncated. That is, Splunk clearly knows there's more events than it's showing but is only display 50K of them for some reason. It's not out of the realm of possibility that I'm just doing something wrong here that I'm missing or have a typo or something, but I'm not seeing it. Thanks ... View more

Thanks. I'd tried that previously as well. I thought I'd mentioned that above, but I just tried again [ search ] | sendemail to="me@domain.com" from="me@domain.com" inline=false maxinputs=1000000 sendcsv=true server=mysmtpserver subject="My Test" sendresults=true format=csv As usual, I received the email with an attached CSV file. Still 50,000 results. The body of the email says: Search results attached. Attached csv results have been truncated Dangit. ... View more

Yeah, I see that now that you mention it. But as I said, I had tried this without that line (i.e. maxresults = 500000) before that and it didn't matter. And ultimately, I believe that what's in the saved search definition should override all that. ... View more

Are you looking to prevent those asterisk lines from being indexed at all? If so, have a look at https://answers.splunk.com/answers/232641/is-there-a-way-to-ignore-certain-events.html ... View more

I opened a case with Splunk on this. Despite the same dire warning about THP, it's probably not quite as essential on a universal forwarder. One would need to look at the specifics of THP and the generalizations about how it impacts applications. It seems that Splunk could benefit from it when you've got a lot of open files. Perhaps less essential if you've only got a few. It's not likely to be impactful to a non-Splunk app, unless it's doing operations on large files on a system with very little memory. ... View more

Right, but then why the warning in the universal forwarder logs? ... View more

While I'd secured the data port and splunkweb with SSL, I had done nothing to change the default not-so-secure key used for the management port and anywhere else it was used (replication, kvstore, etc). Somehow I missed that part of it. We have a rather extensive environment and I'll need to move carefully to fix this everywhere. Would have been much easier had I spotted it earlier :-. Have the issues Duane references in his PDF presentation regarding using SSL for clustered indexer replication gone away now? I see the doc is several years old now. We currently do not use SSL with the deployment server. I've gone back and forth as to why this is even necessary as compared to the other ports. I suppose if I'm passing an unencrypted password within a deployment that could represent an issue. This will mean I have to pass a certificate to each initial install of a UF, I guess. It's possible that this is difficult enough to do that "that ship has sailed", but I'll investigate further. Anyway, I've now got a server that has an encrypted private key and the "missing" ports are now using it without issue. (It still really sucks that you can't use the same password-protected private key that you use in other configuration with Splunkweb so you have to generate a password-less version of it just for splunkweb). Thanks very much, jkat54 and mmodestino! ... View more

I see this on all hosts everywhere when using Splunk, not just on indexers. Per my comment with mmodestino above, I'm focusin on a simple heavy forwarder that really isn't doing anything right now except forwarding its own internal events to some clustered indexers. Referencing Splunk network ports 8000 - shows our certificate 8088 - shows Splunk's certificate (index replication port, but not using it on this server) 8089 - shows Splunk's certificate (as I suspected) 9997 - not listening As it turns out, I did use Duane's doc to make a final pass at the setup (I'd already figured out and setup most of it, but used it to fill in some cracks). After doing a quick re-read, I can see there's some stuff in there that I did not do as I didn't think they were applicable in our network. I see a references to securing the REST interface and indexer replication port. However, unlike the other port types, I don't quite see that mechanism spelled out. I'm guessing that's slide 26? It looks like I need to do some experimentation... Thanks ... View more

OK, so let's take a heavy forwarder with no event activity passing through it, that I was playing with yesterday. That is, no other host has this server in its outputs.conf anywhere. It does have the best-practices outputs.conf mechanism in place to forward its own logs to an indexer, but that would be the only event traffic involving this server. This server does have its own internal-CA-generated certificate as is common here. Note that in the output below, names and encrypted passwords have been changed to protect the innocent. "hostA" refers to the local server's physical hostname, however we use CNAMEs for our servers and in this case, this server is "fwd01" below. Our certificates have SANs that allow both names plus the IP of the server as valid names. Hence, physical hostname, alias or IP would all work fine through Splunkweb. inputs.conf [default] host = hostA [SSL] sslPassword = zzz requireClientCert = false serverCert = $SPLUNK_HOME/etc/auth/mycerts/fwd01.pem outputs.conf (the local event forwarding is in another file...) [tcpout] clientCert = $SPLUNK_HOME/etc/auth/mycerts/fwd01.pem defaultGroup = prod_default sslRootCAPath = $SPLUNK_HOME/etc/auth/mycerts/cacerts.pem sslVerifyServerCert = true useClientSSLCompression = true server.conf [general] serverName = fwd01 pass4SymmKey = xxx [sslConfig] sslPassword = yyy sslRootCAPath = $SPLUNK_HOME/etc/auth/mycerts/cacerts.pem web.conf [settings] enableSplunkWebSSL = true privKeyPath = $SPLUNK_HOME/etc/auth/mycerts/fwd01.privatekey.pem serverCert = $SPLUNK_HOME/etc/auth/mycerts/fwd01.pem lsof -p -P | grep IPv4 reports the following splunkd 23350 splunk 5u IPv4 481132186 0t0 TCP *:8089 (LISTEN) splunkd 23350 splunk 31u IPv4 481135551 0t0 TCP *:8088 (LISTEN) splunkd 23350 splunk 79u IPv4 481159127 0t0 TCP hostA:35646->idx2:9997 (ESTABLISHED) splunkd 23350 splunk 84u IPv4 481133325 0t0 TCP *:8000 (LISTEN) splunkd 23350 splunk 87u IPv4 481159129 0t0 TCP hostA:57190->idx1:9997 (ESTABLISHED) splunkd 23350 splunk 93u IPv4 481133327 0t0 TCP localhost:32931->localhost:8191 (ESTABLISHED) splunkd 23350 splunk 94u IPv4 481134009 0t0 TCP localhost:32932->localhost:8191 (ESTABLISHED) splunkd 23350 splunk 95u IPv4 481135617 0t0 TCP localhost:32933->localhost:8191 (ESTABLISHED) splunkd 23350 splunk 96u IPv4 481134011 0t0 TCP localhost:32934->localhost:8191 (ESTABLISHED) Just to see, I removed the deploymentclient.conf entry here so this heavy forwarder doesn't have a deployment server to talk to -- all local config -- and restarted. The X509 warning doesn't go away, so I doubt it's the deployment client/server. I assumed everyone got this warning in their logs... I can say for instance, that if I spin up a "virgin" Splunk Enterprise with no configuration at all, this warning shows up in the logs as well. If it's worth of displaying a warning but it isn't a problem, I'm not sure why it would show up in the logs? It's like saying there's an "OK error". Thanks! ... View more

Why didn't I think of that? 🙂 Just prior I see 03-29-2017 14:06:11.948 -0500 WARN HttpListener - Socket error from 1.2.3.4 while idling: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request 03-29-2017 14:06:13.153 -0500 WARN HttpListener - Socket error from 1.2.3.5 while idling: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request 3-29-2017 14:06:13.543 -0500 ERROR ScopedLDAPConnection - Invalid search filter: Attribute and value must be non-empty. Attempted to constrain attribute="samaccountname" to value="" 03-29-2017 14:06:13.612 -0500 ERROR ScopedLDAPConnection - Invalid search filter: Attribute and value must be non-empty. Attempted to constrain attribute="samaccountname" to value="" 03-29-2017 14:06:13.612 -0500 ERROR UserManagerPro - Failed to get LDAP user="" from any configured servers I think we have some internal tool that is scanning servers and attempting to break in on any ports it can find. Just to see, I tried going to the web interface and hitting enter without entering a userid or password to see if that would generate this and it did not. I guess there's something about the way this is hitting the port that is triggering an LDAP search with no user information. ... View more

I'm embarrassed to admit that I hadn't considered blacklists to solve this. This was still quite a pain to resolve even with the blacklist but it did help a lot. (Why can't Splunk just use regular expressions in the monitor line now?) I had a problem with constructing a monitor line that would properly match files in any presell* directory -- problems I don't think I should have had but I finally landed on the following which seems to work. [monitor:///var/weblogs/presell*/*(access|error).log] crcSalt = <SOURCE> index = presell [monitor:///home/splunk/weblogs/] crcSalt = <SOURCE> whitelist = /(access|error)\.log$ blacklist = /presell index = apache The docs seem to say that [monitor:///var/weblogs/presell*/(access|error)*.log] would work to trigger Splunk to recognize both its own wildcard and then a PCRE in the final segment, but that never seemed to work, but [monitor:///var/weblogs/presell*/*(access|error).log] does. The '*' in the final segment is completely superfluous other than to tell Splunk to honor the PCRE. Thanks! ... View more