Depending upon your goals and level of desired integration with splunk, you might want to consider talking to splunk directly over HTTP at some point via REST at some point. A lot more work, a lot more power. ... View more

Some day, I should sit down and actually read TCP/IP Illustrated. I'm only familiar with 'connection reset by peer' by bumping into it. Apparently, it means the other end of the socket is not aware of the connection. This could mean something where the other side shut down the link abruptly and the normal shutdown didn't happen for some reason, and thus your side finds out about it late, when still trying to use the connection. But more likely some network equipment or firewall is deciding it no longer wishes to remember or keep up this socket. You should defintely begin by looking at how often you see these messages for a given ip, for a given indexer. What are the timing patterns? It might be useful to see what the forwarder side reports at the same time window. Probably this becomes more of a tech support issue, but it is possible that it's really just a network configuration issue in your environment which is hard for the application to analyze. I think pursuing both is the right strategy ... View more

TRANSFORMS-foo=a,b,c,d says to use all four of those. Then TRANSFORMS-bar=e says to use that one too. ... View more

Some characters are smashed to underscore at search time. It might be that they are other characters in the FIELD list in transforms.conf. Use Manager or splunk cmd btool transforms list to find them. ... View more

That's the documentation for inputs.conf. Props.conf syntax is source::the_source_string where your source string is udp:514. Maybe we should explicitly document how the source string is formatted for these events, but I prefer simply checking search results. ... View more

Okay, but splunk isn't applying the regex to the whole file, it's applying it to each event. In fact, from splunk's perspective there isn't a file at all, because the data is arriving as udp packets. ... View more

Oh, Ayn's solution is superior if it is really precisely as you describe. ... View more

Not-Exist OR blank is a bit tricky. Try (ERROR="" OR NOT ERROR="*") ... View more

UDP syslog events aren't split by newlines, but by packets, so there's no newline even on the wire. In splunk, we cut on newlines in the LINE_BREAKER, so they're typically still not available except for multiline events. Use the ^ character for the beginning of your event. ... View more

It's hard to understand how that search could be wrong. My guess is you're getting alerts for license violations on hosts you don't care about, like forwarders. Try running the search manually to see what comes up, say over the past several weeks? ... View more

Are logins and accesses the same? Are we starting with the set of logins, and wanting to find, for each access point, the top 10% of users and their count of logins? ... View more

Just tried to hack up the searchstring to be a bit more readable in answers ... View more

At least one customer has gone the route of constructing conf files eg inputs at splunk-start time. I'm not sure if they wanted includes... I think not, but obviously the goal of doing things your own way exists. Generally we created the bundle system (apps) with the idea of encapsulating logical sets of configuration. The idea of them having a life in the UI, and permissions came much later. You can make apps not have any life in the UI. ... View more

I suspect your use of limit=100000 is defeating the where clause. ... View more

Does your event-stream have the _time field available prior to invocation of the transaction command? It might be most convenient to check with ... | eval test_time=_time where ... is your full search string up to the point of transaction. At this point you can use the field inspection tools on test_time. One possibility is that all these events got timestamped with the current time, (DATE_CONFIG=current) and were loaded in as historical data.. or something along those lines. It seems a bit unlikely given the recent nature of these events, but not sure what's going on so far. ... View more

Essentially yes, it's saying that you got a big message. Since a 68MB data item is highly unlikely, there was probably some breakage in the datastream. The protocol for splunk->splunk forwarding includes a length indicator number, which causes the receiving code to allocate memory. To avoid breaking the receiving Splunk, it does not blindly trust the size, but for cases of very large length numbers logs the problem and does not allocate the memory. This could be a case where the forwarder is encountering some kind of memory corruption bug, where something is communicating to a splunktcp:// socket which is not quite conformant (hard to imagine, but possible), or when the stream of bytes in the tcp socket is getting messed up via some other means. We had a known problem with early versions of 4.0.x and late versions of 3.4.x where the forwarder would sometimes inject 'heartbeat' pseudo-messages in the middle of other messages, corrupting the datastream. You may want to evaluate if tchuxxx.xxxx.com may be running an older version of splunk. ... View more