The use case is laptops. People disconnect from the corporate network. UF keeps reading events but doesn't persist many of them. When user later shuts down their laptop, the events get lost. ... View more

What better mechanism is there to persist data when Splunk can't reach the indexers? Splunk continues to read log files. It is committing the data to a memory queue, but not a pqueue, because the memory hasn't filled up. So when the endpoint with the UF is shutdown, those events are just lost. It should either stop reading inputs when it can't reach the indexer or commit to disk as it halts. ... View more

Having same problem using splunk-python-sdk 1.6.5 against 7.0.2 Error : Traceback: Traceback (most recent call last): File "anonymizer2.py", line 99, in collection = myservice.kvstore['usercoll'] File "/opt/splunk/splunk-sdk-python/splunklib/client.py", line 1227, in getitem response = self.get(key) File "/opt/splunk/splunk-sdk-python/splunklib/client.py", line 1655, in get return super(Collection, self).get(name, owner, app, sharing, **query) File "/opt/splunk/splunk-sdk-python/splunklib/client.py", line 753, in get **query) File "/opt/splunk/splunk-sdk-python/splunklib/binding.py", line 289, in wrapper return request_fun(self, *args, **kwargs) File "/opt/splunk/splunk-sdk-python/splunklib/binding.py", line 71, in new_f val = f(*args, **kwargs) File "/opt/splunk/splunk-sdk-python/splunklib/binding.py", line 669, in get response = self.http.get(path, self._auth_headers, **query) File "/opt/splunk/splunk-sdk-python/splunklib/binding.py", line 1167, in get return self.request(url, { 'method': "GET", 'headers': headers }) File "/opt/splunk/splunk-sdk-python/splunklib/binding.py", line 1228, in request raise HTTPError(response) HTTPError: HTTP 400 Bad Request -- Must use user context of 'nobody' when interacting with collection configurations (used user='intuser') ... View more

Why so it is! You are everywhere! upvoted. ... View more

Thanks! That is what I arrived at. I found this example to include the button handler in the simple XML dashboard. https://answers.splunk.com/answers/581652/how-can-i-create-a-button-switcher.html ... View more

any way to do this in simple XML? ... View more

Thanks to both of you. Very helpful! ... View more

Update to my own post: I am able to get close by using kv_mode=multi in my props.conf. However I find that although this works for 99.8% of the hosts, there are some hosts that seem to not get parsed by multikv. I am able to duplicate this by removing the kv_mode line from props.conf and running the netstat output manually through multikv. It works for almost all the hosts but only works for 100% if I run it with "noheader=true" It's driving me nuts because the output of the netstat that parses and the netstat that doesn't parse looks exactly the same. I am wishing that I could get a hex dump of _raw so I could look for differences, but I don't know of a way to do so. ... View more

An example of how I would use this netstat data would be to look for a new LISTENING port open on a host..... so I might do something like (in rough psuedocode) search earliest=-2w latest=now netstat.data| stats first(event) by port,host | where first > -1d Thank you for your suggestion. That was my initial approach to the problem. Then when an admin installed the Splunk_TA_nix and splunk_app_for_nix, other events like ps started getting broken up, so I figured that I could reverse engineer what it was doing and apply it to netstat. No luck. The only concern with summary indexes is the additional space being used. ... View more

I got a similar error when I accidentally put a state=disabled in my etc/apps/SplunkUniversalForwarder/local/app.conf ... View more

I figured out what was going on with this in my case. My issue what that I was using the client package from the Cloudera Hadoop distro and that my /usr/lib/hadoop had the binaries and many libraries for hadoop, but it did not have the hadoop-hdfs jar file. Coudera has a separate directory, /usr/lib/hadoop-hdfs, which contained a jar for hdfs. It's klugey, but I copied hadoop-hdfs-2.6.0-cdh5.7.1.jar into /usr/lib/hadoop and then it worked. ... View more

It doesn't accept a "/" after the 8020. It errors with "invalid stanza". ... View more

More on this. Making the change above allowed me to see what was going on and get the rest of the event. What I saw was that it was dropping at "message=" because the tool that I was logging wasn't installed on my event collector. When I installed that tool, I ended up having to change the event format back to RenderedText (wecutil ss "subscription name" /cf:RenderedText" in order to get everything to parse correctly. Quite the adventure. ... View more

Thanks for the response. I should have been more clear. I my case, the log message is being truncated after 14 lines. All my log entries end with "message =" I am using tcp. I've turned on DEBUG for TailingProcessor, WatchedFile, TailReader, and the WinEventLog options on the forwarder and I am reviewing the splunkd.log but so far, I haven't seen anything useful. ... View more

In my case, I see all sorts of notifications regarding things being truncated in the _internal index and splunkd.log, but nothing talking about the ForwardedEvents file that I am concerned about. Btools only shows me the config from inputs.conf [WinEventLog://ForwardedEvents] current_only = 0 disabled = 0 evt_dc_name = evt_dns_name = index = FEvents The default section has nothing defined. ... View more

Having same problem ... View more

you can bin _time and then count the number of bins that something appears in: maybe something like eval foo = host + " : " + dest| bin _time span=1h |stats count foo AS concount by _time | stats value(concount) count by foo This should give you the numbers of connections per hour and the number of hours that they occurred. ... View more

So would thins indicate that _internal is a decent indicator of license usage for the data, while eventcount (plus a bit for metadata, etc. ) is a good indicator of storage use? Thanks for the answer! ... View more

Thanks! I am looking at Windows event logs. As a consumer of the information, I want as many event types. Mgmt wants cost vs benefit. I can't approach cost without knowing impact to license and storage. Even if I send some hosts to a server where I have admin, is there a good way to determine impact of certain eventtypes w/o sending them to their own index? A command to see size of a field (including _raw) would be useful. ... View more

Geez. Single quotes versus double. Feeling brain dead. Thanks! ... View more