right, but when you add it...does it add | spath ? or does it add it as a kv pair? when I add to search, my search becomes: index=n00blab sourcetype="answers_json" host="n00b-noah-01"| spath "mdc.mdcKeybIwMa" | search "mdc.mdcKeybIwMa"=mdcValueNYUGJgYJaTFaWcdicara ... View more

yeah, at least in my example, the mdc is being extracted by the syntax highlighting... If I search verbose mode, it gets the mdc...and if I click on that value and add to search , it actually pipes it thru spath. So you were right, my previous examples were search time...but they weren't KV_MODE as I had that off.... It looks like the gui does auto extract of JSON with SPATH in the search results. I tried turning syntax highlighting off...will try and find out more about the voodoo at play here... When you click the mdc and 'add to search' is it piping thru spath for you? ... View more

so on your working example, if you search fast mode, do you see the json fields extracted? ... View more

So on the hunch that the underscores were your hurdle, I took your file and removed the underscores, then set up a file monitor on a windows 10 UF running 6.5.2. {"version":"1.1","host":"t800.skynet.com","short_message":"Msg TzjTJPqvUaGqaOpHXFdKXyXVaHiLpbTKfhePqbEtammLeaZVaaTb \r\n","full_message":"Msg TzjTJPqvUaGqaOpHXFdKXyXVaHiLpbTKfhePqbEtammLeaZVaaTb \r\n","timestamp":1484920098.408,"level":4,"app":"skynet","level_name":"WARN","mdcKeybIwMa":"mdcValueNYUGJgYJaTFaWcdicara","thread_name":"sample","logger_name":"common.log.json.LogFileProducer","env":"ut"} {"timestamp":"2017-01-20 14:48:18.428","level":"DEBUG","thread":"sample","logger":"common.log.json.LogFileProducer","msg":"Msg TzjTJPqvUaGqaOpHXFdKXyXVaHiLpbTKfhePqbEtammLeaZVaaTb","mdc":{"mdcKeybIwMa":"mdcValueNYUGJgYJaTFaWcdicara"}} inputs on the UF was : [your input] disabled=false index=n00blab sourcetype=_json And now all fields are parsed.....So one option here is to pre-parse or re-configure logging source to remove the leading underscores to allow index time field creation.. Obviously easy for me to say with 2 events... I also got it to work by setting both indexed_extractions= json AND KV_MODE=json. Bit of a hacky workaround, but it works... [salem34json] CHARSET=UTF-8 INDEXED_EXTRACTIONS=json KV_MODE=json NO_BINARY_CHECK=true SHOULD_LINEMERGE=true TIMESTAMP_FIELDS=timestamp category=Custom description=indexed_extracted json for eventbreak and timestamp, kv_mode json for kvpairs to workaround underscores disabled=false pulldown_type=true I decided to keep the indexed_extractions=json to enable us to declare the timestamp field for timestamp extraction, which you need here because you have multiple time formats.... then a KV_MODE=json props on your SH to pull your key value pairs to workaround the leading underscores that are messing with those fields at index time... ... View more

Field name syntax restrictions You can assign field names as follows: Valid characters for field names are a-z, A-Z, 0-9, or _ . Field names cannot begin with 0-9 or _ . Splunk reserves leading underscores for its internal variables. Avoid assigning field names that match any of the default field names. Do not assign field names that contain international characters. http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Configureindex-timefieldextraction ... View more

You're right, the indexed_extractions should be on the UF if you are going to use indextime, but if the props is not working that is not the only option for json extraction. (see Spath or KV_MODE=json) Anyhow, like you said, it should work, and I have a win10 box so ill try it shortly. As for the fields you expect, I think it may be due to the fields starting with an underscore...dont think that will fly...will get it working and share the results. Don't let it get to ya, we'll get ya sorted...but u'll likely need searchtime ... View more

your issue sounds more like limits.. ... View more

Hmm, to start, maybe try removing the props from the UF and simply keep the inputs that explicitly sets your json sourcetype? That should be similar to the test you did, in that you let the indexer do the extractions. ...and do it at searchtime? If that gets you rocking, then we can chase of whether this is a windows uf thing, or just a uf thing... What version of windows are you dealing with? What is the input on the UF? tailing a file? EDIT: AH "inputs.conf which simply points to the appropriate directory using monitor and applying the sourcetype "my_json_sourcetype". Did you create your sourcetype by hand? Try running the data through the add data wizard and building on top of the stock _json sourcetype on you idx or using the app builder. Then apply that props to the UF? Like you said, my 6.5.2 Linux Indexer using upload via browser from a Mac running Sierra (with is batch proc IIRC.), ate this no problem. ... View more

Check out the pagerduty app on splunkbase! https://splunkbase.splunk.com/app/3013/ It's integrated with Splunk alert actions which will simplify your solution and pagerduty offers a free trial and low monthly cost. I used it for production support and it worked really well. ... View more

np! yeah, I was surprised to see the loss of the NAS mount point cause splunk to simply write to the same location locally. I am not confident that will be true in all cases, thus the encouragement to test, but I can definitely advise that is one experience I have had with losing NAS while splunk is still running. I guess the main idea is that it is hard to know what to expect in that scenario. I have totally seen indexers die when losing attached storage, so it wouldn't surprise me to see it fall over....anyway you can test in the lab? would love to know what you end up experiencing. ... View more

Hi koshyk, I have worked on environments with this same situation and we simply coordinated with the storage teams, and relied on the redundancy of multisite to allow us to completely shutdown the indexers during the work. Our UF were configured to send data to both sites to ensure that we could take down an indexing site when need be. There is no command to stop rolling of buckets to cold. In my experience - and this may depend on your environment/configuration - Splunk simply began writing locally. I don't think this is a Splunk designed behavior, was probably provided by the underlying os ! Your results may vary! PLEASE TEST! I would think it is completely within the realm of possibility that you could lose the indexer completely. Do you have a multi-site cluster? Do you have a lab environment where you can run this scenario to understand your configuration and what you can expect in your environment? Have you ever lost your NAS in your environment? My personal opinion is that your environment should be built to withstand failure of your storage, but also to accommodate the inevitable outages that maintenance work brings. If this is a production environment, these failure scenarios should be tested! Hope that helps and that you have a smooth storage upgrade! ... View more

Ah! I see what you mean. What version of Splunk are you running? Also, try removing the wildcard from this search and hitting the endpoint for that indexer only. | rest splunk_server=Idx10 /services/server/status/partitions-space Is this the only Idx with this situation? ... View more

Don't be sorry! It's a great and necessary learning experience! the post will likely help people in the future! Glad to see you got it sorted! ... View more

Hi rsathish47! The DMC alerts are configurable and can be set by navigating to the Platform Alerts Setup page. You can then select the DMC Alert - Near Critical Disk Usage and configure the desired threshold. The default threshold is 80%, which based on the output of that rest call, you have definitely breached. Is the output you shared accurate when comparing to check from the indexer's cli? ... View more

Hi hadiyan! Could you provide more detail about what you are trying to accomplish, what setting you tried to change and where that "error" message was seen? ... View more

Why were u stopping splunk?? Any chance you were trying to enable ssl or install new certs?? ... View more

check out the overlay option in the chart settings. https://docs.splunk.com/Documentation/Splunk/6.5.1/Viz/Chartcontrols It allows you to choose a field returned by your search to be overlayed. Should do the trick for the searches you have shared ... View more

Hey Juan, that is definitely it. That setting explicitly sets all your logs received on that input to access_combined regardless of what they actually are. I would double check whether your environment was set up so you would only receive weblogs on this port before changing it, but generally if you receive multiple log types on that port, you can remove the sourcetype and rely on your props.conf on the indexers to identify the sourcetype. I would recommend you download the ASA app from splunkbase and take a look at the props/transforms to see how you can dynamically sourcetype based on message https://splunkbase.splunk.com/app/1620/ ... View more

Hi Juan! I would start by checking the udp input in inputs.conf to ensure the sourcetype wasn't explicitly set. I'm not sure the access combined regex in props would ever mistake the the asa syslog... If the sourcetype is not set on the inputs, then move to reviewing your props to see if you can identify what is causing Splunk to categorize these messages. As for getting it back to cisco, do you have any of the Cisco TA's installed?? There is a TA for ASA that should help you properly identify these logs... ... View more

Hi Dave! I am not sure that the license gets added to Splunk at all...(Splunk licenses are xml) Splunkbase notes that the tripwire apps are "Hosted Externally" and points you to tripwire for the download. I would check for support or install instructions from Tripwire. Perhaps their customer portal? My google-fu has not returned any info on install steps beyond whats on splunkbase, and that makes no mention of license. All I could find was this EULA making pretty clear Tripwire is on the hook for anything to do with the app, and the instructions on the splunkbase page: http://www.tripwire.com/legal/splunk-app-eula/ https://splunkbase.splunk.com/app/3052/ I see that Splunk Answers user @JimWachhaus developed one of the tripwire apps...maybe he can help?? Hopefully tripwire can help you out! Be sure to post the answer when you find it! Good Luck! ... View more

What is the ulimit setting for the user running splunk on this server?? Usually ulimits crashes will cause a crash file to be present, which I believe you said there are none, but it is worth a look. ulimit -a Also be sure to check splunkd.log for any errors or warns. ... View more

The alert should be using the timeframe you set when you created the alert. Navigate to the alerts section of the app you are working in, or go to Settings > Searches, Reports, Alerts and open your alert for editing. Please share your settings here: As you can see my alert triggers every 5 minutes and looks back 5 minutes. Let's make sure you have vaild time selectors in the configuration of the alert. ... View more

Yep, like coltwanger said, if your HF is not set up for distributed search, you wont see the logs from there! Check from the search head, ideally. ... View more

Are you searching from your searchhead or from the heavyforwarder? can you see internal logs from the forwarder? index=_internal host=<yourUF> check metrics.log for your sourcetype. (sidenote: i don't think you need to be setting the sourcetype like that for a windows input, are you using any windows apps or TAs?) index=_internal host=<yourUF> source=*metrics.log check inputstatus on the forwarder using the splunk list inputstatus command (depends on uf version - 6.3 or 6.4+ i think?) https://docs.splunk.com/Documentation/Splunk/6.5.1/Admin/AbouttheCLI ... View more