indeed - I might add (just to document it) that to have multiple metadata tags in your forwarder, just do something like the following space-delimited _meta=key1::value1 key2::value2 key3::value3 ... View more

No, ignore the step 5 stuff, that's not relevant. I'm asking about unix permissions on the filesystem only. More detail - I'm trying to permit shared group-write for development where a developer could ssh into the search-head and build their app via some combination of manual editing and using the gui to build dashboards etc. If they stay all gui for their development, the app works, but the group and world permissions are getting reset by the gui 'save' actions to something other than the state the tree started with (in general, the group-write gets removed, sometimes even group-read). If they work all interactively in an editor, the app doesn't always work due to needing service resets all the time if they edit any .conf files or the like. So I'm trying to dig into what the 'unix' filesystem permissions are supposed to be in $SPLUNK_HOME/etc/apps/myappname Also, the permissions there differ a lot for unix and mac variants of splunk, which I can't explain either. Is there a way to reverse engineer what the splunk gui is setting when you hit 'save' ? ... View more

A screenshot would help us know to bother (or not)..... ... View more

We deploy apps via puppet, pulling each from its own Bitbucket repo, if that's what you're asking. One thing to be aware of is permissions as well as the need to reset the splunk process if you change certain files in the app (app.conf, savedsearches.conf, etc.) ... View more

Rainier - that format works in inputs.conf on a universalforwarder for me, just as you wrote it....assuming the tag1 and tag2 fields are known on the server side of course. ... View more

absolutely - me too...... I've been trying to find a way to add multiple custom metadata for months. The best I can come up with is one long ugly value that I can tear apart at search time: _meta = myfields::tag1=value1,tag2=value2,tag3=value3 I'd much better be able to add greater-than-one custom fields at index time.... ... View more

yes, on centos it's root:root - I know 'how' to hack the system up, I just don't want to do it. Still awaiting an official Splunk chime-in on what they want us to do. Their silence on these kinds of things is a bit disconcerting. ... View more

Well, this is centos, so it's not adm, and many interesting files are mode 600 so group permissions wouldn't help. I'm reluctant to hack up the system to force things. Good idea though... ... View more

Sounds pretty counter-intuitive to me, but thanks for the response. Could somebody from Splunk Engineering chime in with their recommendation please ? I keep finding these edge cases where simple stuff is difficult/impossible/too-complicated in this product. Other than "don't do that" it would be nice to have a "do this instead" from Splunk Engineering 'officially' as a best practice. You know we will want to monitor our splunk server logs just like any other forwarding computer. What is the way we are supposed to do this while meeting your recommendation to not run the server processes as root ? ... View more

On a similar note - I know that Splunk recommends not running a 'server' (not forwarder) as root, but then it can't monitor itself for the same reason. I set up a new server to run as user splunk group splunk and couldn't monitor /var/log/messages due to permission denied. Is there any reasonable way to let the non-privileged splunk processes monitor non-splunk-owned files ? Do I need to do something crazy like forwarding syslog to the indexer rather than using splunk ? Seems dumb when you get to that level of complexity when I can packet filter things and block input to the whole server. Ideas ? Best practices that actually work ? ... View more

Not seeing the relevance. The first link doesn't resolve, and the second one is search-engine-side. The indexer and search engine do not know what categorization we want to apply to the client computers, but the client forwarder systems 'do'. We set custom facts in puppet deploying the forwarder configs that we can use to generate the inputs.conf (etc.) on the forwarder from ERB templates. Again - I am trying to do 'client' side categorization of the data at forwarding time. Is this somehow not possible in splunk ? ... View more

Do you have an example of how to use a tag to do this on the forwarder from the shell only ? I can't find any reference of what to edit where on the universal forwarder client hosts (we deploy with puppet, so no gui there) and what if anything (hoping nothing) we need to edit (and where) on the indexer or search head boxes..... ... View more

works great on 6.1.2 but given the date of the reply I'm wondering if it's deprecated now. Great having one way to do it so thanks ! ... View more

I think you overreacted if you think that was rude. If you want the site useful, helping the original poster know how to report a bug would have added value. ... View more

Ummm, that doesn't help at all. It references a maze of circular references to using Splunk Web. I'm trying to use it all CLI from text-mode-only splunk forwarders. As I mentioned in my original post, I'm looking to CLI working examples or references, not references to the (horrible) web-centric maze of splunk docs. ... View more

Thanks for the detailed answer which is great for how/why to use indexes, but I was specifically asking about tags. You mentioned 'Another way is to add tags for all your servers so that you can categorize servers into different category. *' which is my actual question - *how do I implement tags ? What files do I edit where ? Could you perhaps add a little to your example above with adding a tag to that monitor to complete the picture ? Thanks again for the help... ... View more

really ? shouldn't you have answered with "thanks for the bug report and solution - please go to to report this so it gets fixed in a future version of splunk" ? ... View more

Topology is what looks to me like a proof-of-concept server that we inherited from somebody who left before I got here - one splunk VM on AWS doing all the lifting, a handful of Linux VM universal forwarders monitoring the usual syslogs and a few app-specific logfiles. Very small license, just a few GB/day. No custom sourcetypes. Everything getting tossed into main index. Couple reports/alerts/dashboards. Basically what to me looks like a throwaway 'lets fiddle for a week' demo setup. Nobody here has any splunk experience at all, so I'm trying to corral this thing and go as small/simple/slowly as possible to try to figure out some reasonable best-practices to spin up a real environment as well as provide some guidance re: how to get some value from splunk here. I 'did' figure out yesterday that I needed touch labor on the server to create the custom index, and saw the forwarded data show up ok afterward, which was good. Did a trivial app on the forwarder to generate the date ala cron, so learned a lot (albeit too slowly/frustratingly). Next up I guess is creating the custom sourcetype (back part of (1) in your original response) and extracting the data into named fields (2) then trying your reports. Tough going since the splunk docs assume you know far more about their product than somebody getting their feet wet really knows. I'll keep banging at it...appreciate the help, so thanks ! ... View more

We have what I'd call a proof-of-concept server basically inherited from somebody who left. One splunk server doing all the lifting. A few dozen client systems (Linux now, will add Windows) running universal forwarders for basic logfile kinds of things (syslogs, app-specific logs, etc.). License is only for a few GB/day so we'll be very very small for the forseeable future. What I inherited just tosses stuff in main. No custom sourcetypes. Really really quickie setup in my opinion other than maybe a dashboard with stuff nobody knows anything about. Basically nobody here knows a thing really about how to get much value from splunk or how architecturally to best set it up.....so I'm trying to go very small/minimal/simple/slowly to try to figure this beast out. I 'did' figure out I needed to touch the server to create the custom index, and data got there from the trivial app I built for one forwarder. Next need to figure out how to do the custom sourcetype and extract data into the fields per your original reply items (1) and (2) and then try your example reports. Nibbling away at it. Definitely appreciate your help and time...... ... View more

ok - got a scripted input working (cool) with a cron-like period specified. Where I'm lost now is how do I create a custom index from the 'forwarder' side only without needing to touch the splunk server it forwards to. Basically if I define a index=whatever line in inputs.conf, it seems to run but the custom index doesn't get automagically created on the splunk server, and there are no errors in the forwarder VM at all saying anything at all. geez this stuff is dense and the stackoverflow-like answers interface is horribly inefficient to navigate through. I 'think' that I'm trying to do something very basic to the product, but there are no gentle getting-started howtos that i can find to even know what obscure terminology to use to ask the question. Very frustrating product to try to spin up on. Anyway, appreciate any help of course ! ... View more