timewarp! 2014 -- same error, I just updated to Splunk 6.1.0 and geoip broke! Love that app, hope it's updated soon... ... View more

http://docs.splunk.com/Documentation/Splunk/5.0.4/AdvancedDev/3rdParty "Hi! This page does not exist, or has been removed from the Documentation." ... View more

Post dates on this are 2010 -- anyone know if they ever came up with that tool? (to reclaim space...) I have 4.4 billion events -- an export/clean/import would be way ugly... 😉 ... View more

Does the target need to be an existing extraction? For example: LOOKUP-rdns = dnsLookup ip AS Client_Address Does "Client_Address" have to be an existing field that's extracted (and presumably populated with IP addresses) -- or is this a new field that is created? (in either case, still not getting this to work...) /opt/splunk/etc/system/local/transforms.conf [dnsLookup] external_cmd = external_lookup.py host ip fields_list = host, ip /opt/splunk/etc/system/local/props.conf [access_log-too_small] LOOKUP-rdns = dnsLookup ip AS Client_Address Also tried "source_ip" (v.s. Client_Address) because it's an existing extraction that's populated with IP addresses, yes appearing as the sourcetype "access_log-too_small". ... View more

Ha! I just heard from a friend of mine who happens to be Splunk engineer (ya, I try not bother him all the time with questions like this... ;-). Turns out, this is all backwards. The suggested way is to build a new app, then construct the dashboard within the app -- not the other way around. Of course, it doesn't help the fact that I already have a dashboard... ... View more

Oh, AND there's a place to set the default app to the role, also -- yes that's set to the new "default app". So, I have a user default app set to the new one, the user has a new role assigned to it, and that default app is set to the new one -- and when logging in, we go to the search screen... ... View more

well, not really. Missed some steps...after creating a role and assigning the new user to it, ensure the permissions of the dashboard so that the new role can view it. ...and I'm still getting "No results found..." in some of the windows in the dashboard (some work). And when logging on, it takes me to the search screen where I have to select the correct dashboard -- yes I set the user's default app to the proper app (and not to "search"). ... View more

Wow, here's a lesson learned for me during my hardware upgrade/migration process: (BTW, no one consciously duplicates bucket IDs, so telling us not to do it is moot -- telling us to look for them is another story...). I shut down my Splunk process (moving hot to warm). Did a giant RSYNC (SCP) to the new hardware. For unrelated issues, I had to fall back to the "old" box. Restarted Splunk on the old box, a couple days pass... When I was ready to try again, I shut the process down, Did an RSYNC to the new hardware (hoping to save time in the copy process). Bam! Duplicate bucket IDs on the new machine. But not just in the last couple of them, but at random locations. For example, even though I was in the _700 range -- I had duplicates in the areas of _320, _22, _699, etc.. Go figure... Didn't make much sense, I had about 20 duplicate bucket IDs in total. So, just to be safe, I blew it away on the new machine and did a new SCP... Lesson learned: don't use RSYNC -- just do one giant copy and be done. ... View more

Solved it by the old Windows trick: uninstalling and re-installing. Corrupt something, somewhere? I opened a case, and after a condescending reply from their tech support that told me I was digesting .gz files and such, they pointed me to the on-line documentation on how to edit the inputs.conf file. Admittedly, I did have it miss-configured initially, but corrected it days ago. They overlooked the fact that I disabled all inputs during testing, and could enable/disable the /var/log on the local machine to duplicate the problem each time -- standard stuff in the /var/log -- no .gz files. Also confirmed the inputs.conf file (and sent them a copy) only had this and other /var/log sources in it (again, all disabled during testing). Go figure...anywho, fixed now. ... View more

It was just updated and rebooted this morning... (that was a good answer though!) ... View more

sample of the area I'm talking about: " Subject: Security ID: S-1-5-18 Account Name: SERVERNAME$ Account Domain: PORT Logon ID: 0x3e7 Logon Type: 10 Account For Which Logon Failed: Security ID: S-1-0-0 Account Name: bubba Account Domain: PORT " I just want to know about Bubba's failed logon, not "SERVERNAME$" or "-". This can't be the first time this has been asked... ... View more

Those results filter it down alright, but it finds the instances where the "-" is replaced by the name of the server -- that's just as bad as having a boat full of "-" as results -- it doesn't boil it down the account names that failed a logon. So I guess an extension of my original quest is to return the Account_Names -- without the "-" or the name of the server (why is the servername even listed as an Account Name anyways, Redmond? geez!). ... View more

Thank you! Gosh that would have been nice if it came up when I did a search on "invalid format string" in the first place. ... View more

Will check. Wouldn't be surprised, you can do the same thing on 10 different Microsoft installs and get 10 different results... 😉 thanks. ... View more

Nope, didn't do that...stand-by... Hey it worked! Thanks! OK, now: WTF? Shouldn't have to do this work-around Splunk engineers... ... View more

To add: This happened to me when I accidentally enabled the "splunklooger" index (thinking it might provide some useful info). Using 4.2.4 on Linux, this did not happen in conjunction with an upgrade. Cratered it, couldn't restart. "'homePath' tag required in config..." Edited the file Splunker_J mentioned (thanks!) -- he was right on. (../system/local/indexes.conf) Look for: [splunklogger] disabled = 0 and change to: [splunklogger] disabled = 1 Restarted Splunk. ... View more

Bug in "splunk enable boot-start"? I seem to experience this same thing when I restart my Linux (Redhat) systems. It appears that it's not shutting down properly -- and when it attempts to start back up, it sees an improperly closed database -- then, while it appears that the process is running, it's not until you issue a manual "splunk start" you'll see the "unclean shutdown errorr" and be prompted to fix it. The "fix" noted above merely shows you how to do this ("./splunk start --answer-yes ") -- and gets you going, but doesn't fix the underlying issue. You can easily re-create this by simply killing the process (without a "splunk stop" then doing a "splunk start" -- this is essentially what's happening when you reboot. In looking at the rc.d files, there's startup commands issued (/etc/rc.d/rc3.d/S90splunk) for Splunk, but no shutdown (i.e., missing: /etc/rc.d/rc3.d/K90splunk). In fact, I find a start file in rc3.d, rc4.d, and rc5.d -- but no shutdown ones. You have to create your own "K file" to shut it down properly in the first place (/opt/splunk/bin/splunk stop). I would think that a database system with it's own mechanism for creating startups, would also provide for a clean shutdown in init.d. I'm chalking this up to a faulty "splunk enable boot-start". ... View more

Bug in "splunk enable boot-start"? I seem to experience this same thing when I restart my Linux (Redhat) systems. It appears that it's not shutting down properly -- and when it attempts to start back up, it sees an improperly closed database -- then, while it appears that the process is running, it's not until you issue a manual "splunk start" you'll see the "unclean shutdown errorr" and be prompted to fix it. The "fix" noted above merely shows you how to do this ("./splunk start --answer-yes ") -- and gets you going, but doesn't fix the underlying issue. You can easily re-create this by simply killing the process (without a "splunk stop" then doing a "splunk start" -- this is essentially what's happening when you reboot. In looking at the rc.d files, there's startup commands issued (/etc/rc.d/rc3.d/S90splunk) for Splunk, but no shutdown (i.e., missing: /etc/rc.d/rc3.d/K90splunk). In fact, I find a start file in rc3.d, rc4.d, and rc5.d -- but no shutdown ones. You have to create your own K file to shut it down (/opt/splunk/bin/splunk stop). I would think that a database system with it's own mechanism for creating startups, would also provide for a clean shutdown in init.d. I'm chalking this up to a faulty "splunk enable boot-start" -- I'm calling it a bug. ... View more

Thanks Genti for the idea. Here's what we ended up doing. Using a separate box (not our main indexer) we installed the same version of Splunk. Splunk stop. Copied the hot_v1_1, etc. directories into ../splunk/var/lib/splunk/defaultdb/db. Then I manually edited .metaManifest to make sure there was a line for each instance of hot directory, i.e.: /opt/splunk/var/lib/splunk/defaultdb/db/hot_v1_1 /opt/splunk/var/lib/splunk/defaultdb/db/hot_v1_2 etc. Then edited .bucketManifest to include a line for each hot directory (these apparently get transformed by Splunk once you start it back up): i.e.: 1 : hot_v1_1 2 : hot_v1_2 Splunk start. After their conversion, they looked something like: 1 : db_1296702713_1295776921_1 2 : db_1296397258_1296021601_2 Worked like a champ! ... View more