Ok, if i understand you correct, you would like to dynamically extract Error messages from logs and assign it to a field? If yes, that is very much possible using eval You could you regular expressions to extract the Error string and assign it to field Syntax: ....|rex field=_raw "...(?P"Error...blah blah")" Refer: http://docs.splunk.com/Documentation/Splunk/4.3/SearchReference/Rex Using evals to look for Error and assign a generic value to the new Field. Example Syntax: ....|eval Valid=if(match(_raw,"Error"),"Error","") etc (Endless possibilities here) Hope this leads you where you want to be 🙂 Thanks, Raghav ... View more

First, script looks very good. It gives a lot of options to pick....I do a lot of backfilling myself based on search names/app etc i have modified the fill_summary_index.py to suit the best for a Search head clustering environment and pick a time when the schedules are minimum. The option j cannot exceed the number of cores for the search head (I can put a 1000 in there but if my machine has only 16 cores, 16 searches is all it can run at any give time (concurrently)) I typically rely heavily on dedup -true as that's not going to harm the performance (It simply does not execute if the job has already run). That being said, there's no backfilling best practice perse . I however pick a list f searches that have a lot in common (Example: schedule time ranges) If i have 10 searches that run in 15 min difference, i will pick a -et,-lt to cover the search window for all the 10 and use -dedup true true to ignore the ones that already ran. ./splunk cmd python fill_summary_index.py -app search -name "All the crazy summaries" -dedup true -showprogress true -j 16 (That's all my search head can handle, 16 searches concurrently) -owner admin -auth admin:admin. Since the script you wrote covers everything...only way to overcome performance is , run few summary backfills from a different SHC member (If you have search head clustering) or even pooling.....Reason i had to edit fill_summary_index.py is i do not store any summary data on Search heads and i forward everything rom SHC to Indexers. Hope this helps! Thanks, Raghav ... View more

Thanks Michael , @MuS 🙂 ... View more

Excellent...glad it worked out...please accept this as answer so that it could help folks catch it easily 🙂 Thanks, Raghav ... View more

Please accept the answer that helped you, @Mus's or @Raghav2384's 🙂 Hope this helps! Thanks, Raghav ... View more

Please chown the directory to the user:group and try to do what you were doing. Thanks, Raghav ... View more

To add to what @MuS has stated index=_audit action=add|rex field=_raw "/etc/users/(?P\w+)/" (Select All time) this will give you the list of user/account name irrespective of the search head they are created on (I know we all have seen one rogue useraccount :)). Limitation is it will just give me the userids Hope you can expand this and get what you want. Thanks, Raghav ... View more

Unfortunately, if a user is created on a SHC member locally, that search doesn't find that user.Also, |rest /services/authentication/users is nothing but localhost/services/authentication/users so it would display only users from the search head you ran it. Thanks, Raghav ... View more

Correct and i see the reason why it's hard to have a default value for cumulativeSrchJobsQuota Since srchJobsquota has a defauilt of 3 , adding 6 users to this role kinda have to bump the role quota to 18 🙂 Considering the above 6 user example, if you do not set srchjobsquota but set cumulativequota to let's say 15. Delta is 3 (6 users with default 3 but role permits only 15 concurrent searches) having the 3 get in line. 3 searches jump on to "Queued" section. Hope this help! Thanks, Raghav ... View more

Guess i am not lucky as you 😞 updated the pass4SymmKey on deployer's server.conf ($SPLUNK_HOME/etc/system/local/), restarted splunkd on deployer. Once deployer is back up, ran ./splunk apply shcluster-bundle --answer-yes -target https://xyz.com:8089 -auth admin:password I get the same error again Error while deploying apps to first member: ConfDeploymentException: Error while fetching apps baseline on target=https://xyz.abc.com:8089 Non-200/201 status_code=401; {"messages":[{"type":"WARN","text":"call not properly authenticated"}]} Any thing else you suggest? Thanks, Raghav ... View more

Thank you Somesh....here's what i did I updated the pass4SymmKey on deployer's server.conf ($SPLUNK_HOME/etc/system/local/), restarted splunkd on deployer. Once deployer is back up, ran ./splunk apply shcluster-bundle --answer-yes -target https://xyz.com:8089 -auth admin:password I get the same error again Error while deploying apps to first member: ConfDeploymentException: Error while fetching apps baseline on target=https://xyz.abc.com:8089 Non-200/201 status_code=401; {"messages":[{"type":"WARN","text":"call not properly authenticated"}]} Any thing else you suggest? Thanks, Raghav ... View more

Thanks for the input. Looks like the link is expired. Question is, when you said you updated the pass4SymmKey, have you changed it on all the Search head cluster members first and then update it on deployer OR first on deployer and then the search head cluster members? My only worry/concern here is, deployer can knock out changes made if the content under shcluster/app/* is not the same as on the SHC members. Appreciate your help! Thanks, Raghav ... View more

Thanks Somesh. I can update the password by adding [shclustering]pass4SymmKey under deployer for sure. My question is, looks like the password is encrypted and the person who set do not remember it. Can i delete the existing encrypted password from deployer and introduce the same password to all the SHC members and initiate a rolling-restart. My only worry is, since deployer is capable of deleting content from SHC members, i need a little courage and words of wisdom from you guys 🙂 Appreciate all your help! Thanks, Raghav ... View more

How about you set the siteB's indexers in distsearch.conf? That way, unless you specify the index in the search, it won't search from the search peers of SiteB? I have done similar in the past where two instances needed to talk to each other but then, i didn't wanted certain peers to be searched by default (Unless otherwise specified in the search by calling splunk_server="X OR y OR z") Check out distsearch.conf http://docs.splunk.com/Documentation/Splunk/6.0/admin/Distsearchconf for more info. Hope this helps! Thanks, Raghav ... View more

This is the right syntax perc90() Thanks, Raghav ... View more

Have you used stream.filter function? I wrote my own py script using tweepy and it's much simpler as i can control the data outside of Splunk and feed the data i want to SPlunk. from tweepy.streaming import StreamListener. stream.filter(track=['This That','Movies','Audio'.....]) Hope this helps! Thanks, Raghav ... View more

Hey There, I am surprised how Splunk recommended you 24 cores for Indexers. The proven recommendation is to have multiple small chunks as indexers and off course fast disks and high I/O. search heads : these guys need more horse power : 24 cores, whatever max RAM you could get is better. indexers : fast disks , high I/O (example, if you plan to index 500GB / day, i would start with a min of 4 indexers with magic 12s (12 core, 12 RAM and a min 1200 IOPS) As far as the app/add-on load, if the app comes with a ton of custom extractions (check props, transforms etc) it is going to add extra load for sure. Also depends on the type of data you are consuming. i have seen some radius type logs with 170 kv pairs in each event. Several factors that could add load and experts from this forum can explain you better. Hope this helps! thanks, Raghav ... View more

Several reasons that can cause this issue. without looking at splunkd.log, it's hard to tell i can list few possibilities from my experience if it's an indexer, 1. see if there are any bucket clashes....if there are two buckets with same id, example db__120 in db and db__120 in colddb if it's a search head, 1. if you pushed an encrypted password file from deployer, shc member fail to parse the file as it doesn't know what the encrypted password is. Again, hard to tell without seeing the splunkd.log. Hope this helps! Thanks, Raghav ... View more

Adding to what muebel said, please read migrating search app twice if not more.....That's is going to tricky and please check the app.conf thoroughly after you consolidate all of the artifacts on Deployer before you push. Might take a lot of time to consolidate initially but will save you from this hassle on a long run 🙂 Thanks, Raghav ... View more