Hi, i have a similar problem... at first my config to forward internal splunk indexes... [tcpout] forwardedindex.0.whitelist = _.* forwardedindex.filter.disable = false defaultGroup = TEST_IDX-CLUSTER [tcpout:TEST_IDX-CLUSTER] forceTimebasedAutoLB = true autoLBFrequency = 30 server = ID01.SPLUNK-TEST.local:9997,ID02.SPLUNK-TEST.local:9997,ID03.SPLUNK-TEST.local:9997 Then i debug with this search... index="_internal" | eval bucket=_bkt | eval indextime=_indextime | table _time, indextime, bucket splunk_server _raw | convert ctime(indextime) | stats count list(*) as * by _raw | where count>1 | fields * _raw | sort - indextime Output: bucket = every bucket is another in one event count = 2 or sometimes 3 indextime = every entry is equal splunk_server = 01,02,03 or 01,01 or 02,03 or 03,03 (many different combinations) Anyone an idea? Regards - Markus ... View more

Hi Jim,   Found this thread, as I am facing an issue with Tripwire, on splunk. It is generating this humongous file ( te_assets.csv ) approximately 26 GB within an hour. Due to this the splunk service stops as in the Search Head there is only a single partition. Seems like Tripwire keeps updating splunk lookup with a fresh copy of assets data. Is there a way to limit the generated lookup file on Splunk? Or any config changes to be done from Tripwire Side? Your help/feedback is highly appreciated.   ... View more

Not to revive this old thread, but to folks who visit this later with a similar question, the following app will do what OP is asking for: https://splunkbase.splunk.com/app/5237/ ... View more

Hi @xsstest , you could use an approach like this (this is an example because I haven't your search): index=wineventlog | eval day=if(date_mday=strftime(now(),"%d"),"today","yesterday") | stats count(eval(day="yesterday")) AS yesterday count(eval(day="today")) AS today BY EventCode Then make all the filters you like (greater than, less than, perc of, etc...). Ciao. Giuseppe ... View more

In addition to @woodcock's great answer, you should avoid the intermediate HF if you don't need it for a specific purpose. UFs distribute events among indexers better than an HF. Also, the HF can actually make the indexers work harder to process events. If you eliminate the HF, be sure to set useAck=true on the UFs. ... View more

The question still not resolved, and no one knows why? ... View more

very good answer... ... View more

Because there are some problems with the default 9997 port ... View more

Alternatively i would suggest you have a dynamic lookup with valid user list and refer that in your search to verify if the users attempting login are geniune ... View more

Does anyone know the answer? ... View more

This worked for me: | makeresults | eval raw="time=2017/11/23 9:27:58.000,table=reg_basic_info,session_id=A,mobile=13888888888,UUID=AAAAAA-AAAAA-BBBBB,sip=12.12.12.12,sn=0123144::time=2017/11/23 9:27:59.000,table=reg_basic_info,session_id=B,mobile=13888888888,UUID=AAAAAA-AAAAA-BBBBB,sip=12.12.12.12,sn=1231444::time=2017/11/23 9:27:58.000,table=reg_success,session_id=A,uid=123,register_result=success,sip=12.12.12.12::time=2017/11/23 9:27:59.000,table=reg_location,session_id=C,mobile=13888888888,province=Bejing,city=Bejing,district=Dongcheng,sn=0123144::time=2017/11/23 10:17:18.000,table=reg_location,session_id=A,mobile=13888888888,province=Bejing,city=Bejing,district=Dongcheng,sn=0447777::time=2017/11/23 9:27:57.000,table=reg_device_info,session_id=D,UUID=AAAAAA-AAAAA-BBBBB,mfr=Apple,models=iPhone,DevBase=My_Iphone::time=2017/11/23 9:31:57.000,table=reg_device_info,session_id=E,UUID=AAAAAA-AAAAA-BBBBB,,mfr=Apple,models=iPhone,DevBase=My_Iphone" | makemv delim="::" raw | mvexpand raw | rename raw AS _raw | kv | eval _time=strptime(time, "%Y/%m/%d %H:%M:%S.%3N") | fields - time | sort 0 _time | eventstats values(UUID) AS UUID BY session_id | eventstats values(DevBase) AS DevBase values(mfr) AS mfr values(models) AS models BY UUID | stats min(_time) AS _time values(*) AS * BY UUID session_id | search uid="*" | table _time mobile sip UUID register_result uid province city district mfr models DevBase So try this: (index=tomcat sourcetype=regsiter "reg_basic_info" "mobile" "udid") OR (index=tomcat sourcetype=regsiter "reg_success" "uid" "mobile") OR (index=tomcat sourcetype=regsiter "reg_location" "province" "city") OR (index=tomcat sourcetype=regsiter "reg_device_info" "udid" "models") | eventstats values(UUID) AS UUID BY session_id | eventstats values(DevBase) AS DevBase values(mfr) AS mfr values(models) AS models BY UUID | stats min(_time) AS _time values(*) AS * BY UUID session_id | search uid="*" | table _time mobile sip UUID register_result uid province city district mfr models DevBase ... View more

Ill answer some of these in parts... 1、There are a lot of duplicate events in summary index,I don't know why, the summary index will have duplicate events, and I'll make sure that the original event is not repeated ,(For example, there are no duplicate events in indexes such as nginx and Apache) SI (Summary Indexes) are populated based on scheduled searches or searches with the collect command. If you have not scheduled your searches that create the summary indexes properly, then you're going to get duplicated events.. E.g., if you have a schedule search that runs every hour searching earliest of -1d, then youre going to have huge overlapping of events. I'd check that first. 2、Sometimes, after the alarm is triggered, it doesn't necessarily write exactly to summary index (I'm sure the result is not empty)。That is to say, sometimes it doesn't write the results to summary index This could be the way you have the alert setup, if there are no results, it wont write the events... Hard to answer this. 3、The third question is a silly question. I'd like to ask if the summary index is for the report or for the alert? In Web UI - >settings - >Searches, reports, and alert, I choose an alert, and then enable summary indexing。Then I clicked on "Alerts" on the navigation bar, that alert has disappeared, but that alert appears in “Reports”. why ? Why does it transfer from alerts to reports when I enable the summary index? That enable option means write to summary index... Im not sure I understand the context full of this... Alerts will age out. But this being said, a SI can be used for reporting or alerts. Some customers aggregate data into SI, then run alerts against it. Additionally, ive seen customers create reports from SI, because its much much faster.. For your use case, you can most definitely use alerts to write events out to a summary index, and then report and alert off of that. You may want to look at the SI related commands such as collect (http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/SearchReference/Collect) You can use this in your base searches to populate your SI indexes.. There are some downsides to using summary indexing. Mainly being, sourcetype changes (to stash) and also that SI is reliant on your saved searches to run and populate. This means if you miss a scheduled search, the SI wont populate and you need to back fill for that missing time window. This could lead to duplicate events. This is where Datamodels have come in.. They automatically backfill jobs, and you can aggregate data as neccesary.. Hope that helps. ... View more

The simplest is to define an eventtype, as suggested, and use that for searching your useful logs. I'd recommend reading up on it and applying it to your problem to see if that helps. If that doesn't solve your problem, please describe your problem in more detail. ... View more

If you use the GUI or the Lookup File Editor app (https://splunkbase.splunk.com/app/1724/), these changes will be synchronized across the cluster. Do not use the Deployer for a simple Lookup File change. You are risking big trouble if you do. ... View more

event per second ... View more

you need to enable one of meta woot!'s scheduled searches. I generally use the 5 min one ... View more

@nick405060 - yes, you can default to every minute if that's appropriate to the use case. Ideally, though, you want to spread them out more, so that some are running every 3 minutes on (0,3...57), some on (1,4...58) and some on (2,5,...59). This avoids having a half zillion skipped searches affecting your health stats, just because they are all trying to run at the same moment every 1 minute. Also, where accurate for the particular use case and KPI, you really want to reduce the frequency on the less urgent/critical KPIs to leave bandwidth for the most critical ones. ... View more

Well, have you read any of it? 😉 If you want to permanently decommission an indexer peer from your indexer cluster, i.e. you don't want to add it back later, follow the link I shared and read the documentation on "Take a peer down permanently". It talks about using splunk offline --enforce-counts to decommission the peer. Before shutting down, this command will ensure that the cluster is put into a healthy state, i.e. RF and SF are met by the remaining peers before it shuts down And when it's shut down, why would you even try to restart it!? Maybe I am misunderstanding... ... View more

@xsstest - see the other question for information about multiple duplicate results in this kind of search. https://answers.splunk.com/answers/558452/too-many-search-jobs-found-in-the-dispatch-directo-3.html ... View more

Hi xsstest, I reckon this is still the best place to read about Where do I configure my Splunk settings? http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings but if you prefer the docs page here it is http://docs.splunk.com/Documentation/Splunk/latest/Admin/Configurationparametersandthedatapipeline cheers, MuS ... View more

hi xsstest, warm regards, alert 1 ... View more

no need to setup for each host. you can get a list of all hosts that are not connected/sent events for last 1hr, 24hrs, etc. do you want alerts depending on host or depending on host plus the exact logfile? The following search will alert you if there are any hosts that haven't sent any data for more than one hour (3600 seconds) compare last event's time to now |metadata type=hosts | eval since=now()-lastTime | search since>3600 |... OR # compare indexer's time when last event came to now |metadata type=hosts | eval since=now()-recentTime| search since>3600 |... ... View more