We have a 4 node SHC connected to a deployer.  For a usecase, I created a simple custom app that is just putting handful of dashboards together. Due to ease of use, I create this directly on SHC and all knowledge objects replicated among the members. During the next bundle push, will deployer delete this app from SHC as it has no knowledge of it? Should I move this app under shcluster/apps folder on the Deployer as well to be safe? Thanks, ~Abhi  ... View more

Hi, We have data from Change Auditor coming via HEC setup on a Heavy Forwarder. This HF instance was upgraded to Version 9.2.2. After that, I am seeing a difference in the way Splunk displays new events on SH. It is now converting UTC->PST.  I ran a search for previous week and for those events it is converting timestamp correctly, from UTC-> Eastern.  I am a little confused since both searches are done from same search head against same set of indexers. If there was a TZ issue, woudn't Splunk have converted both incorrectly?  I also ran same searches on indexer with identical output. Recent events in PST whereas older events continue to show as EST. Here are some examples For previous week   Recent. Splunk shows a UTC->PST conversion instead. I did test this manually via Add Data and Splunk is correctly formatting it to Eastern. How can I troubleshoot why recent events in search are showing PST conversion? My current TZ setting on SH is still set to Eastern Time. Also confirmed that system time for HF, indexers and Search Heads is set to Eastern.  Thanks  ... View more

Hi, We use Splunk Forwarder to monitor application data. There are multiple folders on a given server, each with same set of log files, but since the folder names are a distinguishing factor, we are using crcSalt=<SOURCE> so that Splunk treats all log files differently.  We also make sure to lock the stanza to a specific extension as needed, e.g. logname.log, or log*.txt, so that rotated files are ignored. That being said, I still want to find out if there are any situations where splunk could be re-indexing files multiple times and might warrant the use of initCrcLen instead.  Is this something that's possible via search? Does Splunk forwarder keeps some type of internal record/tracker that it is now re-indexing previously seen file again? Thanks, ... View more

Hi, Need some help with the following JSON data. ModifiedProperties: [ [-] { [-] Name: Group.ObjectID NewValue: 111111-2222222-333333-444444 OldValue: } { [-] Name: Group.DisplayName NewValue: Group A OldValue: } { [-] Name: Group.WellKnownObjectName NewValue: OldValue: } ] I want to extract the 2nd set of values for each event such that Group.DisplayName can become a field in itself, e.g. Group.DisplayName.NewValue=A, Group.DisplayName.OldValue=B. But right now, default extraction is doing something like this     How can I create KV pairs for Group.DisplayName within this JSON array? I tried few combinations using spath but was not successful.   Thank you ... View more

Hi, We have a datamodel built against application data. All the tstats searches against the DM were running fine, including the ones using summariesonly=true. I was noticing some discrepancy between data model and raw data when plotting timechart for the exact same time range. Checked on the Data Model and found _time field was not added. But after adding that and re-accelerating the data model, now i cant use summariesonly=true. No results are returned.  I do get data back without summariesonly=true.    What could have gone wrong here?   UPDATE I am able to search using  summariesonly=true (Maybe DM needed more time to regenerate) but now I see massive difference in counts between  summariesonly=true. Vs false. Data with false closely matches the raw data stats. Before that _time change, even  summariesonly=true was matching the counts precisely.  I see the _time field is set to "required" in the model but I don't think that would be preventing certain events from going into summary. All events in raw data do have default _time field.  Am I missing some key fact here on how summary calculation might have changed with addition of this _time field? ... View more

Hi all, I am trying to authenticate a user against REST API but when testing via CURL, it is failing when using LB URL(F5). User has replicated across all SHC members and can login via UI. # curl -k https://Splunk-LB-URL:8089/services/auth/login -d username=user -d password='password' <?xml version="1.0" encoding="UTF-8"?> <response> <messages> <msg type="WARN" code="incorrect_username_or_password">Login failed</msg> </messages> </response>   But when I try this same against the SH member directly, it works. # curl -k https://Splunk-SearchHead:8089/services/auth/login -d username=user -d password='password' <response> <sessionKey>gULiq_E7abGyEchXyw7rxzwi83Fhdh8gIGjPGBouFUd37GuXF</sessionKey> <messages> <msg code=""></msg> </messages> </response>   Initially I thought it could be something on the LB side but then for "admin" user, LB URL works just fine.  # curl -k https://Splunk-LB-URL:8089/services/auth/login -d username=admin -d password='password' <response> <sessionKey>gULiq_E7abGyEchXyw7rxzwi83Fhdh8gIGjPGBouFUd37GuXF</sessionKey> <messages> <msg code=""></msg> </messages> </response>   Has anyone come across issue like this? Why would admin work fine on LB but a new local user works only against direct SH and not via load balancer?  ... View more

Hi all, We are trying to deploy pre trained Deep Learning models for ESCU. DSDL has been installed and container are loaded successfully. Connection with docker is also in good shape.  But when running the ESCU search, I am getting the following error messages.    MLTKC error: /apply: ERROR: unable to initialize module. Ended with exception: No module named 'keras_preprocessing' MLTKC parameters: {'params': {'mode': 'stage', 'algo': 'pretrained_dga_model_dsdl'}, 'args': ['is_dga', 'domain'], 'target_variable': ['is_dga'], 'feature_variables': ['domain'], 'model_name': 'pretrained_dga_model_dsdl', 'algo_name': 'MLTKContainer', 'mlspl_limits': {'handle_new_cat': 'default', 'max_distinct_cat_values': '100', 'max_distinct_cat_values_for_classifiers': '100', 'max_distinct_cat_values_for_scoring': '100', 'max_fit_time': '600', 'max_inputs': '100000', 'max_memory_usage_mb': '4000', 'max_model_size_mb': '30', 'max_score_time': '600', 'use_sampling': 'true'}, 'kfold_cv': None, 'dispatch_dir': '/opt/splunk/var/run/splunk/dispatch/1704812182.86156_AC9C076F-2C37-4E94-9DD0-0AE04AEB7952'}     From search.log   01-09-2024 09:56:44.725 INFO ChunkedExternProcessor [47063 ChunkedExternProcessorStderrLogger] - stderr: MLTKC endpoint: https://docker_host:32802 01-09-2024 09:56:44.850 INFO ChunkedExternProcessor [47063 ChunkedExternProcessorStderrLogger] - stderr: POST endpoint [https://docker_host:32802/apply] called with payload (2298991 bytes) 01-09-2024 09:56:45.166 INFO ChunkedExternProcessor [47063 ChunkedExternProcessorStderrLogger] - stderr: POST endpoint [https://docker_host:32802/apply] returned with payload (134 bytes) with status 200 01-09-2024 09:56:45.166 ERROR ChunkedExternProcessor [47063 ChunkedExternProcessorStderrLogger] - stderr: MLTKC error: /apply: ERROR: unable to initialize module. Ended with exception: No module named 'keras_preprocessing' 01-09-2024 09:56:45.167 ERROR ChunkedExternProcessor [47063 ChunkedExternProcessorStderrLogger] - stderr: MLTKC parameters: {'params': {'mode': 'stage', 'algo': 'pretrained_dga_model_dsdl'}, 'args': ['is_dga', 'domain'], 'target_variable': ['is_dga'], 'feature_variables': ['domain'], 'model_name': 'pretrained_dga_model_dsdl', 'algo_name': 'MLTKContainer', 'mlspl_limits': {'handle_new_cat': 'default', 'max_distinct_cat_values': '100', 'max_distinct_cat_values_for_classifiers': '100', 'max_distinct_cat_values_for_scoring': '100', 'max_fit_time': '600', 'max_inputs': '100000', 'max_memory_usage_mb': '4000', 'max_model_size_mb': '30', 'max_score_time': '600', 'use_sampling': 'true'}, 'kfold_cv': None, 'dispatch_dir': '/opt/splunk/var/run/splunk/dispatch/1704812182.86156_AC9C076F-2C37-4E94-9DD0-0AE04AEB7952'} 01-09-2024 09:56:45.167 ERROR ChunkedExternProcessor [47063 ChunkedExternProcessorStderrLogger] - stderr: apply ended with options {'params': {'mode': 'stage', 'algo': 'pretrained_dga_model_dsdl'}, 'args': ['is_dga', 'domain'], 'target_variable': ['is_dga'], 'feature_variables': ['domain'], 'model_name': 'pretrained_dga_model_dsdl', 'algo_name': 'MLTKContainer', 'mlspl_limits': {'handle_new_cat': 'default', 'max_distinct_cat_values': '100', 'max_distinct_cat_values_for_classifiers': '100', 'max_distinct_cat_values_for_scoring': '100', 'max_fit_time': '600', 'max_inputs': '100000', 'max_memory_usage_mb': '4000', 'max_model_size_mb': '30', 'max_score_time': '600', 'use_sampling': 'true'}, 'kfold_cv': None, 'dispatch_dir': '/opt/splunk/var/run/splunk/dispatch/1704812182.86156_AC9C076F-2C37-4E94-9DD0-0AE04AEB7952'}     Has anyone run into this before?  We have Golden Image CPU running .  Following shows up in container logs.  Thanks ... View more