Hello, I'm figuring out the best way to address the above situation. We do have a huge multisite cluster with 10 indexers on each site, a dedicated instance should act as the sc4s instance and send everything to a load balancer whose job will be to forward everything to the cluster.  now, there are several documentations about the implementation but I still can't wrap my head around the direct approach.  the SC4S config stanza would currently look something like this :   [http://SC4S] disabled = 0 source = sc4s sourcetype = sc4s:fallback index = main indexes = main, _metrics, firewall, proxy persistentQueueSize = 10MB queueSize = 5MB token = XXXXXX     several questions about that tho: - I'd need to create a hec token first, before configuring SC4S, but in a clustered environment - where do I create the hec token? I've read that I should create it on the CM and then push it to the peers but how exactly? I can't find much info about the specifics. especially since I try to configure it via config files.. so an example of the correct stanza that has to be pushed out would be somehow great - just can't find any.  - once pushed I need to configure the sc4s on the other side including the generated token (as seen above), does the config here seem correct? theres a lack of example configs so I'm spitballing here a little bit.   Kind regards ... View more

I'm creating a splunk multisite cluster. the configuration is done as the documentation shows, so I did with the cluster node. all peers show up and tell me they are up and are happily replicating. but for whatever reason the search factor and replication factor is not met. the notification about the unhealthy system tells me it's the cluster node :      but - why is that? how can I check what is wrong with it? If I look up the cluster status it all seems fine (via cli)   ... View more

Hello, I have let's say "inherited" a few searches and try to understand them. here is the search:   | lookup lu_cisco_umbrella_top_1_million domain as ut_domain output domain as cisco_umbrella_domain | lookup lu_majestic_top_1_million domain as ut_domain output domain as majestic_domain | search cisco_umbrella_domain=no_matches AND majestic_domain=no_matches     where I stumble uppon now is I understand how usually I could match domain to a field in my search results and in the output match another one to another field (like in the splunk lookup examples documentation) but  I can't wrap my head around how this could work with the same field twice? I also don't see the fields majestic_domain or cisco_umbrella_domain in the previous search results so how could this possibly work?  also, is there a way to take those two lookups and transfer it into something like this:      .... (NOT [|inputlookup lu_majestic_top_1_million domain |table domain ]) AND (NOT [|inputlookup lu_cisco_umbrella_top_1_million |table domain ])...     I bet I work in the totally false direction seemingly so I hope someone can help 🙂 tnx a lot ... View more

Hello, the following search      index=index1 message_type=query NOT ([|inputlookup lookup1 | fields ip_address |rename ip_address as dns_request_client_ip]) NOT dns_request_client_ip=127.0.0.1 |stats count by dns_request_client_ip     shows me 23300 matched events and shows me a table in statistics with those results.  but when I try to use tstats (in that case the datamodel Network_Resolution has all the data for index1) it  shows me 0 results, even tho when I only search the tstats datamodel with no other things like lookup etc. it gives me 5 million matches but doesn't show me anything in statistics.  also, in job inspector it shows me that the highlighted portion didn't result in any results and the only highlightet part is behind |tstats (in which nothing should be) and it says "NONE |tstats .... " why is this none there? my tstat is as follows:        |tstats count as count from datamodel=Network_Resolution where (message_type=query) by dns_request_client_ip     and then I try to combine it with the rest of the search as stated above via |search:        |search NOT ([|inputlookup lookup1 | fields ip_address |rename ip_address as dns_request_client_ip]) NOT dns_request_client_ip=127.0.0.1 |stats count by dns_request_client_ip       there must be something logically wrong with my approach, right?    thanks a lot for any help.  ... View more

Currently we're getting data from Azure Cloud which sends certain logs to a event hub our customer set up. then we pull the data from the eventhub just as stated in the documentation with the ms cloud services add-on.  our problem is now that our customer wanted to see some dashboards filled out with the incoming data. normal request we thought so we installed the microsoft azure app for splunk. there we saw nothing.  after further investigation we saw two things: - the incoming data fields are all extracted but horribly named with long strings of names - the sourcetype for all logs (around 7 different ones) is all something like xyz_eventhub which the app understandably doesn't know and can't use.    so my question is how to fix the issue of only having one sourcetype even tho the props/transforms within the cloud services add-on should extract everything perfectly. we currently think about splitting the data with help of regex and props/transforms conf into the needed sourcetypes but I'm like "why the frick doesn't it work in the first place? I mean the vendor is microsoft and not a third party no-name"   glad for any ideas guys! ... View more

Hello! I try onboarding several Trend Micro Cloud Applications like Apex One as a Service but it just doesn't work.  On the Apex One Cloud Platform I can get the URL, Application ID and API Key necessary to connect.  but it doesn't seem to work. I get the following errors in the apex_one_as_a_service_api.log :  2021-11-12 09:56:08,859 DEBUG pid=105063 tid=MainThread file=connectionpool.py:_make_request:437 | https://xj7qb2.manage.trendmicro.com:443 "GET /WebApp/api/v1/Logs/officescan_virus?output_format=CEF&page_token=0&since_time=1636707248 HTTP/1.1" 404 1245   and: 2021-11-12 10:00:08,804 ERROR pid=122037 tid=MainThread file=base_modinput.py:log_error:309 | Get error when collecting events. Traceback (most recent call last): File "/opt/splunk/etc/apps/Apex-One-as-a-Service/bin/apex_one_as_a_service/aob_py3/modinput_wrapper/base_modinput.py", line 128, in stream_events self.collect_events(ew) File "/opt/splunk/etc/apps/Apex-One-as-a-Service/bin/apex_one_as_a_service_api.py", line 64, in collect_events input_module.collect_events(self, ew) File "/opt/splunk/etc/apps/Apex-One-as-a-Service/bin/input_module_apex_one_as_a_service_api.py", line 91, in collect_events r_json = response.json() File "/opt/splunk/etc/apps/Apex-One-as-a-Service/bin/apex_one_as_a_service/aob_py3/requests/models.py", line 897, in json return complexjson.loads(self.text, **kwargs) File "/opt/splunk/lib/python3.7/json/__init__.py", line 348, in loads return _default_decoder.decode(s) File "/opt/splunk/lib/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/opt/splunk/lib/python3.7/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)    splunkd.log itself says the same:  11-12-2021 10:02:08.931 +0100 ERROR ExecProcessor - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/Apex-One-as-a-Service/bin/apex_one_as_a_service_api.py" ERRORExpecting value: line 1 column 1 (char 0)   I'm trying to use the following app for it:  https://splunkbase.splunk.com/app/5431/   What is wrong? does anyone know how to make this work?  PS: I'm sorry I can't use the "insert code" function here since it throws errors when I try.    ... View more

I was wondering what, i.e., the following means : 24 physical cores or 48 vcores . does that mean for a virtual environment I need double the physical cores to be vcores (physical cores = vcores) or is there some kind of relation I can deduce the amount of physical cores for a virtual environment from?  Also, if a virtual environment would be prefered partially because it would save up the amount of hardware needed (physical racks) and if a virtual server then would need to host two servers witch 24vcores each as a recommendation, would that mean the physical server would need 48 physical cores to provide those vcores for both machines? thanks a lot for clearing this up. I didn't find clear information in splunk docs and in the community as of now.  ... View more