I am investigating higher CPU usage on my indexers, and am finding that this is a hard topic to really pinpoint. I run this search on my search head to identify different searches and the resource consumption, but the results are confusing me.         index=_introspection host=* source=*/resource_usage.log* component=PerProcess data.process_type="search" | stats latest(data.pct_cpu) AS resource_usage_cpu latest(data.mem_used) AS resource_usage_mem by _time, data.search_props.type,data.search_props.mode,data.search_props.user, data.search_props.app, host data.search_props.label data.elapsed data.search_props.search_head | sort - resource_usage_cpu           _time  data.search_props.type  data.search_props.mode  host  data.search_props.label data.elapsed data.search_props.search_head resource_usage_cpu 2022-11-01 10:23:54.338 scheduled historical batch idx04-k Process-Creation-Events-DomainController 1431.6000 sh02-g 95.40 2022-11-01 10:23:52.815 scheduled historical batch idx03-k Process-Creation-Events-DomainController 1430.0200 sh02-g 115.50 2022-11-01 10:23:50.738 scheduled historical batch idx05-k Process-Creation-Events-DomainController 1427.9800 sh02-g 105.70 2022-11-01 10:23:46.748 scheduled historical batch idx03-g Process-Creation-Events-DomainController 1424.0400 sh02-g 101.90 2022-11-01 10:23:45.081 scheduled historical batch idx02-k Process-Creation-Events-DomainController 1422.3200 sh02-g 97.90 From this, I can see that the search: 1) Was triggered from sh02 2) Was executed across several my indexers 3) Took ~1500 seconds to run 4) Consumed ~1 core on each instance BUT: The search is scheduled for once a day, and that time is not 10:23. It is scheduled for 11. (No window) There  are dozens on "instances" of this search being executed on all 10 of my indexers, triggered by sh02, in the ~10:22 timeframe. Maybe one row in the table above per indexer might make sense, but this is so many. What is happening here? How do I read these results to make a sane performance judgement about this situation? ... View more

I found this, but I am unable to replicate it. I am not understanding where I am messing up here. Problem: I feed btool outputs into splunk, and chop them up by stanza.   /opt/splunk/etc/apps/Splunk_TA_windows/default/transforms.conf [xmlsecurity_eventcode_errorcode_action_lookup] /opt/splunk/etc/system/default/transforms.conf CAN_OPTIMIZE = True /opt/splunk/etc/system/default/transforms.conf CLEAN_KEYS = True /opt/splunk/etc/system/default/transforms.conf DEFAULT_VALUE = /opt/splunk/etc/system/default/transforms.conf DEPTH_LIMIT = 1000 /opt/splunk/etc/system/default/transforms.conf DEST_KEY = /opt/splunk/etc/system/default/transforms.conf FORMAT = /opt/splunk/etc/system/default/transforms.conf KEEP_EMPTY_VALS = False /opt/splunk/etc/system/default/transforms.conf LOOKAHEAD = 4096 /opt/splunk/etc/system/default/transforms.conf MATCH_LIMIT = 100000 /opt/splunk/etc/system/default/transforms.conf MV_ADD = False /opt/splunk/etc/system/default/transforms.conf REGEX = /opt/splunk/etc/system/default/transforms.conf SOURCE_KEY = _raw /opt/splunk/etc/system/default/transforms.conf WRITE_META = False /opt/splunk/etc/apps/Splunk_TA_windows/default/transforms.conf case_sensitive_match = false /opt/splunk/etc/apps/Splunk_TA_windows/default/transforms.conf filename = xmlsecurity_eventcode_errorcode_action.csv   I then wanted to extract the fields, for example "SOURCE_KEY = _raw" should be my key/value pair! I hoped to accomplish this with  (transforms)   [dotheparsething] REGEX = \s([\S-]+)\s=\s([^\/\n]+) LOOKAHEAD = 100000 FORMAT = $1::$2 REPEAT_MATCH = true   (props)   [(?::){0}splunk:config:btool:*] TRUNCATE=10000 MAX_EVENTS=10000 KV_MODE = none BREAK_ONLY_BEFORE = conf[\s]+\[ #SEDCMD-removespaces = s/\ +/\ /g REPORT-dotheparsething = dotheparsething    But I am getting nothing! Regex101 seems happy with my search. ... View more

I have a search that a user recently moved from every hour to every 10 minutes. Cron:    3-59/10 * * * *    The search takes ~2 minutes to run. The window is set to auto. BUT, I see the issue:   10-25-2022 06:13:00.633 +0000 INFO SavedSplunker - savedsearch_id="nobody;rcc; Pull: Pull Domain IOCs from MISP", search_type="scheduled", user="thatOneGuyCausingProblemsForMe", app="myApp", savedsearch_name="Pull: Pull Domain IOCs from MISP", priority=default, status=skipped, reason="The maximum number of concurrent running jobs for this historical scheduled search on this cluster has been reached", concurrency_category="historical_scheduled", concurrency_context="saved-search_cluster-wide", concurrency_limit=1, scheduled_time=1666678380, window_time=-1, skipped_count=1, filtered_count=0     We have 4 very similar searches (similar schedule, duration, window, etc), all with the same error. The error fires off very consistently.   Splunk's complaint is that the given search is trying to run when another instance of the same search is running. But the searches only take ~2 minutes, and there is 10 minutes between them. I understand I can go into the limits.conf and change concurrency, but I do not see how these searches are overlapping themselves? I don't want to just hide the problem behind more CPUs ... View more

I have logs that seem to be extracting perfectly. All fields show up in "Interesting Fields", and each one can be searched (myField=*) gives results. EXCEPT: I have a field called "domain". It cannot be searched. It is there. It shows 100% of events have it. I hover over it and i see the contents. But when I run a search, nada.   index=disa-cbii   Search    index=disa-cbii domain="insight.adsrvr.org"   0 Results found. Same for domain=* Now, if we throw in spath   index=disa-cbii | spath domain | search domain="insight.adsrvr.org"   we get plenty of results. What is happening? From everything i can tell, I should not need spath because the event is extracting just fine. All the other dozen fields are extracted and searchable.    index=disa-cbii | table domain    works fine. How can I table something that doesn't exist? ... View more

I have created a custom search command, using the streaming search templates provided for the splunk SDK.  It is a simple "take in the results from x field, manipulate, and add in a couple new fields". This runs fine on my single instance development server. When I push it out to my search head cluster, I have 2 problems: 1) (Coincidence?) My Indexers all spiked in CPU around the same time i ran my search. Can a search head custom search impact the indexer? Looking at the docs, it seems like it would only impact the search head. 2) My search runs fine in a standalone instance, but in a distributed instance (SHC, Index cluster), I get this error. I dont see any more info on this -- how can I debug something so vague in splunk?  [idx01-g,idx01-k,idx02-g,idx02-k,idx03-g,idx03-k,idx04-g,idx04-k,idx05-g,idx05-k] Streamed search execute failed because: Error in 'punycode' command: External search command exited unexpectedly with non-zero error code 1.   Further investigation throws this (not on standalone, SHC only) but I changed it and am still getting this error.   ... View more

I recently inherited this splunk system, and I am gradually working out how it is set up. When running a search yesterday, I noticed something. We have 10 indexers, 5 at site1, 5 at site2. We have 4 search heads, all assigned to Site0. When inspecting my search job, I saw that my results were only pulled from a single site's peers, not from both. Here are some pictures to explain: My rep factor tells me I should have 2 copies at each site. My search factor tells me I should have 2 searchable copies at each site. This would imply that when I run a search across my 10 indexers, it would be pulling data from both sites. So then i run a search on a specific index, and I see this: I expected to see data pulled equally from both sites, but I see Site k is left completely alone. Even if a single indexer was the ingest point for all the data, it would still be scattered across the 10 indexers as it worked to meet the replication/search factors. There is no reason everything should be stuck on one site. Am I way off base here, or is something configured wrong? ... View more

I am ingesting some JSON events, and one of the fields is just a massive spammy "//0//0//0//0" repeated 15000+ times. I know my regexes are working fine, and I accomplished this by changing my lookahead in transforms:     [extractMessage] REGEX = "original":([\s\S]*?})}," LOOKAHEAD=100000 DEST_KEY= _raw FORMAT = $1 WRITE_META = true     BUT sedcmd doesnt listen to lookahead as defined in transforms, because it has to be called from props, and props has no lookahead! So looking at my props.conf:     [host::xx] SEDCMD-tst = s/(?:a){20,}/yoink/g     I made a bigass file of the letter "a", and counted how many chars were on each event. Then the sedcmd went in and replaced the "a"s with "yoink". Behold.... SEDCMD stops working at 4105 chars. I NEED MORE. How to expand SEDs reach?   ... View more

(Single/standalone instance of splunk) I have been in a fight with these events for over a week now. I was hoping eventually my failures would add up to a glorious success, but it turns out that I am finding EVEN MORE FAILURES. So many more. I am getting data from a source that provides single line json events. I have a few problems here: My JSON data has a consistent field located at ["event"]["original"], BUT the contents of .original often contain more nested data, which is breaking my regexes. I keep making new ones for each new "shape" I find, but it just seems tedious when the json contains it all nice and neat for me. Props:     [source::http:kafka_iap-suricata-log] LINE_BREAKER = (`~!\^<) SHOULD_LINEMERGE = false TRANSFORMS-also = extractSuriStats, extract_suri_protocol_msg, extractMessage​     Transforms:     [extractMessage] REGEX = "original":([\s\S]*?})}," LOOKAHEAD=100000 DEST_KEY= _raw FORMAT = $1 WRITE_META = true [extractSuriStats] REGEX = "event_type":"stats"[\s\S]+({"event_type":"stats".+})}} LOOKAHEAD=100000 DEST_KEY= _raw FORMAT = $1 WRITE_META = true [extract_suri_protocol_msg] REGEX = "original":([\s\S]*})}," LOOKAHEAD=100000 DEST_KEY= _raw FORMAT = $1 WRITE_META = true [sourcetyper] LOOKAHEAD = 100000 ​     This is fragile, and keeps breaking when a new "nested" shape comes through.  Now, lets assume the above works, but then BAM an event comes through with a payload of 47000 characters of "\\0" contained in the json. My above extractions continue to work, but the events themselves no longer parse at (searchtime?). I have pretty json, buy no key/value pairs that I can act off of. Ok, I think! What if i just replace the payload with --deleted--! Well, sedcmd seems to not apply too terribly often, and I wonder if it has the same character limitation but I dont see a limit to configure for it. My seds:     [source::http:kafka_iap-suricata-log] LINE_BREAKER = (`~!\^<) SHOULD_LINEMERGE = false SEDCMD-payload = s/payload_printable":([\s\S]*)",/ ---payload string has been truncated by splunk admins at index time--- /g SEDCMD-response = s/http_response_body_printable":([\s\S]*)"}/ ---payload string has been truncated by splunk admins at index time--- /g SEDCMD-fluff = s/(?:\\\\0){20,}/ ---html string has been truncated by splunk admins at index time--- /g TRANSFORMS-also = extractSuriStats, extract_suri_protocol_msg, extractMessage       What I would much prefer to do is again, just work with the json directly. But I dont think that is possible. My frustration continues, so I think what if i intercept the JSON and throw python things at it! I see a few references to using the unarchive_cmd , and get an idea...     #!/usr/bin/python import json import sys def ReadEvent(jsonSingleLine): data = json.loads(jsonSingleLine) return data def FindOriginalEvent(data): if 'event' in data: if 'original' in data['event']: originalEvent = data["event"]["original"] return originalEvent while True: fromSplunk = sys.stdin.readline() if not len(fromSplunk): break eventString = json.dumps(FindOriginalEvent(ReadEvent(fromSplunk))) sys.stdout.write(eventString) sys.stdout.flush() sys.exit()     Props:     [source::http:kafka_iap-suricata-log] LINE_BREAKER = (`~!\^<) SHOULD_LINEMERGE = false unarchive_cmd = /opt/splunk/etc/apps/stamus_for_splunk/bin/parse_suricata.py [(?::){0}suricata:*] invalid_cause = archive unarchive_cmd = /opt/splunk/etc/apps/stamus_for_splunk/bin/parse_suricata.py [suricata] invalid_cause = archive unarchive_cmd = /opt/splunk/etc/apps/stamus_for_splunk/bin/parse_suricata.py      (i put it everywhere, to make sure it would work.)  The code is ugly and useless. **bleep**. Art imitates life today.... So I am left with either: A bunch of regexes and sedcmds that break when the event is too long A custom script that I am apparently wrong on.  Which direction do I focus my attention? Any suggestions would be a huge help. Sample event:     {"destination": {"ip": "xxx","port": 443,"address": "xxx"},"ecs": {"version": "1.12.0"},"host": {"name": "ptm-nsm"},"fileset": {"name": "eve"},"input": {"type": "log"},"suricata": {"eve": {"http": {"http_method": "CONNECT","hostname": "xxx","status": 200,"length": 0,"http_port": 443,"url": "xxx","protocol": "HTTP/1.0","http_user_agent": "Mozilla/4.0 (compatible;)"},"payload_printable": "xxxxx","alert": {"metadata": {"updated_at": ["2021_11_24"],"created_at": ["2011_12_08"]},"category": "A Network Trojan was detected","gid": 1,"signature": "ET TROJAN Fake Variation of Mozilla 4.0 - Likely Trojan","action": "allowed","signature_id": 2014002,"rev": 10,"severity": 1,"rule": "alert http $HOME_NET any -> $EXTERNAL_NET any (msg:\"ET TROJAN Fake Variation of Mozilla 4.0 - Likely Trojan\"; flow:established,to_server; content:\"Mozilla/4.0|20 28|compatible|3b 29|\"; http_user_agent; fast_pattern; isdataat:!1,relative; content:!\".bluecoat.com\"; http_host; http_header_names; content:!\"BlueCoat\"; nocase; threshold:type limit, track by_src, count 1, seconds 60; classtype:trojan-activity; sid:2014002; rev:10; metadata:created_at 2011_12_08, updated_at 2021_11_24;)"},"packet": "RQA==","stream": 1,"flow_id": "769386515195888","app_proto": "http","flow": {"start": "2022-05-10T10:43:58.911344+0000","pkts_toclient": 3,"pkts_toserver": 4,"bytes_toserver": 1102,"bytes_toclient": 245},"event_type": "alert","tx_id": 0,"packet_info": {"linktype": 12}}},"service": {"type": "suricata"},"source": {"ip": "xxx","port": 64391,"address": "xxx"},"log": {"offset": 1062706606,"file": {"path": "/opt/suricata/eve.json"}},"network.direction": "external","@timestamp": "2022-05-10T10:43:59.106Z","agent": {"hostname": "xxx","ephemeral_id": "xxx","type": "filebeat","version": "7.16.2","id": "xxx","name": "ptm-nsm"},"tags": ["iap","suricata"],"@version": "1","event": {"created": "2022-05-10T10:43:59.340Z","module": "suricata","dataset": "suricata.eve","original": {"http": {"http_method": "CONNECT","hostname": "xxx","status": 200,"url": "xxx:443","http_port": 443,"length": 0,"protocol": "HTTP/1.0","http_user_agent": "Mozilla/4.0 (compatible;)"},"dest_port": 443,"payload_printable": "CONNECT xxx:443 HTTP/1.0\r\nUser-Agent: Mozilla/4.0 (compatible;)\r\nHost: xxx\r\n\r\n","alert": {"metadata": {"updated_at": ["2021_11_24"],"created_at": ["2011_12_08"]},"category": "A Network Trojan was detected","gid": 1,"action": "allowed","signature": "ET TROJAN Fake Variation of Mozilla 4.0 - Likely Trojan","signature_id": 2014002,"rev": 10,"severity": 1,"rule": "alert http $HOME_NET any -> $EXTERNAL_NET any (msg:\"ET TROJAN Fake Variation of Mozilla 4.0 - Likely Trojan\"; flow:established,to_server; content:\"Mozilla/4.0|20 28|compatible|3b 29|\"; http_user_agent; fast_pattern; isdataat:!1,relative; content:!\".bluecoat.com\"; http_host; http_header_names; content:!\"BlueCoat\"; nocase; threshold:type limit, track by_src, count 1, seconds 60; classtype:trojan-activity; sid:2014002; rev:10; metadata:created_at 2011_12_08, updated_at 2021_11_24;)"},"packet": "RQAAKAA9ZMAAA==","stream": 1,"flow_id": 769386515195888,"proto": "TCP","app_proto": "http","src_port": 64391,"dest_ip": "xxx","event_type": "alert","flow": {"start": "2022-05-10T10:43:58.911344+0000","pkts_toserver": 4,"pkts_toclient": 3,"bytes_toserver": 1102,"bytes_toclient": 245},"timestamp": "2022-05-10T10:43:59.106396+0000","tx_id": 0,"src_ip": "xxx","packet_info": {"linktype": 12}}},"network": {"transport": "TCP","community_id": "Ns="}}     ... View more