About zapping575

zapping575

I think your remark regarding the knowledge management process is what is the core of this problem. The motivation, initially, was to do things "the smart way": Keep eventtype defintions in one place only Favor one report that covers many similar eventtypes over one report per eventtype Be able to make changes without requiring re-deployment or restarts (as noted by @tscroggins ) Make changes in one place only (if required) Trying to package all of these together (using whichever method) would however always end up in more complicated queries and/or definitions. These are becoming harder to reason about. If you consider continuity, a more straightforward approach is often desired. Which is why we had to take a step backwards and changed the approach to something which is easier to understand but requires a lot of manual work (I am really missing the option to create / manage knowledge objects in bulk). We now have a single, dedicated report for any eventtype we are interested in, using the REQUIRED_EVENTTYPES directive to speed up the search. Any extra changes we can make in the report definition, leaving the original eventtype untouched (and in its own app) I really appreciate everybodys thoughts and feedback. Thank you.

zapping575 · ‎11-06-2025

Thank you very much for your feedback. Regarding the maintenance: Access to the lookup file in question is controlled and only possible for users with a specific role. It is true that tracking each change to a lookup file is probably difficult (I didnt check if perhaps the splunk app for lookup file editing is logging these things), but since its only accessible to a handful of users, while I think this is definitely a tradeoff, it is still acceptable. Regarding the performance: You are 100% right that this performs way worse than your example. (I presume in your example, EventCode is an indexed field, making the search much faster.) 90% of my eventtypes are based on string matching definitions (sometimes with wildcards in them). We are not doing any field extraction at index time. In the query in the OP, I am passing tag::eventtype=some_types, this will effectively result in a large normalizedSearch with all the eventtypes (associated with the tag) concatenated together. I am definetely not a fan of this solution either, but it beats having to update every single definition in every single app in case changes occur. (referring to my answer to @tscroggins ) To limit the "footprint" of this solution, we are including this in the base search: _index_earliest=-1h _index_latest=now This then runs as a scheduled report every hour and writes the results to a summary index. It takes about 10 seconds to run. Now, since we already have to have all of this overhead, adding the additional filter is -from what I can see- not impacting performance any further. When I remove the filter, the search still takes about 10 seconds to run. I guess that what is becoming apparent is that there is a fundamental "issue" with the way our searches (reports) are constructed and the OP discussed here is just one side effect that comes from that. In the past, I had to change eventtypes, savedsearches and lookups across multiple apps even tho their content was all the same. This is very tedious and error prone. Coming up with a solution that allows to keep and edit a single source of truth for knowledge objects is proving to be difficult to implement with splunk without making some kinds of tradeoffs.

zapping575 · ‎11-06-2025

You are assuming that I pass an actual eventtype into the base query (eventtype=some_types). I am afraid that I cannot "economically" do that. In a nutshell, this is why: I have a large amount of eventtypes It is a maintenance nightmare to have a savedsearch for every eventtype Especially because the eventtypes are used in multiple apps and indices, which are access controlled for different user groups Regardless, thank you very much!

zapping575 · ‎11-05-2025

I sometimes need to make some changes to my eventtype definitions. However, I do not actually want to edit the query in the eventtype definition directly. It appears that the following is a viable solution: Create a new lookup (eventtype-filter.csv) with only two colums: eventtype, qry In the search that uses the eventtypes, add this | index=etc tag::eventtype=some_types | lookup eventtype-filter.csv eventtype OUTPUT qry | eval filtered = if(searchmatch("qry"), "true", "false") | where filtered = "true" The value for qry would look smth like this: NOT src=asdf AND NOT DST=qwer (i know this could be simplified) Now this seems to be working fine but I noticed a few things: It is required to put the OUTPUT from my lookup into quotes when passing it to searchmatch(). Otherwise searchmatch() will say that its input is not correct The actual value for qry (stored in the lookup) should also not have any quotes, as this did also cause errors My questions would be: If anyone is also using this, possibly confirming that this is viable and not "some hack". In case I need to use quotes in my qry value, are there any pitfalls to look out for?

zapping575 · ‎05-23-2025

Thank you for the replies, both of which have been very helpful in resolving this issue. Cleaning up the sslRootCAPath settings on the UF is already a good thing by itself. Investigating the TLS negotiation ultimately lead me to realize that on the indexer, etc/system/local/server.conf did not exist. In the splunk 9.2.5 docker image, the default.yml file did apparently not get processed by ansible. All the other config files (web.conf, authorize.conf) were also nonexistent. The fact that there was not rootCACert stored on the indexer explains why the log message states "unknown CA"

zapping575 · ‎05-22-2025

I recently started using the HEC with TLS on my standalone testing instance and now I am seeing some behavior that I cannot make sense of. I assume that it is related to the fact that I configured both, TCP Input and HEC Input to use different certificates. The HEC Input is working fine, but when a UF tries to connect to the TCP Input, I get this error: 05-22-2025 07:39:18.469 +0000 ERROR TcpInputProc [2339416 FwdDataReceiverThread] - Error encountered for connection from src=REDACTED:31261. error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed - please check the output of the `openssl verify` command for the certificates involved; note that if certificate verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA certificate and the server certificate should not have the same Common Name. 05-22-2025 07:39:18.555 +0000 ERROR X509Verify [2339416 FwdDataReceiverThread] - Client X509 certificate (CN=REDACTED,CN=A,OU=B,DC=C,DC=D,DC=E) failed validation; error=19, reason="self signed certificate in certificate chain" 05-22-2025 07:39:18.555 +0000 WARN SSLCommon [2339416 FwdDataReceiverThread] - Received fatal SSL3 alert. ssl_state='error', alert_description='unknown CA'. 05-22-2025 07:39:18.555 +0000 ERROR TcpInputProc [2339416 FwdDataReceiverThread] - Error encountered for connection from src=10.253.192.20:32991. error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed - please check the output of the `openssl verify` command for the certificates involved; note that if certificate verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA certificate and the server certificate should not have the same Common Name. On the UF, I can see the following error message: 05-22-2025 07:39:17.953 +0000 WARN SSLCommon [1074 TcpOutEloop] - Received fatal SSL3 alert. ssl_state='SSLv3 read server session ticket A', alert_description='unknown CA'. 05-22-2025 07:39:17.953 +0000 ERROR TcpOutputFd [1074 TcpOutEloop] - Connection to host=REDACTED:9997 failed Below are my config files. I appreciate any pointers as to what I did wrong. Note: All files which are storing certificates are the "usual" order: For clientCert and serverCert First certificate, then private key For sslRootCAPath First issuing, then Root CA Standalone/Indexer: Server.conf [sslConfig] sslRootCAPath = /opt/splunk/etc/auth/mycerts/cert.pem Inputs.conf [splunktcp-ssl:9997] disabled=0 [SSL] serverCert = /opt/splunk/etc/auth/mycerts/cert0.pem sslPassword = REDACTED requireClientCert = true sslVersions = tls1.2 [http] disabled = 0 enableSSL = 1 serverCert = /opt/splunk/etc/auth/mycerts/cert1.pem sslPassword = REDACTED [http://whatthehec] disabled = 0 token = REDACTED UF: server.conf [sslConfig] serverCert = /mnt/certs/cert0.pem sslPassword = REDACTED sslRootCAPath = /mnt/certs/cert.pem sslVersions = tls1.2 outputs.conf: [tcpout] defaultGroup = def forwardedindex.2.whitelist = (_audit|_introspection|_internal) [tcpout:def] useACK = true server = server:9997 autoLBFrequency = 180 forceTimebasedAutoLB = false autoLBVolume = 5000000 maxQueueSize =100MB connectionTTL = 300 heartbeatFrequency = 350 writeTimeout = 300 sslVersions = tls1.2 clientCert = /mnt/certs/cert0.pem sslRootCAPath = /mnt/certs/cert.pem sslPassword = REDACTED sslVerifyServerCert = true

zapping575 · ‎01-20-2025

I see that INGEST_EVAL allows for the use of conditionals. Thank you very much, I'll give that a try.

zapping575 · ‎01-20-2025

I have an event like this: ~01~20241009-100922;899~19700101-000029;578~ASDF~QWER~YXCV There are two timestamps in this. I have setup my stanza to extract the second one. But in this particular case, the second one is what I consider "bad". For the record, here is my props.conf: [QWERTY] SHOULD_LINEMERGE = true BREAK_ONLY_BEFORE_DATE = true MAX_TIMESTAMP_LOOKAHEAD = 43 TIME_FORMAT = %Y%m%d-%H%M%S;%3N TIME_PREFIX = ^\#\d{2}\#.{0,19}\# MAX_DAYS_AGO = 10951 REPORT-1 = some-report-1 REPORT-2 = some-report-2 The consequence of this seems to be that splunk indexes the entire file as a single event, which is something i absolutely want to avoid. Also, I do need to use linemerging as the same file may contain xml dumps. So what I need is something that implements the following logic: if second_timestamp_is_bad: extract_first_timestamp() else: extract_second_timestamp() Any tips / hints on how to mitigate this scenario using only options / functionality provided by splunk are greatly appreciated.

zapping575 · ‎07-16-2024

To answer my own question This was a browser issue. Both the splunk REST API and Splunk Web must use https for the REST call to succeed. In my case, this means https://localhost:8000 for splunk web and https://localhost:8090 for the API

zapping575 · ‎07-12-2024

I have been experimenting with splunk-ui and created an app to make calls from splunk web to the splunk REST API. However, I keep getting errors like this: The same origin policy prohibits access to external resource at https://localhost:8090/servicesNS/nobody/path_redacted_but_is_valid?output_mode=json. (Reason: CORS-Header 'Access-Control-Allow-Origin' is missing) This is how the call looks like const url = `https://localhost:8090/servicesNS/nobody/${eventType.acl.app}/saved/eventtypes/${eventType.title}?output_mode=json` const response = await fetch(url, { credentials: "include", method: "POST", redirect: "follow", body: JSON.stringify({'search': eventType.content.search}) }); return response.json(); This is my server.conf [sslConfig] sslRootCAPath = /opt/splunk/etc/auth/mycerts/cert.pem [httpServer] crossOriginSharingPolicy = https://localhost:8090 crossOriginSharingHeaders = * I can access https://localhost:8090/servicesNS/* "by it self" in my browser. I am using Firefox 128 and splunk 9.0.5 I can set crossOriginSharingPolicy to "*" (without quotes), but that will cause the browser to reject any requests that require authentication, so this is no solution

zapping575 · ‎07-12-2024

I am looking into developing a custom splunk app that lets us manage our knowledge objects in bulk. The idea is to create custom REST endpoints and call them from a splunk-ui based app in splunk web. Looking into the documentation of custom REST endpoints (https://dev.splunk.com/enterprise/docs/devtools/customrestendpoints/customrestscript), it strikes me that the sample code imports a splunk module that I can find no documentation for. Am I missing out on something here? Am I just supposed to use splunklib? I would greatly appreciate feedback regarding the plan to use custom REST endpoints with a splunk web based UI.

zapping575 · ‎06-12-2023

Hi @isoutamo, thanks for the reply Unfortunately, I dont know how many processes are writing to said file. I can only use it "as is". You are right however, this issue should be addressed on the side of the application(s) writing to that file. Regards

zapping575 · ‎06-11-2023

Cheers @VatsalJagani Thank you for the help. I cannot use HF, I can only use the UF. Since there are no other answers, I figure that manually preprocessing is the only way to go in this case.

zapping575 · ‎06-06-2023

I have a particularly challenging log format and would appreciate any inputs on how to tackle this problem. Problem Looking for a feasible props.conf setup that will correctly index the log below Example (blank lines only added for readability): SINGLE_LINE_LOG_EVENT SINGLE_LINE_LOG_EVENT OTHER_SINGLE_LINE_LOG_EVENT Tue 06 Jun 10:00:00 UTC 2023 ANOTHER_SINGLE_LINE_LOG_EVENT Tue 06 Jun 10:00:01 UTC 2023 LARGE_MULTILINE_EVENT The first three lines are all single events and should be parsed accordingly. But they have no timestamp The fourth and fifth line together form a single event Lines 6 and 7 also form a single event, but the event from line 7 is a multiline event that shall be parsed as a single event I am prepared to make the sacrifice that the lines without timestamp get assigned the CURRENT timestamp, if there is no other solution for this. What I have already tried I tried using the following (the Regex looks for the timestamp) MUST_NOT_BREAK_AFTER = .{3}\s.{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\sUTC\s\d{4} MUST_BREAK_AFTER = .{3}\s.{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\sUTC\s\d{4} As well as this (I tried various combinations of this, with different capture groups. Note that the file in question only has newlines and no carriage returns, hence no '\r') SHOULD_LINEMERGE = false LINE_BREAKER = ([\n].{3}\s.{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\sUCT\s\d{4})

zapping575 · ‎03-06-2023

Of course, sorry I didnt think about that beforehand <row> <panel> <table> <title>Latest events</title> <search> <query>| rest /servicesNS/-/-/alerts/fired_alerts/- | search eai:acl.app = myapp severity = 3</query> <earliest>$alerts_timepicker.earliest$</earliest> <latest>$alerts_timepicker.latest$</latest> <refresh>10m</refresh> <refreshType>delay</refreshType> </search> <option name="drilldown">row</option> <option name="refresh.display">progressbar</option> <drilldown> <set token="search_id">$row.sid$</set> </drilldown> </table> </panel> </row> <row> <panel> <event> <title>Alert Details</title> <search> <query>| loadjob $search_id$ </query> <earliest>$alerts_timepicker.earliest$</earliest> <latest>$alerts_timepicker.latest$</latest> </search> <option name="list.drilldown">none</option> <option name="refresh.display">progressbar</option> </event> </panel> </row>

zapping575 · ‎03-06-2023

We are using a clustered SH setup. I have a dashboard that lists all triggered alerts. When a user clicks on one of the list items, I would like to use the sid as a token to use as argument for loadjob in another dashboard. The query is as simple as: | loadjob <long-sid> However currently when a row is clicked, the result is always "Search did not return any events. " I have configurered the tokens correctly and permissions also do not seem to be the issue. If I click the "open in search" button at the bottom of the dash I get the results of "| loadjob <sid>" as expected"

zapping575 · ‎02-20-2023

Ciao @gcusello Thank you for your continued help. I must be doing something fundamentally wrong. If I run the search as you describe it, it returns zero results. Current setup Index = "myIndex" eventtype = "myEventtype" returns 1196 events Lookup file "known-events" contains a single event, identified by the "composite primary key": _time, host, source_file I would thus expect that the query you provided returns 1196 - 1 results. To adress your point and make sure that the field names are the exact same, I tried this: index = myIndex eventtype = myEventtype | fields _time, host, source_file NOT [| inputlookup known-events.csv | fields _time, host, source_file ] This gives the following error: Error in 'fields' command: Invalid argument: 'source_file=some_file_name-[some-host_name].txt' I am not quite sure what to make of this

zapping575 · ‎02-17-2023

Ciao @gcusello Before going over to summary indeces, I would like to implement this using lookups instead. The Logic is the same as with summary indices: Run the search, check if any results are already included in the lookup file If there are any, remove the duplicates Count the remaining events Fire Alert (if desired) Write back / append the new events to lookup I cannot however get the inputlookup in a subsearch to work. Here is a very basic example: Note: source_file is a field that I extract in props.conf index = myIndex eventtype = myEventtype | fields _time, host, source_file | search NOT [ | inputlookup known-events.csv | fields _time, host, source_file ] I have added a single line to known-events.csv (for testing purposes), so I would expect that the number of results for "myIndex" and "myEventtype" would decrease by one, but I am getting either the same results as before or none at all. I have confirmed that the single line in known-events.csv is acutally there.

zapping575 · ‎02-14-2023

Hi @gcusello I cannot make that change on the productive system right away. But I have a dev environment where I just tested it. The search for index=index_in_question linecount > 1 now returns zero results, so this solved the problem. Thank you.

zapping575 · ‎02-14-2023

Ciao @gcusello The timestamp of the events that are merged together (in this example) is 2023-01-31 10:40:01 This is how the event in question appears in the original file (some entries truncated for clarity): Note that the first occurrence of the timestamp in question is on the second line. 2023-01-31 10:39:58 message1 2023-01-31 10:40:01 message2 2023-01-31 10:40:08 message3 2023-01-31 10:40:08 message4 2023-01-31 10:40:00 some message 2023-01-31 10:40:01 some message in between 2023-01-31 10:40:01 some message in between 2023-01-31 10:40:01 some message in between 2023-01-31 10:40:01 message5 2023-01-31 10:40:01 message5 This listing illustrates how the single, merged event appears in splunk search (with the timestamp mentioned above). 2023-01-31 10:40:01 message2 2023-01-31 10:40:01 message5 2023-01-31 10:40:01 message5 2023-01-31 10:40:05 some message from later Sharing the original data is difficult so I am hoping that this condensed version will suffice. There are no anomalies regarding missing newlines on any of the offending events. Regards,

zapping575 · ‎02-14-2023

I have a few files in which the log events happen to not be in chronological order. Specifically, an event with say, timestamp "2022-01-01 11:00:00" may occur towards the top of the log, while a different event (with a different event message) with the same timestamp may occur towards the bottom of the log. It is totally acceptable to have log events where the timestamps are exactly equal. What splunk is doing however, is merging all of these "distributed" events together into one single event. This should not happen. These are my config files: props.conf [mySourceType] # example: 2022-07-01T23:53:54 2022-07-01T23:53:54 TIME_FORMAT = %Y-%m-%dT%H:%M:%S REPORT-default = sourcefields-default transforms.conf [sourcefields-default] SOURCE_KEY = source REGEX = /files/(.*?)/(.*?)/(.*?)/(.*?)\-(.*) FORMAT = field1::$1 field2::$2 field3::$3 field4::$4 field5::$5

zapping575 · ‎02-14-2023

Ciao @gcusello thank you very much for your suggestion. I will post again as soon as I managed to make it work.

zapping575 · ‎02-14-2023

Ciao @gcusello Could you please direct me towards a wiki or manual article on how to acheive this? Regards

zapping575 · ‎02-14-2023

Ciao @gcusello Your assumption about the time frame is indeed correct. I have to search within a timeframe of the last 30 days. This is a requirement because I may only receive the data in an asynchronous manner (the most recent event in a new file might already be a day old, or even older, when I receive it) This leads to potential problems if I want to equal the scheduling period with the time frame, as you suggest. If I select a short scheduling period, there is a very real chance that I will miss out on some events that are no longer included in the time frame If I select a long scheduling period, there is a very real chance that I will only notice an event long after it occurred. Regards

zapping575 · ‎02-14-2023

I have a scheduled savedsearch that may return a result such as this _time, host, _raw 2023-01-01, host A, <some message> 2023-01-02, host A, <some message> 2023-01-03, host A, <some message> In this example, the content of <some message> causes an alert to fire, which is what I expect. Now, assume that a new event occurs and the next scheduled search returns this (changes in bold): 2023-01-01, host A, <some message> 2023-01-02, host A, <some message> 2023-01-03, host A, <some message> 2023-01-04, host A, <some message> 2023-01-05, host A, <some message> Problem: The next scheduled search will return the entire list (5 events) and thus trigger an alert containing these 5 events. However, 3 of these events were contained in a previous alert and are thus superfluous. Desired outcome: The new alert should only be triggered based on the two "new" events (in bold) What I have tried: Set trigger type to "for each event" and suppress for fields _time and host because I would assume that the combination of _time and host will uniquely identify the event to suppress I also tried to learn about dynamic input lookups, but the documentation seems to be lost / unavailable (http://wiki.splunk.com/Dynamically_Editing_Lookup_Tables)

Posts	40
Solutions	2
Karma Given	15
Karma Received	2
Member Since	‎12-03-2021

Online Status	Offline
Date Last Visited	4 weeks ago

Using lookups to augment searches with additional ...

Using different certificates for TCP and HEC Input...

In an event with two timestamps, automatically cho...

Cannot get splunk web to send CORS headers

Splunk internal python modules documentation

How to index file with multiline events and interm...

Using loadjob in a dashboard returns no results?

Why are events with equal timestamps merged into o...

How to exclude previous events in alert throttling...

Detection of duplicate files in batch mode

Re: Using lookups to augment searches with additio...

Re: Using lookups to augment searches with additio...

Re: Using lookups to augment searches with additio...

Using lookups to augment searches with additional ...

Re: Using different certificates for TCP and HEC I...

Using different certificates for TCP and HEC Input...

Re: In an event with two timestamps, automatically...

In an event with two timestamps, automatically cho...

Re: Cannot get splunk web to send CORS headers

Cannot get splunk web to send CORS headers

Splunk internal python modules documentation

Re: How to index file with multiline events and in...

Re: How to index file with multiline events and in...

How to index file with multiline events and interm...

Re: Using loadjob in a dashboard returns no result...

Using loadjob in a dashboard returns no results?

Re: Exclude previous events in alert throttling

Re: Exclude previous events in alert throttling

Re: Events with equal timestamps are merged into o...

Re: Events with equal timestamps are merged into o...

Why are events with equal timestamps merged into o...

Re: Exclude previous events in alert throttling

Re: Exclude previous events in alert throttling

Re: Exclude previous events in alert throttling

How to exclude previous events in alert throttling...

Join the Conversation