About zapping575

zapping575 · ‎05-05-2026

I am genuinely grateful for all of your feedback, thank you very much. ❤️ And I very much agree with the observation that the business logic is not solid and that I should have worked that one out earlier. What @yuanliu suggests is the solution that offers the best compromise. If I were to go with a "super basic" approach, I would just alert every time I see a starting event But: there are too many of those to check each of them manually. At least some degree of automation would be immensely helpful even if not ALL corner cases are covered The mininum viable solution would be: In case a starting event has been found and its subsequent ending event occurs more than 30 seconds later in time, I want to be notified. In case starting and/or ending events are found that do not fit the transaction pattern, I want to be notified (1). This is how i tried to do it: index=my_index sourcetype=my_sourcetype ("to FAIL" OR "from FAIL") | transaction host component startswith("to FAIL") endswith("from FAIL") maxevents=2 keepevicted=true keeporphans=true | where duration > 30 or closed_txn = 0 As an example, this is the very first result of this search 2026-04-24T11:04:52.2 component1 from FAIL... 2026-04-24T11:11:15.2 component1 ...to FAIL Both starting and ending events are present and for the same component which is good. But for some reason, the ending supposedly event occurs BEFORE the starting event. That one may have confused me a bit 😞 If we look into the original source file, we can see this: 2026-04-24T11:11:17.3 component1 from FAIL... 2026-04-24T11:11:15.2 component1 ...to FAIL 2026-04-24T11:04:52.2 component1 from FAIL... 2026-04-24T11:04:48.1 component1 ...to FAIL So actually, the source contained two perfectly benign transactions: Events (1,2) and events (3,4). But for some reason, splunk INCORRECTLY considers events (2,3) to be a transaction. @PickleRickto address your 4 main questions: For the sake of simplicity, lets assume that "...to FAIL" always denotes a starting event Fields that uniquely identify a transaction are "host" and "component" Situations where transactions may overlap are described in (1). Apologies for the confusion regarding the original timestamps. In my defense, have no control how they are created. Sometimes they are broken (in which case splunk falls back to using indextime), sometimes events are out of sequence. It's a given that I cannot change. It is an acceptable tradeoff to first identify all the good transactions, exclude them from the results and then check the "outliers" (caused by possibly bad timestamps) individually. (1) Listing of reasons that may break the transaction pattern: After a starting event occurs, the system may be rebooted. Therefore, no ending event may occur. The transaction is not closed. Because of incorrect timestamps in the source files, splunk might index a starting event and ending event with the same timestamp A starting event may occur in one source file, while the ending event may occur in another. We are using batch inputs for data ingestion

zapping575 · ‎05-04-2026

My goal is to solve the following: I have what I consider "starting" events. They contain the string "to FAIL". I have also what I consider "ending" events. They contain the string "from FAIL". I need to find cases in which a pair of these events is more than 30 seconds aparat (doesnt matter which one comes first, starting or ending) Similar to how transaction shows the results. I would like to have both the starting and the ending event in my result set Constraints: The search needs to be for each host, as well as the same component on said host (both component and host are available as fields) Sometimes, the order of my events can be unpredictable (timestamps may not be consistent). Because of this, the search should check both 30 sec in the future AND the past for an ending event. I tried using transactions as mentioned here: https://community.splunk.com/t5/Splunk-Dev/transaction-with-one-start-and-several-end-conditions/m-p/311697#M4229 , but I end up with some events being grouped together incorrectly. I also tried to go the streamstats way (as described in the post above) but I cannot adapt it to my requirements quite right. Here is an event sample that should be detected by the desired search (events more than 30 sec apart): 2020-2-20T11:11:11 host1 component1 ... to FAIL ... [random events] 2020-2-20T11:11:55 host1 component 1... from FAIL to ...

zapping575 · ‎04-29-2026

Rather unsurprisingly, this was another case of PEBKAC. Changing the value of HEADER_FIELD_LINE_NUMBER back to 0 did the trick. props.conf on the UF: [source:://my.csv] INDEXED_EXTRACTIONS = csv TIME_FORMAT = %Y%m%d%H%M%S%Q TIMESTAMP_FIELDS = Year,Month,Day,Hour,Minute,Second,Seq HEADER_FIELD_LINE_NUMBER = 0 HEADER_FIELD_DELIMITER = ; FIELD_DELIMITER = ; SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+) sourcetype = foo_bar props.conf on my local stand alone instance: [foo_bar] REPORT-foo_bar-default = sourcefields-default

zapping575 · ‎04-29-2026

One of my sourcetypes is a CSV file (with CSV header) I was using this sourcetype stanza in props.conf: [foo_bar] INDEXED_EXTRACTIONS = csv TIME_FORMAT = %Y%m%d%H%M%S%Q TIMESTAMP_FIELDS = Year,Month,Day,Hour,Minute,Second,Seq HEADER_FIELD_LINE_NUMBER = 1 HEADER_FIELD_DELIMITER = ; FIELD_DELIMITER = ; SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+) REPORT-foo_bar-default = sourcefields-default I have come to notice that with splunk cloud, the indexed extractions are apparently no longer being applied. While the sourcetype is correctly assigned and the REPORT is also being applied, the actual indexed extraction does not take place. When I run a search for sourcetype foo_bar, I cannot see any of the fields defined by the CSV header. I also made sure the CSV header is actually present. My setup is like this: (1) Universal Forwarder -> (2) Universal ("Intermediate") Forwarder -> (3) Splunk Cloud I tried applying these settings at both (1) and (3). In both cases it did not work. Am I missing out on something?

zapping575 · ‎03-04-2026

This was the deciding tip. Turs out there was an issue with the deployment server which is why the files were not rolled out to the forwarder. Thank you very much!

zapping575 · ‎03-03-2026

Hi @gcusello apologies if my previous post may have been ambiguous. Out of the two source/sourcetype stanza combinations I posted, only the first one (eventlogger) is working as expected. The second one does not work. Splunk just defaults to an "xml-2" sourcetype.

zapping575 · ‎02-27-2026

Hi @gcusello Thats a good question actually. I was just presuming that splunk software would support the use case, where two different files have the same kind of data and can thus be treated as the same sourcetype. I ended up creating a second sourcetype, as suggested but the outcome is still the same. I also introduced a filter to drop undesired events before ingestion (only keep events that begin with "<event "). props.conf [source::/tmp/logs/.../qwe/.../*EventLogger.log*] priority = 100 TZ = UTC sourcetype = eventlogger [eventlogger] KV_MODE = xml TIME_PREFIX = timestamp=" TIME_FORMAT = %Y-%m-%dT%H:%M:%S%z MAX_TIMESTAMP_LOOKAHEAD = 25 REPORT-mydefault = sourcefields-mydefault [source::/tmp/logs/.../qwe/.../*eventlog*.xml*] priority = 100 TZ = UTC sourcetype = eventlog [eventlog] KV_MODE = xml TIME_PREFIX = timestamp=" SHOULD_LINEMERGE = false BREAK_ONLY_BEFORE_DATE = false LINE_BREAKER = <event TIME_FORMAT = %Y-%m-%dT%H:%M:%S%z MAX_TIMESTAMP_LOOKAHEAD = 25 REPORT-mydefault = sourcefields-mydefault TRANSFORMS-null=setnull,eventlog-strip transforms.conf [setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue [eventlog-strip] REGEX = ^<event\s DEST_KEY = queue FORMAT = indexQueue [sourcefields-mydefault] SOURCE_KEY = source REGEX = /tmp/logs/(.*?)/(.*?)/(.*?)/ FORMAT = asd::$1 qwe::$2 yxc::$3

zapping575 · ‎02-27-2026

I have these two files: EventLogger.log HOSTNAME-eventlog-TIMESTAMP.xml The structure of their content is the same. Except that the .xml file may contain "<eventlog>" and "</eventlog>" at beginning and end. So I thought to use the same sourcetype for them. This stanza is working fine: [source::/tmp/logs/.../qwe/.../*EventLogger.log*] priority = 100 TZ = UTC sourcetype = eventlogger This one however, is not (splunk will apparently default and make a "xml-2" sourcetype): [source::/tmp/logs/.../qwe/.../*eventlog*.xml*] priority = 100 TZ = UTC sourcetype = eventlogger Using btool, I did confirm that on my indexer, the two stanzas above are actually present. I feel like I am missing out on something here.

zapping575 · ‎02-19-2026

Thanks!

zapping575 · ‎01-06-2026

We have setup SAML authentication using MS Azure as IDP using the following instructions: https://help.splunk.com/en/splunk-enterprise/administer/manage-users-and-security/9.3/use-saml-as-an-authentication-scheme-for-single-sign-on/configure-authentication-extensions-to-interface-with-your-saml-identity-provider#configure-authentication-extensions-for-microsoft-azure-using-splunk-web-0 Thinking ahead, we would like to document how to renew the clientSecret. The question is: When the value for clientSecret requires renewal, can we just add the new value (key value pair), or do we have to revamp/redo the entire Authentication Extensions section? (Script Path, Script Functions, etc...) Ideally, we are hoping that we can just add the new clientSecret and the platform will take care of it.

zapping575 · ‎11-13-2025

I think your remark regarding the knowledge management process is what is the core of this problem. The motivation, initially, was to do things "the smart way": Keep eventtype defintions in one place only Favor one report that covers many similar eventtypes over one report per eventtype Be able to make changes without requiring re-deployment or restarts (as noted by @tscroggins ) Make changes in one place only (if required) Trying to package all of these together (using whichever method) would however always end up in more complicated queries and/or definitions. These are becoming harder to reason about. If you consider continuity, a more straightforward approach is often desired. Which is why we had to take a step backwards and changed the approach to something which is easier to understand but requires a lot of manual work (I am really missing the option to create / manage knowledge objects in bulk). We now have a single, dedicated report for any eventtype we are interested in, using the REQUIRED_EVENTTYPES directive to speed up the search. Any extra changes we can make in the report definition, leaving the original eventtype untouched (and in its own app) I really appreciate everybodys thoughts and feedback. Thank you.

zapping575 · ‎11-06-2025

Thank you very much for your feedback. Regarding the maintenance: Access to the lookup file in question is controlled and only possible for users with a specific role. It is true that tracking each change to a lookup file is probably difficult (I didnt check if perhaps the splunk app for lookup file editing is logging these things), but since its only accessible to a handful of users, while I think this is definitely a tradeoff, it is still acceptable. Regarding the performance: You are 100% right that this performs way worse than your example. (I presume in your example, EventCode is an indexed field, making the search much faster.) 90% of my eventtypes are based on string matching definitions (sometimes with wildcards in them). We are not doing any field extraction at index time. In the query in the OP, I am passing tag::eventtype=some_types, this will effectively result in a large normalizedSearch with all the eventtypes (associated with the tag) concatenated together. I am definetely not a fan of this solution either, but it beats having to update every single definition in every single app in case changes occur. (referring to my answer to @tscroggins ) To limit the "footprint" of this solution, we are including this in the base search: _index_earliest=-1h _index_latest=now This then runs as a scheduled report every hour and writes the results to a summary index. It takes about 10 seconds to run. Now, since we already have to have all of this overhead, adding the additional filter is -from what I can see- not impacting performance any further. When I remove the filter, the search still takes about 10 seconds to run. I guess that what is becoming apparent is that there is a fundamental "issue" with the way our searches (reports) are constructed and the OP discussed here is just one side effect that comes from that. In the past, I had to change eventtypes, savedsearches and lookups across multiple apps even tho their content was all the same. This is very tedious and error prone. Coming up with a solution that allows to keep and edit a single source of truth for knowledge objects is proving to be difficult to implement with splunk without making some kinds of tradeoffs.

zapping575 · ‎11-06-2025

You are assuming that I pass an actual eventtype into the base query (eventtype=some_types). I am afraid that I cannot "economically" do that. In a nutshell, this is why: I have a large amount of eventtypes It is a maintenance nightmare to have a savedsearch for every eventtype Especially because the eventtypes are used in multiple apps and indices, which are access controlled for different user groups Regardless, thank you very much!

zapping575 · ‎11-05-2025

I sometimes need to make some changes to my eventtype definitions. However, I do not actually want to edit the query in the eventtype definition directly. It appears that the following is a viable solution: Create a new lookup (eventtype-filter.csv) with only two colums: eventtype, qry In the search that uses the eventtypes, add this | index=etc tag::eventtype=some_types | lookup eventtype-filter.csv eventtype OUTPUT qry | eval filtered = if(searchmatch("qry"), "true", "false") | where filtered = "true" The value for qry would look smth like this: NOT src=asdf AND NOT DST=qwer (i know this could be simplified) Now this seems to be working fine but I noticed a few things: It is required to put the OUTPUT from my lookup into quotes when passing it to searchmatch(). Otherwise searchmatch() will say that its input is not correct The actual value for qry (stored in the lookup) should also not have any quotes, as this did also cause errors My questions would be: If anyone is also using this, possibly confirming that this is viable and not "some hack". In case I need to use quotes in my qry value, are there any pitfalls to look out for?

zapping575 · ‎05-23-2025

Thank you for the replies, both of which have been very helpful in resolving this issue. Cleaning up the sslRootCAPath settings on the UF is already a good thing by itself. Investigating the TLS negotiation ultimately lead me to realize that on the indexer, etc/system/local/server.conf did not exist. In the splunk 9.2.5 docker image, the default.yml file did apparently not get processed by ansible. All the other config files (web.conf, authorize.conf) were also nonexistent. The fact that there was not rootCACert stored on the indexer explains why the log message states "unknown CA"

zapping575 · ‎05-22-2025

I recently started using the HEC with TLS on my standalone testing instance and now I am seeing some behavior that I cannot make sense of. I assume that it is related to the fact that I configured both, TCP Input and HEC Input to use different certificates. The HEC Input is working fine, but when a UF tries to connect to the TCP Input, I get this error: 05-22-2025 07:39:18.469 +0000 ERROR TcpInputProc [2339416 FwdDataReceiverThread] - Error encountered for connection from src=REDACTED:31261. error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed - please check the output of the `openssl verify` command for the certificates involved; note that if certificate verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA certificate and the server certificate should not have the same Common Name. 05-22-2025 07:39:18.555 +0000 ERROR X509Verify [2339416 FwdDataReceiverThread] - Client X509 certificate (CN=REDACTED,CN=A,OU=B,DC=C,DC=D,DC=E) failed validation; error=19, reason="self signed certificate in certificate chain" 05-22-2025 07:39:18.555 +0000 WARN SSLCommon [2339416 FwdDataReceiverThread] - Received fatal SSL3 alert. ssl_state='error', alert_description='unknown CA'. 05-22-2025 07:39:18.555 +0000 ERROR TcpInputProc [2339416 FwdDataReceiverThread] - Error encountered for connection from src=10.253.192.20:32991. error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed - please check the output of the `openssl verify` command for the certificates involved; note that if certificate verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA certificate and the server certificate should not have the same Common Name. On the UF, I can see the following error message: 05-22-2025 07:39:17.953 +0000 WARN SSLCommon [1074 TcpOutEloop] - Received fatal SSL3 alert. ssl_state='SSLv3 read server session ticket A', alert_description='unknown CA'. 05-22-2025 07:39:17.953 +0000 ERROR TcpOutputFd [1074 TcpOutEloop] - Connection to host=REDACTED:9997 failed Below are my config files. I appreciate any pointers as to what I did wrong. Note: All files which are storing certificates are the "usual" order: For clientCert and serverCert First certificate, then private key For sslRootCAPath First issuing, then Root CA Standalone/Indexer: Server.conf [sslConfig] sslRootCAPath = /opt/splunk/etc/auth/mycerts/cert.pem Inputs.conf [splunktcp-ssl:9997] disabled=0 [SSL] serverCert = /opt/splunk/etc/auth/mycerts/cert0.pem sslPassword = REDACTED requireClientCert = true sslVersions = tls1.2 [http] disabled = 0 enableSSL = 1 serverCert = /opt/splunk/etc/auth/mycerts/cert1.pem sslPassword = REDACTED [http://whatthehec] disabled = 0 token = REDACTED UF: server.conf [sslConfig] serverCert = /mnt/certs/cert0.pem sslPassword = REDACTED sslRootCAPath = /mnt/certs/cert.pem sslVersions = tls1.2 outputs.conf: [tcpout] defaultGroup = def forwardedindex.2.whitelist = (_audit|_introspection|_internal) [tcpout:def] useACK = true server = server:9997 autoLBFrequency = 180 forceTimebasedAutoLB = false autoLBVolume = 5000000 maxQueueSize =100MB connectionTTL = 300 heartbeatFrequency = 350 writeTimeout = 300 sslVersions = tls1.2 clientCert = /mnt/certs/cert0.pem sslRootCAPath = /mnt/certs/cert.pem sslPassword = REDACTED sslVerifyServerCert = true

zapping575 · ‎01-20-2025

I see that INGEST_EVAL allows for the use of conditionals. Thank you very much, I'll give that a try.

zapping575 · ‎01-20-2025

I have an event like this: ~01~20241009-100922;899~19700101-000029;578~ASDF~QWER~YXCV There are two timestamps in this. I have setup my stanza to extract the second one. But in this particular case, the second one is what I consider "bad". For the record, here is my props.conf: [QWERTY] SHOULD_LINEMERGE = true BREAK_ONLY_BEFORE_DATE = true MAX_TIMESTAMP_LOOKAHEAD = 43 TIME_FORMAT = %Y%m%d-%H%M%S;%3N TIME_PREFIX = ^\#\d{2}\#.{0,19}\# MAX_DAYS_AGO = 10951 REPORT-1 = some-report-1 REPORT-2 = some-report-2 The consequence of this seems to be that splunk indexes the entire file as a single event, which is something i absolutely want to avoid. Also, I do need to use linemerging as the same file may contain xml dumps. So what I need is something that implements the following logic: if second_timestamp_is_bad: extract_first_timestamp() else: extract_second_timestamp() Any tips / hints on how to mitigate this scenario using only options / functionality provided by splunk are greatly appreciated.

zapping575 · ‎07-16-2024

To answer my own question This was a browser issue. Both the splunk REST API and Splunk Web must use https for the REST call to succeed. In my case, this means https://localhost:8000 for splunk web and https://localhost:8090 for the API

zapping575 · ‎07-12-2024

I have been experimenting with splunk-ui and created an app to make calls from splunk web to the splunk REST API. However, I keep getting errors like this: The same origin policy prohibits access to external resource at https://localhost:8090/servicesNS/nobody/path_redacted_but_is_valid?output_mode=json. (Reason: CORS-Header 'Access-Control-Allow-Origin' is missing) This is how the call looks like const url = `https://localhost:8090/servicesNS/nobody/${eventType.acl.app}/saved/eventtypes/${eventType.title}?output_mode=json` const response = await fetch(url, { credentials: "include", method: "POST", redirect: "follow", body: JSON.stringify({'search': eventType.content.search}) }); return response.json(); This is my server.conf [sslConfig] sslRootCAPath = /opt/splunk/etc/auth/mycerts/cert.pem [httpServer] crossOriginSharingPolicy = https://localhost:8090 crossOriginSharingHeaders = * I can access https://localhost:8090/servicesNS/* "by it self" in my browser. I am using Firefox 128 and splunk 9.0.5 I can set crossOriginSharingPolicy to "*" (without quotes), but that will cause the browser to reject any requests that require authentication, so this is no solution

zapping575 · ‎07-12-2024

I am looking into developing a custom splunk app that lets us manage our knowledge objects in bulk. The idea is to create custom REST endpoints and call them from a splunk-ui based app in splunk web. Looking into the documentation of custom REST endpoints (https://dev.splunk.com/enterprise/docs/devtools/customrestendpoints/customrestscript), it strikes me that the sample code imports a splunk module that I can find no documentation for. Am I missing out on something here? Am I just supposed to use splunklib? I would greatly appreciate feedback regarding the plan to use custom REST endpoints with a splunk web based UI.

zapping575 · ‎06-12-2023

Hi @isoutamo, thanks for the reply Unfortunately, I dont know how many processes are writing to said file. I can only use it "as is". You are right however, this issue should be addressed on the side of the application(s) writing to that file. Regards

zapping575 · ‎06-11-2023

Cheers @VatsalJagani Thank you for the help. I cannot use HF, I can only use the UF. Since there are no other answers, I figure that manually preprocessing is the only way to go in this case.

zapping575 · ‎06-06-2023

I have a particularly challenging log format and would appreciate any inputs on how to tackle this problem. Problem Looking for a feasible props.conf setup that will correctly index the log below Example (blank lines only added for readability): SINGLE_LINE_LOG_EVENT SINGLE_LINE_LOG_EVENT OTHER_SINGLE_LINE_LOG_EVENT Tue 06 Jun 10:00:00 UTC 2023 ANOTHER_SINGLE_LINE_LOG_EVENT Tue 06 Jun 10:00:01 UTC 2023 LARGE_MULTILINE_EVENT The first three lines are all single events and should be parsed accordingly. But they have no timestamp The fourth and fifth line together form a single event Lines 6 and 7 also form a single event, but the event from line 7 is a multiline event that shall be parsed as a single event I am prepared to make the sacrifice that the lines without timestamp get assigned the CURRENT timestamp, if there is no other solution for this. What I have already tried I tried using the following (the Regex looks for the timestamp) MUST_NOT_BREAK_AFTER = .{3}\s.{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\sUTC\s\d{4} MUST_BREAK_AFTER = .{3}\s.{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\sUTC\s\d{4} As well as this (I tried various combinations of this, with different capture groups. Note that the file in question only has newlines and no carriage returns, hence no '\r') SHOULD_LINEMERGE = false LINE_BREAKER = ([\n].{3}\s.{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\sUCT\s\d{4})

zapping575 · ‎03-06-2023

Of course, sorry I didnt think about that beforehand <row> <panel> <table> <title>Latest events</title> <search> <query>| rest /servicesNS/-/-/alerts/fired_alerts/- | search eai:acl.app = myapp severity = 3</query> <earliest>$alerts_timepicker.earliest$</earliest> <latest>$alerts_timepicker.latest$</latest> <refresh>10m</refresh> <refreshType>delay</refreshType> </search> <option name="drilldown">row</option> <option name="refresh.display">progressbar</option> <drilldown> <set token="search_id">$row.sid$</set> </drilldown> </table> </panel> </row> <row> <panel> <event> <title>Alert Details</title> <search> <query>| loadjob $search_id$ </query> <earliest>$alerts_timepicker.earliest$</earliest> <latest>$alerts_timepicker.latest$</latest> </search> <option name="list.drilldown">none</option> <option name="refresh.display">progressbar</option> </event> </panel> </row>

Posts	50
Solutions	3
Karma Given	23
Karma Received	2
Member Since	‎12-03-2021

Online Status	Offline
Date Last Visited	‎08-26-2026 10:05 AM

Transaction with starting and ending event

INDEXED_EXTRACTIONS, Universal Forwarder & Splunk ...

props source stanza will not match

About updating SAML Azure ClientSecret

Using lookups to augment searches with additional ...

Using different certificates for TCP and HEC Input...

In an event with two timestamps, automatically cho...

Cannot get splunk web to send CORS headers

Splunk internal python modules documentation

How to index file with multiline events and interm...

Re: Transaction with starting and ending event

Transaction with starting and ending event

Re: INDEXED_EXTRACTIONS, Universal Forwarder & Spl...

INDEXED_EXTRACTIONS, Universal Forwarder & Splunk ...

Re: props source stanza will not match

Re: props source stanza will not match

Re: props source stanza will not match

props source stanza will not match

Re: About updating SAML Azure ClientSecret

About updating SAML Azure ClientSecret

Re: Using lookups to augment searches with additio...

Re: Using lookups to augment searches with additio...

Re: Using lookups to augment searches with additio...

Using lookups to augment searches with additional ...

Re: Using different certificates for TCP and HEC I...

Using different certificates for TCP and HEC Input...

Re: In an event with two timestamps, automatically...

In an event with two timestamps, automatically cho...

Re: Cannot get splunk web to send CORS headers

Cannot get splunk web to send CORS headers

Splunk internal python modules documentation

Re: How to index file with multiline events and in...

Re: How to index file with multiline events and in...

How to index file with multiline events and interm...

Re: Using loadjob in a dashboard returns no result...

Join the Conversation