About zapping575

zapping575 · ‎05-23-2025

Thank you for the replies, both of which have been very helpful in resolving this issue. Cleaning up the sslRootCAPath settings on the UF is already a good thing by itself. Investigating the TLS negotiation ultimately lead me to realize that on the indexer, etc/system/local/server.conf did not exist. In the splunk 9.2.5 docker image, the default.yml file did apparently not get processed by ansible. All the other config files (web.conf, authorize.conf) were also nonexistent. The fact that there was not rootCACert stored on the indexer explains why the log message states "unknown CA"

zapping575 · ‎05-22-2025

I recently started using the HEC with TLS on my standalone testing instance and now I am seeing some behavior that I cannot make sense of. I assume that it is related to the fact that I configured both, TCP Input and HEC Input to use different certificates. The HEC Input is working fine, but when a UF tries to connect to the TCP Input, I get this error: 05-22-2025 07:39:18.469 +0000 ERROR TcpInputProc [2339416 FwdDataReceiverThread] - Error encountered for connection from src=REDACTED:31261. error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed - please check the output of the `openssl verify` command for the certificates involved; note that if certificate verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA certificate and the server certificate should not have the same Common Name. 05-22-2025 07:39:18.555 +0000 ERROR X509Verify [2339416 FwdDataReceiverThread] - Client X509 certificate (CN=REDACTED,CN=A,OU=B,DC=C,DC=D,DC=E) failed validation; error=19, reason="self signed certificate in certificate chain" 05-22-2025 07:39:18.555 +0000 WARN SSLCommon [2339416 FwdDataReceiverThread] - Received fatal SSL3 alert. ssl_state='error', alert_description='unknown CA'. 05-22-2025 07:39:18.555 +0000 ERROR TcpInputProc [2339416 FwdDataReceiverThread] - Error encountered for connection from src=10.253.192.20:32991. error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed - please check the output of the `openssl verify` command for the certificates involved; note that if certificate verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA certificate and the server certificate should not have the same Common Name. On the UF, I can see the following error message: 05-22-2025 07:39:17.953 +0000 WARN SSLCommon [1074 TcpOutEloop] - Received fatal SSL3 alert. ssl_state='SSLv3 read server session ticket A', alert_description='unknown CA'. 05-22-2025 07:39:17.953 +0000 ERROR TcpOutputFd [1074 TcpOutEloop] - Connection to host=REDACTED:9997 failed Below are my config files. I appreciate any pointers as to what I did wrong. Note: All files which are storing certificates are the "usual" order: For clientCert and serverCert First certificate, then private key For sslRootCAPath First issuing, then Root CA Standalone/Indexer: Server.conf [sslConfig] sslRootCAPath = /opt/splunk/etc/auth/mycerts/cert.pem Inputs.conf [splunktcp-ssl:9997] disabled=0 [SSL] serverCert = /opt/splunk/etc/auth/mycerts/cert0.pem sslPassword = REDACTED requireClientCert = true sslVersions = tls1.2 [http] disabled = 0 enableSSL = 1 serverCert = /opt/splunk/etc/auth/mycerts/cert1.pem sslPassword = REDACTED [http://whatthehec] disabled = 0 token = REDACTED UF: server.conf [sslConfig] serverCert = /mnt/certs/cert0.pem sslPassword = REDACTED sslRootCAPath = /mnt/certs/cert.pem sslVersions = tls1.2 outputs.conf: [tcpout] defaultGroup = def forwardedindex.2.whitelist = (_audit|_introspection|_internal) [tcpout:def] useACK = true server = server:9997 autoLBFrequency = 180 forceTimebasedAutoLB = false autoLBVolume = 5000000 maxQueueSize =100MB connectionTTL = 300 heartbeatFrequency = 350 writeTimeout = 300 sslVersions = tls1.2 clientCert = /mnt/certs/cert0.pem sslRootCAPath = /mnt/certs/cert.pem sslPassword = REDACTED sslVerifyServerCert = true

zapping575 · ‎01-20-2025

I see that INGEST_EVAL allows for the use of conditionals. Thank you very much, I'll give that a try.

zapping575 · ‎01-20-2025

I have an event like this: ~01~20241009-100922;899~19700101-000029;578~ASDF~QWER~YXCV There are two timestamps in this. I have setup my stanza to extract the second one. But in this particular case, the second one is what I consider "bad". For the record, here is my props.conf: [QWERTY] SHOULD_LINEMERGE = true BREAK_ONLY_BEFORE_DATE = true MAX_TIMESTAMP_LOOKAHEAD = 43 TIME_FORMAT = %Y%m%d-%H%M%S;%3N TIME_PREFIX = ^\#\d{2}\#.{0,19}\# MAX_DAYS_AGO = 10951 REPORT-1 = some-report-1 REPORT-2 = some-report-2 The consequence of this seems to be that splunk indexes the entire file as a single event, which is something i absolutely want to avoid. Also, I do need to use linemerging as the same file may contain xml dumps. So what I need is something that implements the following logic: if second_timestamp_is_bad: extract_first_timestamp() else: extract_second_timestamp() Any tips / hints on how to mitigate this scenario using only options / functionality provided by splunk are greatly appreciated.

zapping575 · ‎07-16-2024

To answer my own question This was a browser issue. Both the splunk REST API and Splunk Web must use https for the REST call to succeed. In my case, this means https://localhost:8000 for splunk web and https://localhost:8090 for the API

zapping575 · ‎07-12-2024

I have been experimenting with splunk-ui and created an app to make calls from splunk web to the splunk REST API. However, I keep getting errors like this: The same origin policy prohibits access to external resource at https://localhost:8090/servicesNS/nobody/path_redacted_but_is_valid?output_mode=json. (Reason: CORS-Header 'Access-Control-Allow-Origin' is missing) This is how the call looks like const url = `https://localhost:8090/servicesNS/nobody/${eventType.acl.app}/saved/eventtypes/${eventType.title}?output_mode=json` const response = await fetch(url, { credentials: "include", method: "POST", redirect: "follow", body: JSON.stringify({'search': eventType.content.search}) }); return response.json(); This is my server.conf [sslConfig] sslRootCAPath = /opt/splunk/etc/auth/mycerts/cert.pem [httpServer] crossOriginSharingPolicy = https://localhost:8090 crossOriginSharingHeaders = * I can access https://localhost:8090/servicesNS/* "by it self" in my browser. I am using Firefox 128 and splunk 9.0.5 I can set crossOriginSharingPolicy to "*" (without quotes), but that will cause the browser to reject any requests that require authentication, so this is no solution

zapping575 · ‎07-12-2024

I am looking into developing a custom splunk app that lets us manage our knowledge objects in bulk. The idea is to create custom REST endpoints and call them from a splunk-ui based app in splunk web. Looking into the documentation of custom REST endpoints (https://dev.splunk.com/enterprise/docs/devtools/customrestendpoints/customrestscript), it strikes me that the sample code imports a splunk module that I can find no documentation for. Am I missing out on something here? Am I just supposed to use splunklib? I would greatly appreciate feedback regarding the plan to use custom REST endpoints with a splunk web based UI.

zapping575 · ‎06-12-2023

Hi @isoutamo, thanks for the reply Unfortunately, I dont know how many processes are writing to said file. I can only use it "as is". You are right however, this issue should be addressed on the side of the application(s) writing to that file. Regards

zapping575 · ‎06-11-2023

Cheers @VatsalJagani Thank you for the help. I cannot use HF, I can only use the UF. Since there are no other answers, I figure that manually preprocessing is the only way to go in this case.

zapping575 · ‎06-06-2023

I have a particularly challenging log format and would appreciate any inputs on how to tackle this problem. Problem Looking for a feasible props.conf setup that will correctly index the log below Example (blank lines only added for readability): SINGLE_LINE_LOG_EVENT SINGLE_LINE_LOG_EVENT OTHER_SINGLE_LINE_LOG_EVENT Tue 06 Jun 10:00:00 UTC 2023 ANOTHER_SINGLE_LINE_LOG_EVENT Tue 06 Jun 10:00:01 UTC 2023 LARGE_MULTILINE_EVENT The first three lines are all single events and should be parsed accordingly. But they have no timestamp The fourth and fifth line together form a single event Lines 6 and 7 also form a single event, but the event from line 7 is a multiline event that shall be parsed as a single event I am prepared to make the sacrifice that the lines without timestamp get assigned the CURRENT timestamp, if there is no other solution for this. What I have already tried I tried using the following (the Regex looks for the timestamp) MUST_NOT_BREAK_AFTER = .{3}\s.{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\sUTC\s\d{4} MUST_BREAK_AFTER = .{3}\s.{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\sUTC\s\d{4} As well as this (I tried various combinations of this, with different capture groups. Note that the file in question only has newlines and no carriage returns, hence no '\r') SHOULD_LINEMERGE = false LINE_BREAKER = ([\n].{3}\s.{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\sUCT\s\d{4})

zapping575 · ‎03-06-2023

Of course, sorry I didnt think about that beforehand <row> <panel> <table> <title>Latest events</title> <search> <query>| rest /servicesNS/-/-/alerts/fired_alerts/- | search eai:acl.app = myapp severity = 3</query> <earliest>$alerts_timepicker.earliest$</earliest> <latest>$alerts_timepicker.latest$</latest> <refresh>10m</refresh> <refreshType>delay</refreshType> </search> <option name="drilldown">row</option> <option name="refresh.display">progressbar</option> <drilldown> <set token="search_id">$row.sid$</set> </drilldown> </table> </panel> </row> <row> <panel> <event> <title>Alert Details</title> <search> <query>| loadjob $search_id$ </query> <earliest>$alerts_timepicker.earliest$</earliest> <latest>$alerts_timepicker.latest$</latest> </search> <option name="list.drilldown">none</option> <option name="refresh.display">progressbar</option> </event> </panel> </row>

zapping575 · ‎03-06-2023

We are using a clustered SH setup. I have a dashboard that lists all triggered alerts. When a user clicks on one of the list items, I would like to use the sid as a token to use as argument for loadjob in another dashboard. The query is as simple as: | loadjob <long-sid> However currently when a row is clicked, the result is always "Search did not return any events. " I have configurered the tokens correctly and permissions also do not seem to be the issue. If I click the "open in search" button at the bottom of the dash I get the results of "| loadjob <sid>" as expected"

zapping575 · ‎02-20-2023

Ciao @gcusello Thank you for your continued help. I must be doing something fundamentally wrong. If I run the search as you describe it, it returns zero results. Current setup Index = "myIndex" eventtype = "myEventtype" returns 1196 events Lookup file "known-events" contains a single event, identified by the "composite primary key": _time, host, source_file I would thus expect that the query you provided returns 1196 - 1 results. To adress your point and make sure that the field names are the exact same, I tried this: index = myIndex eventtype = myEventtype | fields _time, host, source_file NOT [| inputlookup known-events.csv | fields _time, host, source_file ] This gives the following error: Error in 'fields' command: Invalid argument: 'source_file=some_file_name-[some-host_name].txt' I am not quite sure what to make of this

zapping575 · ‎02-17-2023

Ciao @gcusello Before going over to summary indeces, I would like to implement this using lookups instead. The Logic is the same as with summary indices: Run the search, check if any results are already included in the lookup file If there are any, remove the duplicates Count the remaining events Fire Alert (if desired) Write back / append the new events to lookup I cannot however get the inputlookup in a subsearch to work. Here is a very basic example: Note: source_file is a field that I extract in props.conf index = myIndex eventtype = myEventtype | fields _time, host, source_file | search NOT [ | inputlookup known-events.csv | fields _time, host, source_file ] I have added a single line to known-events.csv (for testing purposes), so I would expect that the number of results for "myIndex" and "myEventtype" would decrease by one, but I am getting either the same results as before or none at all. I have confirmed that the single line in known-events.csv is acutally there.

zapping575 · ‎02-14-2023

Hi @gcusello I cannot make that change on the productive system right away. But I have a dev environment where I just tested it. The search for index=index_in_question linecount > 1 now returns zero results, so this solved the problem. Thank you.

zapping575 · ‎02-14-2023

Ciao @gcusello The timestamp of the events that are merged together (in this example) is 2023-01-31 10:40:01 This is how the event in question appears in the original file (some entries truncated for clarity): Note that the first occurrence of the timestamp in question is on the second line. 2023-01-31 10:39:58 message1 2023-01-31 10:40:01 message2 2023-01-31 10:40:08 message3 2023-01-31 10:40:08 message4 2023-01-31 10:40:00 some message 2023-01-31 10:40:01 some message in between 2023-01-31 10:40:01 some message in between 2023-01-31 10:40:01 some message in between 2023-01-31 10:40:01 message5 2023-01-31 10:40:01 message5 This listing illustrates how the single, merged event appears in splunk search (with the timestamp mentioned above). 2023-01-31 10:40:01 message2 2023-01-31 10:40:01 message5 2023-01-31 10:40:01 message5 2023-01-31 10:40:05 some message from later Sharing the original data is difficult so I am hoping that this condensed version will suffice. There are no anomalies regarding missing newlines on any of the offending events. Regards,

zapping575 · ‎02-14-2023

I have a few files in which the log events happen to not be in chronological order. Specifically, an event with say, timestamp "2022-01-01 11:00:00" may occur towards the top of the log, while a different event (with a different event message) with the same timestamp may occur towards the bottom of the log. It is totally acceptable to have log events where the timestamps are exactly equal. What splunk is doing however, is merging all of these "distributed" events together into one single event. This should not happen. These are my config files: props.conf [mySourceType] # example: 2022-07-01T23:53:54 2022-07-01T23:53:54 TIME_FORMAT = %Y-%m-%dT%H:%M:%S REPORT-default = sourcefields-default transforms.conf [sourcefields-default] SOURCE_KEY = source REGEX = /files/(.*?)/(.*?)/(.*?)/(.*?)\-(.*) FORMAT = field1::$1 field2::$2 field3::$3 field4::$4 field5::$5

zapping575 · ‎02-14-2023

Ciao @gcusello thank you very much for your suggestion. I will post again as soon as I managed to make it work.

zapping575 · ‎02-14-2023

Ciao @gcusello Could you please direct me towards a wiki or manual article on how to acheive this? Regards

zapping575 · ‎02-14-2023

Ciao @gcusello Your assumption about the time frame is indeed correct. I have to search within a timeframe of the last 30 days. This is a requirement because I may only receive the data in an asynchronous manner (the most recent event in a new file might already be a day old, or even older, when I receive it) This leads to potential problems if I want to equal the scheduling period with the time frame, as you suggest. If I select a short scheduling period, there is a very real chance that I will miss out on some events that are no longer included in the time frame If I select a long scheduling period, there is a very real chance that I will only notice an event long after it occurred. Regards

zapping575 · ‎02-14-2023

I have a scheduled savedsearch that may return a result such as this _time, host, _raw 2023-01-01, host A, <some message> 2023-01-02, host A, <some message> 2023-01-03, host A, <some message> In this example, the content of <some message> causes an alert to fire, which is what I expect. Now, assume that a new event occurs and the next scheduled search returns this (changes in bold): 2023-01-01, host A, <some message> 2023-01-02, host A, <some message> 2023-01-03, host A, <some message> 2023-01-04, host A, <some message> 2023-01-05, host A, <some message> Problem: The next scheduled search will return the entire list (5 events) and thus trigger an alert containing these 5 events. However, 3 of these events were contained in a previous alert and are thus superfluous. Desired outcome: The new alert should only be triggered based on the two "new" events (in bold) What I have tried: Set trigger type to "for each event" and suppress for fields _time and host because I would assume that the combination of _time and host will uniquely identify the event to suppress I also tried to learn about dynamic input lookups, but the documentation seems to be lost / unavailable (http://wiki.splunk.com/Dynamically_Editing_Lookup_Tables)

zapping575 · ‎12-02-2022

Hehe 🙂 Alright, thanks for your input!

zapping575 · ‎12-02-2022

Hi @PickleRick Thanks for your reply. I think also that keeping track of files is something that I will have to implement myself. I was just hoping to be able to use what splunk has. On why not using monitor inputs: I have a http endpoint that receives log files from another system and extracts them to disk, where the forwarder then picks them up. I could use monitor mode, but because there is no log rotation or similar it will ultimately result in filling up the disk. What is charming about batch mode is that it ensures that the disk space is free again after a new file has been completely ingested.

zapping575 · ‎12-02-2022

Dear all, I have the use case that my splunk universal forwarder does not continuously monitor my logs. Because of this nature, I am using batch mode to have the files deleted after ingestion. Now, I occasionally receive log files which I have already received at an earlier point in time. Problem is: The features crcSalt, initCrcLength etc. are only available in monitor mode. This means that I am not able to benefit from splunks features to prevent duplicate ingestion of the same data. Any help on a solution for this is greatly appreciated.

zapping575 · ‎07-27-2022

Ciao @gcusello Yes, its working! Funnily enough, before I tried the RegEx shown in the OP, I tried one which is very close to yours, but didnt work :): ^Time:\s+ Thank you very much. While I know that this is an off topic question, a different problem has turned up now: The file contains a header that I cannot seem to get rid of. I tried setting FIELD_HEADER_REGEX = ==\[ but this again did not work. Here is the header: (notice the "==[", after which the first event begins) Info1: qwe Info2: asd Info3: yxc Info4: foo Info5: bar Info6: baz Info7: fizz Info8: buzz ==[

Posts	36
Solutions	2
Karma Given	9
Karma Received	2
Member Since	‎12-03-2021

Online Status	Offline
Date Last Visited	3 weeks ago

Using different certificates for TCP and HEC Input...

In an event with two timestamps, automatically cho...

Cannot get splunk web to send CORS headers

Splunk internal python modules documentation

How to index file with multiline events and interm...

Using loadjob in a dashboard returns no results?

Why are events with equal timestamps merged into o...

How to exclude previous events in alert throttling...

Detection of duplicate files in batch mode

How to extract timestamp in multiline event?

Re: Using different certificates for TCP and HEC I...

Using different certificates for TCP and HEC Input...

Re: In an event with two timestamps, automatically...

In an event with two timestamps, automatically cho...

Re: Cannot get splunk web to send CORS headers

Cannot get splunk web to send CORS headers

Splunk internal python modules documentation

Re: How to index file with multiline events and in...

Re: How to index file with multiline events and in...

How to index file with multiline events and interm...

Re: Using loadjob in a dashboard returns no result...

Using loadjob in a dashboard returns no results?

Re: Exclude previous events in alert throttling

Re: Exclude previous events in alert throttling

Re: Events with equal timestamps are merged into o...

Re: Events with equal timestamps are merged into o...

Why are events with equal timestamps merged into o...

Re: Exclude previous events in alert throttling

Re: Exclude previous events in alert throttling

Re: Exclude previous events in alert throttling

How to exclude previous events in alert throttling...

Re: Detection of duplicate files in batch mode

Re: Detection of duplicate files in batch mode

Detection of duplicate files in batch mode

Re: Timestamp extraction in multiline event

Are you a member of the Splunk Community?