About ww9rivers

ww9rivers · ‎10-03-2025

I think I got it figured out! My search filter actually works with the :: syntax. However, I initially had double-quotes around the value for source, which is part of the search. Once I removed the double-quotes, it started working. So, 👇this does not work: source::"value" But this does: source::value Thank you all for the discussion!

ww9rivers · ‎10-02-2025

That may be true. But the document I linked in the original post clearly states "The SPL search filter field in the 4. Restrictions tab accepts any of the following search terms:" and "index::" is listed under that literally. Is the documentation wrong, then?

ww9rivers · ‎10-01-2025

Thank you for responding. My problem is not exactly with the "status=404" part. My problem is that when I use the "index=web" syntax, I get results. But when I change that to "index::web" I get nothing.

ww9rivers · ‎10-01-2025

I created a search filter that looks like this: (index=web NOT status=404) OR (index!=web) which works to limit the role to search for events with `status!=404` in the index `web`. However, Splunk document says this about the search filter syntax: "When you specify search term filters, use the key::value syntax, rather than key=value, where possible, to restrict search terms to indexed fields." So I changed that filter to: (index::web NOT status=404) OR (NOT index::web) I no longer get any results at all. No error / warning either. I guess I can use the `key=value` syntax instead. But I am curious why the `key::value` syntax does not work? Also, the document states that the `key::value` syntax provides better performance and security. We are using Splunk Cloud, currently running version 9.3.2411.116. The document for v9.3 and v9.4 say the same thing.

ww9rivers · ‎10-01-2025

Just saw this in my notifications... We resolved the issue with help from Splunk engineer. As it turned out, the logs that we believed to be FTD events (with the %FTD tag in them) were actually very likely still ASA logs (in ASA format, is Cisco's technical term). The CiscoSecurityCloud app comes with a built-in transform (set_ftd_sourcetype) that would convert the sourcetype from "cisco:asa" to "cisco:ftd:syslog" when it sees the "%FTD" tag in the events. We had to apply an "override" add-on that basically turns off the set_ftd_sourcetype transform. Cisco also came out with a new version (3.4.1) of the app during this whole ordeal. So we are currently running that plus the "override" to get the events properly parsed.

ww9rivers · ‎08-23-2025

Local accounts in Splunk Cloud with tokens work for search head API access. With "service account", I guess you mean to run unattended API actions. Token is the way to go for authentication. Not sure why you would need to white-list the Splunk Cloud IPs on-prem. You would initiate connection from on-prem server(s) to search heads in the Splunk Cloud. Right? I think you would only need white-list your corporate public IPs in the Splunk Cloud.

ww9rivers · ‎08-22-2025

I checked the Network pane in the DevTools, this seems to be the request that failed: GET https://<MYHOSTNAME>:8000/en-US/splunkd/__raw/services/appsbrowser/v1/app/?order=latest&offset=0&include=release%2Crelease.manifest%2Ccategories%2Ccreated_by%2Cicon%2Cdownload_count&limit=20 The response code is 502, a small XML body, with this message: Error connecting: error:0A000086:SSL routines::certificate verify failed - please check the output of the `openssl verify` command for the certificates involved; note that if certificate verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA certificate and the server certificate should not have the same Common Name. I ran openssl verify: echo | /app/splunk/bin/openssl s_client -connect <MYHOSTNAME>:8000 | /app/splunk/bin/openssl verify -CAfile /app/splunk/etc/auth/cacert.pem That verifies OK. The MYHOSTNAME is resolved with the `/etc/hosts` file on the host. And that does show it resolved OK in the browser DevTools. I guess this is a problem local to the host itself, as the GET request is likely trying to get a list of apps installed. But it is unclear what it is.

ww9rivers · ‎08-21-2025

Thank you for the response. I have not tried disabling extensions in Chrome. But I do not have any extensions in Edge, and I ran Firefox in Private mode. So I am not sure this is the issue. By the way, those were done on my work laptop with the same browsers I use for work where Splunk runs in the cloud, the version is 9.3.2411.112. I don't have the same issue there. That does remind me to try using my personal PC. I tried Edge there (Private window, no extension). The same error. I installed Chrome fresh on that PC . . . the same error.

ww9rivers · ‎08-20-2025

I am running Splunk Enterprise 10.0.0 on Rocky Linux 9.6 with a developer license. When I try to "Browse more apps" to install an app directly from Splunkbase, I get this in Chrome and Edge: Unexpected token '<', "<?xml vers"... is not valid JSON In Firefox, the message becomes "JSON.parse: unexpected character at line 1 column 1 of the JSON data" I can download apps from Splunkbase and then install them from files. But this is kinda annoying! Anyone else seeing this issue?

ww9rivers · ‎08-15-2025

Thank you. I exported 5 events from the FTD device: Aug 15 10:56:42 <FTD-IP-redacted> : Aug 15 14:56:41 UTC: %FTD-session-6-302015: Built inbound UDP connection 2853365636 for int-inside:<INSIDE-IP>/54408 (<PUBLIC-IP>/54408) to ext-outside:<OUTSIDE-IP>/443 (<OUTSIDE-IP>/443) Aug 15 10:56:42 <FTD-IP-redacted> : Aug 15 14:56:41 UTC: %FTD-session-4-106023: Deny udp src int-inside:<INSIDE-IP>/53542 dst ext-outside:<OUTSIDE-IP>/53 by access-group "CSM_FW_ACL_" [0xbb8ba4cf, 0x53f0f2f7] Aug 15 10:56:42 <FTD-IP-redacted> : Aug 15 14:56:41 UTC: %FTD-session-6-302013: Built inbound TCP connection 2853365631 for int-inside:<INSIDE-IP>/55126 (<PUBLIC-IP>/39066) to ext-outside:<OUTSIDE-IP>/443 (<OUTSIDE-IP>/443) Aug 15 10:56:42 <FTD-IP-redacted> : Aug 15 14:56:41 UTC: %FTD-session-6-302015: Built inbound UDP connection 2853365629 for int-inside:<INSIDE-IP>/51617 (<PUBLIC-IP>/51617) to ext-outside:<OUTSIDE-IP>/443 (<OUTSIDE-IP>/443) Aug 15 10:56:42 <FTD-IP-redacted> : Aug 15 14:56:41 UTC: %FTD-session-4-106023: Deny udp src int-inside:<INSIDE-IP>/62133 dst ext-outside:<OUTSIDE-IP>/53 by access-group "CSM_FW_ACL_" [0xbb8ba4cf, 0x53f0f2f7] I think I may need to change the message logging format to remove the rsyslog server timestamp and other added elements.

ww9rivers · ‎08-15-2025

I have test smart mode and verbose mode. Does not change the results. The other funny thing is, under the standard list of fields shown on the left side, there is "84 more fields". I click that, and get these strange stuff👇

ww9rivers · ‎08-15-2025

"A couple of logs"? Like what? This is running in the Splunk Cloud, maybe you could give me a search to show what logs are of interest?

ww9rivers · ‎08-14-2025

We are running the Cisco Security Cloud app in the Splunk Cloud. It does not seem to parse events of the `[cisco:ftd:syslog]` sourcetype -- The app does have that stanza in its `props.conf` file. Any one has the same issue? 2025-08-20 Edit: I installed the CiscoSecurityCloud app on a test instance, uploaded some FTD logs, with timestamp in rfc3339 format, the app fails to parse those logs -- I tried uploading one file with sourcetype set to [cisco:ftd:syslog] and another one set to [cisco:asa]. Both data sets ended up in Splunk with `sourcetype=cisco:ftd:syslog` without the fields.

ww9rivers · ‎08-12-2025

As it turns out, this TA is built different from the TA-pps_ondemand in that the TA-Proofpoint-TAP does NOT have its own UI. I guess I was misled by the TA redirecting me to the `/en-US/app/TA-Proofpoint-TAP/inputs` URL when I try to open the TA. To actually configure the input, one would have to go to Settings > Data inputs. Then page though the different input types to find "Proofpoint TAP SIEM Modular Input" and click + Add new to add a new input. I guess there is no configuring account for this type of input that can be shared between multiple instances. I guess version 1.3.150 is no different. Hope that helps.

ww9rivers · ‎08-11-2025

This turned out to be a really interesting problem caused by human error. We eventually figured out that the problem was caused by a typo in the `/etc/hosts` file for the host with `server-name-a` in my example. For the sake of discussion, say instead of `server-name-a`, it was `server-nam-a`. When Splunk parses and indexes data from `/var/log/messages`, it actually takes the host name parsed out of the events as `host`, which is not the case for `/var/log/secure`. That is why when we search for events with `host=server-name-*` we got the results from both files for host `server-name-b` but only events from `/var/log/secure` for host `server-name-a`. Usually we do not put the hostname in `/etc/hosts`. This might be a case where the admin on that host experimented with DNS name resolution or something.

ww9rivers · ‎08-06-2025

I am having the exact same issue with version 1.4.1 (also with 1.4.0). Page shows: /en-US/app/TA-Proofpoint-TAP/search Page blank: /en-US/app/TA-Proofpoint-TAP/inputs Page blank: /en-US/app/TA-Proofpoint-TAP/configuration Has there been a solution to the problem? I have entered a Splunk support case but they are not of any help as usual. I've also sent email to Proofpoint but no reply yet. Thanks for any fix/thoughts/ideas!

ww9rivers · ‎06-09-2025

Thank you very much for the detailed comments. I edited my post with some details. I did not suspect anything with regards to the monitor stanza because another host with essentially the same configuration works as expected. Where it doesn't work, I do find events from the /var/log/secure (from the same monitor stanza). I will run a btool debugging and report back. Thanks again!

ww9rivers · ‎06-09-2025

Thank you for your reply. I edited my post with some more details. It's a custom TA with a simple file monitor stanza. I don't think the inputs configuration is an issue.

ww9rivers · ‎06-09-2025

I have checked the file. Besides, one of the messages (Will begin reading at offset=13553847 for file='/var/log/messages') would indicate that Splunk has found the file which has contents. The host I have problem with is actually one in a pair (A/B) of servers. I do see events in Splunk from the B server as expected. I also find events from the A server from /var/log/secure, just nothing from /var/log/messages.

ww9rivers · ‎06-06-2025

I have a puzzle with a Linux host running RHEL 8.10, which is running Splunk Universal Forwarder 9.4.1, configured to forward data from local syslog files "/var/log/secure" and "/var/log/messages" to our Splunk indexers in the Splunk Cloud. Events from /var/log/secure are found there as expected. But no events are found from /var/log/messages. To troubleshoot, I did find these messages in the _internal index from the host: 06-02-2025 15:01:05.507 -0400 INFO WatchedFile [3811453 tailreader0] - Will begin reading at offset=13553847 for file='/var/log/messages'. 06-01-2025 03:21:02.729 -0400 INFO WatchedFile [2392 tailreader0] - File too small to check seekcrc, probably truncated. Will re-read entire file='/var/log/messages'. So the file was read but no events found in Splunk? [Edit 2025-06-09] The file inputs are configured with a simple stanza in a custom TA: [monitor:///var/log] whitelist = (messages$|secure$) index = os disabled = 0 As the stanza shows, two files are forwarded: /var/log/messages and /var/log/secure. With this search: | tstats count where index=os host=server-name-* by host source I get these results: host source count server-name-a /var/log/secure 39795 server-name-b /var/log/messages 112960 server-name-b /var/log/secure 21938 Server a and b are a pair running the same OS, patches, applications, etc..

ww9rivers · ‎02-24-2025

My earlier reply was "marked as spam" by the message board. Let me try again. Thank you for the reply. That is the one thing I checked and double checked in my attempt to fix my problem (the events data do not reach my Splunk Cloud instance)> But that may not be the problem as the message below (I removed data payload and hostnames for sanitary reasons) indicates that an index is provided: 02-21-2025 01:06:04.001 -0500 WARN TcpOutputProc [2061704 indexerPipe] - Pipeline data does not have indexKey. [_path] = /app/splunk/etc/apps/TA-json-modinput/bin/nix_input.py\n[python.version] = python3\n[_raw] = </data><done /></event><event stanza="nix_input://ni" unbroken="1"><source>nix_input://ni</source><sourcetype>hits:unix:hosts</sourcetype><index>test</index><data>{"hostname":"......",......}\n[_meta] = timestamp::none punct::"</><_/></><_=\"://\"_=\"\"><>://</><>::</><></><>{\"\":\""\n[_stmid] = GUsvaYoWsFPrNDD.H\n[MetaData:Source] = source::nix_input\n[MetaData:Host] = host::......\n[MetaData:Sourcetype] = sourcetype::nix_input\n[_linebreaker] = _linebreaker\n[_nfd] = _nfd\n[_charSet] = UTF-8\n[_time] = 1740117963\n[_conf] = source::nix_input|host::......|nix_input|28\n[_channel] = 28\n Much appreciated!

ww9rivers · ‎02-24-2025

Thank you much for the help. That is one thing I checked and double checked in my attempts of trying to fix the problem. But it may not be the case. In fact, the message shows that an index is provided in the event forwarding (I removed the payload and hostname for sanitary reasons): 02-21-2025 01:06:04.001 -0500 WARN TcpOutputProc [2061704 indexerPipe] - Pipeline data does not have indexKey. [_path] = /app/splunk/etc/apps/TA-json-modinput/bin/nix_input.py\n[python.version] = python3\n[_raw] = </data><done /></event><event stanza="nix_input://ni" unbroken="1"><source>nix_input://ni</source><sourcetype>hits:unix:hosts</sourcetype><index>test</index><data>{"hostname":"......",......}\n[_meta] = timestamp::none punct::"</><_/></><_=\"://\"_=\"\"><>://</><>::</><></><>{\"\":\""\n[_stmid] = GUsvaYoWsFPrNDD.H\n[MetaData:Source] = source::nix_input\n[MetaData:Host] = host::......\n[MetaData:Sourcetype] = sourcetype::nix_input\n[_linebreaker] = _linebreaker\n[_nfd] = _nfd\n[_charSet] = UTF-8\n[_time] = 1740117963\n[_conf] = source::nix_input|host::splunkhf-prod02|nix_input|28\n[_channel] = 28\n Since this is actually a WARN message, now I suspect if this is not the reason that my events data do not get into my Splunk Cloud instance (the TA is run on a heavy forwarder). Much appreciated!

ww9rivers · ‎02-23-2025

I am writing a simple TA to read a text file and turn it into a list of JSON events. I am getting a WARN message for each event from the TcpOutputProc process, such as the one below: 02-21-2025 01:06:04.001 -0500 WARN TcpOutputProc [2061704 indexerPipe] - Pipeline data does not have indexKey. I removed the rest of the message containing details. It seems that I am missing something simple. I would greatly appreciate some insights/pointers towards debugging this issue. The TA code is here in GitHub: https://github.com/ww9rivers/TA-json-modinput Many thanks in advance!

ww9rivers · ‎02-21-2025

Well, I expected it to authenticate the account when there is a token present. When Splunk knows that the account exists (It has authenticated before AND it has a token), why is that not sufficient for authentication?

ww9rivers · ‎02-21-2025

We ended up creating local (splunk) accounts for authenticating with token. Sorry for the late response.

Posts	124
Solutions	7
Karma Given	29
Karma Received	22
Member Since	‎07-27-2011

Online Status	Offline
Date Last Visited	‎10-03-2025 07:16 PM

Role-based search filter syntax

Splunk Enterprise 10.0.0 unable to "browse more ap...

CiscoSecurityCloud app not parsing fields for cisc...

Data missing from universal forwarder on a Linux h...

"Pipeline data does not have indexKey" in a simple...

Where is the extracted_eventtype field in the duo_...

Splunk DB connect 3.17.2 fail to start Task Server...

Got error after installation: OOOOPS - UNAUTHORIZE...

"Run as" owner not seem to work for saved searches

TA-asngen for Splunk Cloud?

Re: Role-based search filter syntax

Re: Role-based search filter syntax

Re: Role-based search filter syntax

Role-based search filter syntax

Re: CiscoSecurityCloud app not parsing fields for ...

Re: Splunk Cloud Search Head API access

Re: Splunk Enterprise 10.0.0 unable to "browse mor...

Re: Splunk Enterprise 10.0.0 unable to "browse mor...

Splunk Enterprise 10.0.0 unable to "browse more ap...

Re: CiscoSecurityCloud app not parsing fields for ...

Re: CiscoSecurityCloud app not parsing fields for ...

Re: CiscoSecurityCloud app not parsing fields for ...

CiscoSecurityCloud app not parsing fields for cisc...

Re: TA Proofpoint TAP v1.3.150 WebUI not displayin...

Re: Data missing from universal forwarder on a Lin...

Re: TA Proofpoint TAP v1.3.150 WebUI not displayin...

Re: Data missing from universal forwarder on a Lin...

Re: Data missing from universal forwarder on a Lin...

Re: Data missing from universal forwarder on a Lin...

Data missing from universal forwarder on a Linux h...

Re: "Pipeline data does not have indexKey" in a si...

Re: "Pipeline data does not have indexKey" in a si...

"Pipeline data does not have indexKey" in a simple...

Re: Does Splunk have a problem with token authenti...

Re: Does Splunk have a problem with token authenti...

Join the Conversation