I'm in the process of setting up a new Splunk GovCloud instance, and I'm having no luck getting field extractions to work. We have an index that ingests JSON that includes a field in the following format, which is basically a comma separated set of values: { ... "customField":"s,TLS_CHACHA20_POLY1305_SHA256,0.e53c3217.1768417540.1550260,curl_D92CE15881831761FA790081ADA5975B,-,-,-,-,3%7e0480d07b4b8c1898", ... } The sourcetype for this data is cloned from the _json sourcetype and it's parsing all the fields properly.  I've created this regex that matches the above customField, and I've verified that it's working with a bunch of tests data via regex101.com: (?<Network>[ps]),(?<tlsCipher>[A-Z][^,]+),(?<requestID>[0-9a-f\.]+),(?<BotID>[^,]+),.,(?<is_mobile>.),(?<is_tablet>.),(?<is_wireless>.),(?<tlsFingerprint>.+)  The first thing I tried was to add an entry for my sourcetype that looks like this: EXTRACT-cf1 = "(?<AkamaiNetwork>[ps]),(?<tlsCipher>[A-Z][^,]+),(?<requestID>[0-9a-f\.]+),(?<BotID>[^,]+),.,(?<is_mobile>.),(?<is_tablet>.),(?<is_wireless>.),(?<tlsFingerprint>.+)" in customField But I never see these fields show up in my search results. I've tried both with and without quotes around the regular exprewssion. I know the regular expression is working because if I use rex to test it via | rex field=customField  "<regex>"  then it returns the fields. So what an I missing? Is there any way of debugging/troubleshooting this sort of issue short of a whole lot of trial and error? Would it make more sense to create a custom app that contains the sourcetype definition and a transform in transforms.conf to handle this?   ... View more

We are using Splunk Cloud 9.0.2303.201 and have version 9.0.4 of the Splunk Universal Forwarder installed on a RHEL 7.9 server. The UF is configured to monitor a log file that outputs JSON in this format:   {"text": "Ending run - duration 0:00:00.249782\n", "record": {"elapsed": {"repr": "0:00:00.264696", "seconds": 0.264696}, "exception": null, "extra": {"run_id": "b20xlqbi", "action": "status"}, "file": {"name": "alb-handler.py", "path": "scripts/alb-handler.py"}, "function": "exit_handler", "level": {"icon": "", "name": "INFO", "no": 20}, "line": 79, "message": "Ending run - duration 0:00:00.249782", "module": "alb-handler", "name": "__main__", "process": {"id": 28342, "name": "MainProcess"}, "thread": {"id": 140068303431488, "name": "MainThread"}, "time": {"repr": "2023-10-13 10:09:54.452713-04:00", "timestamp": 1697206194.452713}}}   Long story short, it seems that Splunk is getting confused by the multiple fields in the JSON that look like timestamps. The timestamp that should be used is the very last field in the JSON. I first set up a custom sourcetype that's a clone of the _json sourcetype by manually inputting some of these records via Settings -> Add Data.  Using that tool I was able to get Splunk to recognize the correct timestamp via the following settings:   TIMESTAMP_FIELDS = record.time.timestamp TIME_FORMAT = %s.%6N     When I load the above record by hand via Settings -> Add Data and use my custom sourcetype with the above fields then Splunk shows the _time field is being set properly,  so in this case it's 10/13/23 10:09:54.452 AM. The exact same record, when loaded through the Universal Forwarder, appears to be ignoring the TIMESTAMP_FIELDS parameter. It ends up with a date/time of 10/13/23 12:00:00.249 AM, which indicates that it's trying to extract the date/time from the "text" field at the very beginning of the JSON (the string "duration 0:00:00.249782"). The inputs.conf on the Universal Forwarder is quite simple:   [monitor:///app/appman/logs/combined_log.json] sourcetype = python-loguru index = test disabled = 0     Why is the date/time parsing working properly when I manually load these logs via the UI but not when being imported via the Universal Forwarder? ... View more