Hi, I'm trying to assign a list from a nested JSON event      { "timestamp": "2023-06-14T18:03:57.047201+00:00", . . "records": [ { "type": "A", "value": [] }, { "type": "AAAA", "value": [] }, { "type": "CNAME", "value": [] }, { "type": "NS", "value": [ "ns-0.blah.com", "ns-1.blah.org", "ns-1.blah.co.uk", "ns-1.blah.net" ] } ], "metadata": { . . } }          using this query       index=test | eval records=mvindex('records{}.value{}', mvfind('records{}.type',"NS"))       instead of getting all 4 entries in the list, I only got one entry (there is no other field similar to 'records', 'value', 'type')       ns-1.blah.net       Side comparison,  to show that a list can be assigned via an eval, when I collapse the nesting (removing the 'records' level) and adjusting the query, it's reads all 4 values (so it doesn't appear to be a variable 'type' problem)     index=test | eval records='value{}'           { "timestamp": "2023-06-14T17:00:00.123073+02:00", . . "value": [ "ns-0.blah.com", "ns-1.blah.co.uk", "ns-1.blah.net", "ns-1.blah.org" ], . . }           ns-0.blah.com ns-1.blah.co.uk ns-1.blah.net ns-1.blah.org         ... View more

Hello all, I'm trying to get extraction to work on a dynamic key value log. I've tried the following without any success (open to other suggestions away from this). Ideally the output should be: Thread=5\=/blah/blah Method=GET URI=/ Protocol=HTTP/1.1 IP=1.2.3.4 Port=54809 Referer=https://referrer field=value . . . field=value props.conf [testsourcetype_log] CHARSET=UTF-8 KV_MODE=none NO_BINARY_CHECK=true SHOULD_LINEMERGE=false category=Testing description=Test KV log sourcetype disabled=false pulldown_type=true REPORT-kv=kv_extraction EXTRACT-status=^(\d{4}\-\d{2}\-\d{2}T\d{2}:\d{2}:\d{2})\s\[(?<status>\w+) transforms.conf [kv_extraction] DELIMS = "]", ":" MV_ADD=true log snip: 2019-03-01T09:42:01 [status] [Thread: 5=/blah/blah] [Method: GET] [URI: /blah/blah] [Protocol: HTTP/1.1] [IP: 1.2.3.4] [Port: 54809] [Referer: https://referrer] [..] ... [..] text string here References: https://www.splunk.com/blog/2008/02/12/delimiter-based-key-value-pair-extraction.html https://answers.splunk.com/answers/170826/set-delimiter.html Thanks in advance UPDATE 6/25: I've tried combinations from @FrankVl, @VatsalJagani, @woodcock but it seems none of them work. Naturally, I've restarted splunk after each change. Here is the output from btool to show that I'm not going insane /opt/splunk/bin/splunk cmd btool props list [testsourcetype_log] ADD_EXTRA_TIME_FIELDS = True ANNOTATE_PUNCT = True AUTO_KV_JSON = true BREAK_ONLY_BEFORE = BREAK_ONLY_BEFORE_DATE = True CHARSET = UTF-8 DATETIME_CONFIG = /etc/datetime.xml DEPTH_LIMIT = 1000 HEADER_MODE = KV_MODE = none LEARN_MODEL = true LEARN_SOURCETYPE = true LINE_BREAKER_LOOKBEHIND = 100 MATCH_LIMIT = 100000 MAX_DAYS_AGO = 2000 MAX_DAYS_HENCE = 2 MAX_DIFF_SECS_AGO = 3600 MAX_DIFF_SECS_HENCE = 604800 MAX_EVENTS = 256 MAX_TIMESTAMP_LOOKAHEAD = 128 MUST_BREAK_AFTER = MUST_NOT_BREAK_AFTER = MUST_NOT_BREAK_BEFORE = NO_BINARY_CHECK = true SEGMENTATION = indexing SEGMENTATION-all = full SEGMENTATION-inner = inner SEGMENTATION-outer = outer SEGMENTATION-raw = none SEGMENTATION-standard = standard SHOULD_LINEMERGE = false TRANSFORMS = TRANSFORMS-kv = kv_extraction TRUNCATE = 10000 category = Testing description = Test KV log sourcetype detect_trailing_nulls = false disabled = false maxDist = 100 priority = pulldown_type = true sourcetype = /opt/splunk/bin/splunk cmd btool transforms list [kv_extraction] CAN_OPTIMIZE = True CLEAN_KEYS = True DEFAULT_VALUE = DEPTH_LIMIT = 1000 DEST_KEY = FORMAT = $1::$2 KEEP_EMPTY_VALS = False LOOKAHEAD = 4096 MATCH_LIMIT = 100000 MV_ADD = true REGEX = \[([^:[]+):\s+([^\]]+)] SOURCE_KEY = _raw WRITE_META = False Updated props.conf [testsourcetype_log] CHARSET=UTF-8 KV_MODE=none NO_BINARY_CHECK=true SHOULD_LINEMERGE=false category=Testing description=Test KV log sourcetype disabled=false pulldown_type=true TRANSFORMS-kv=kv_extraction updated transforms.conf [kv_extraction] REGEX = \[([^:[]+):\s+([^\]]+)] FORMAT = $1::$2 MV_ADD=true UPDATE 6/27: Using a clean splunk docker image, I: recreated indexers, inputs, props, transforms on the docker instance (external volume) stripped those files to a bare minimum renamed the sourcetype (to be sure that Splunk is reading props/transforms) moved the configs from being inside an app to system/local/*.conf checked the knowledge object existence via the gui (new/renamed transform is listed) checked the knowledge object permissions (global) and restarted after each change nada, log is being ingested but no new fields created (except for the value of thread that is field: 5 value: /blah/blah/ ) current config: $ cat system/local/inputs.conf [default] host = 7278c011e1e0 [monitor:///opt/splunk/var/log/testlogs/*.log] disabled=false sourcetype=blahblah index = testindex $ cat system/local/props.conf [blahblah] CHARSET=UTF-8 KV_MODE=none NO_BINARY_CHECK=true SHOULD_LINEMERGE=false category=Testing description=Test KV log sourcetype access disabled=false pulldown_type=true TRANSFORMS-blahkv=blahkvextraction #TRANSFORMS-replace_source = replacedefaultsource2 $ cat system/local/transforms.conf [blahkvextraction] FORMAT = $1::$2 MV_ADD = 1 #REGEX = \[([^:[]+):\s+([^\]]+)] REGEX = \[([^:\]]+):\s+([^\]]+)\] BTW: for @FrankVl, @VatsalJagani, @woodcock, thanks. I have used iterations of each of your code and strongly believe that it works. I've done variations of the below to prove that your solutions work and it does (I get one instance of field1=Thread, field2= value 😞 index=testindex sourcetype=blahblah | rex field=_raw "\[(?<field1>[^:\]]+):\s+(?<field2>[^\]]+)\]" ... View more

Hi, We've set up a new index cluster and is currently cloning the data between the existing and new indexers. https://docs.splunk.com/Documentation/Forwarder/7.2.6/Forwarder/Configureforwardingwithoutputs.conf#Configure_data_cloning_on_a_universal_forwarder_with_outputs.conf We: - can't 'reset' the forwarder as this will affect the existing indexers - can't copy the index across to the new indexers (different versions, etc). - still have the original log files (one shot is an option but we're open to better suggestions) Is there a way to ingest old data into the new cluster? Thanks in advance ... View more