My first instinct is to use the mvmap eval function to iterate over the alert_data.csv.content array and join the results into a new JSON array: | makeresults
| eval _raw="{\"alert_data\": {\"domain...
See more...
My first instinct is to use the mvmap eval function to iterate over the alert_data.csv.content array and join the results into a new JSON array: | makeresults
| eval _raw="{\"alert_data\": {\"domain\": \"abc.com\", \"csv\": {\"id\": 12345, \"name\": \"credentials.csv\", \"mimetype\": \"text/csv\", \"is_safe\": true, \"content\": [{\"username\": \"test1@abc.com\", \"password\":\"1qaz@WSX#EDC\"}, {\"username\": \"test2@abc.com\", \"password\":\"NotComplex\"}]}}}"
| eval _raw=json_set(json(_raw), "alert_data.csv.content", json("[".mvjoin(mvmap(json_array_to_mv(json_extract(json(_raw), "alert_data.csv.content{}")), json_set(_raw, "is_password_meet_complexity", if(len(json_extract(_raw, "password")) >= 8 AND (if(match(json_extract(_raw, "password"), "[[:digit:]]"), 1, 0) + if(match(json_extract(_raw, "password"), "[[:upper:]]"), 1, 0) + if(match(json_extract(_raw, "password"), "[[:lower:]]"), 1, 0) + if(match(json_extract(_raw, "password"), "[[:punct:]]"), 1, 0)) >= 3, "Yes", "No"))), ",")."]")) However, mvmap is not supported by INGEST_EVAL: "The following search-time eval functions are not currently supported at index-time with INGEST_EVAL: mvfilter, mvmap, searchmatch, now, and commands." See https://docs.splunk.com/Documentation/Splunk/latest/Data/IngestEval. To work around the missing functionality, we must analyze the input stream using an external process. We have several options available, but my preference lately for file (monitor) inputs is the props.conf unarchive_cmd setting. unarchive_cmd streams data to an external command over stdin and sends the command's stdout stream to the Splunk ingest pipeline. If we assume your file input is newline delimited JSON, unarchive_cmd allows us to read each object from stdin, process each content array item individually, and write the resulting object to stdout. Given alert_data.ndjson: {"alert_data": {"domain": "abc.com", "csv": {"id": 12345, "name": "credentials1.csv", "mimetype": "text/csv", "is_safe": true, "content": [{"username": "test1@abc.com", "password":"1qaz@WSX#EDC"}, {"username": "test2@abc.com", "password":"NotComplex"}]}}}
{"alert_data": {"domain": "abc.com", "csv": {"id": 67890, "name": "credentials2.csv", "mimetype": "text/csv", "is_safe": true, "content": [{"username": "test3@abc.com", "password":"passw0rd"}, {"username": "test4@abc.com", "password":"j#4kS.0e"}]}}} let's introduce an alert_data source type and construct inputs.conf and props.conf: # inputs.conf
[monitor:///tmp/alert_data.ndjson]
sourcetype = alert_data
# props.conf
[source::...alert_data.ndjson]
unarchive_cmd = python $SPLUNK_HOME/bin/scripts/preprocess_alert_data.py
unarchive_cmd_start_mode = direct
sourcetype = preprocess_alert_data
NO_BINARY_CHECK = true
[preprocess_alert_data]
invalid_cause = archive
is_valid = False
LEARN_MODEL = false
[alert_data]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
EVENT_BREAKER_ENABLE = true
EVENT_BREAKER = ([\r\n]+) Now let's write $SPLUNK_HOME/bin/scripts/preprocess_alert_data.py to read, process, and write JSON objects: import json
import re
import sys
for line in sys.stdin:
line = line.strip()
if not line:
continue
else:
try:
json_object = json.loads(line.rstrip())
for item in json_object["alert_data"]["csv"]["content"]:
meets_length_requirement = len(item["password"]) >= 8
digit_score = 1 if re.search(r"\d", item["password"]) else 0
upper_score = 1 if re.search(r"[A-Z]", item["password"]) else 0
lower_score = 1 if re.search(r"[a-z]", item["password"]) else 0
punct_score = 1 if re.search(r"[^a-zA-Z0-9\s]", item["password"]) else 0
meets_complexity_requirement = True if (digit_score + upper_score + lower_score + punct_score) >= 3 else False
if meets_length_requirement and meets_complexity_requirement:
item["is_password_meet_complexity"] = "Yes"
else:
item["is_password_meet_complexity"] = "No"
print(json.dumps(json_object))
except Exception as err:
print(err, file=sys.stderr)
print(line) On a full instance of Splunk Enterprise, i.e. a heavy forwarder, Splunk will use its local copy of Python. On a universal forwarder, we'll need to install Python 3.x and make sure the executable is in the path. At scale, this solution is better implemented as a modular input, but that's a separate topic for a larger discussion.