Digging up an old thread to comment on a fix because I just ran into this.  If you look at the code in `qualys.py` it's asking for a duration between checks in seconds. So enter your time in seconds, i.e. `86400` instead of `24h` or `0 0 * * *` ... View more

Well I've solved my ingestion issues for CrashPlan's service log, now for the other 6. Here is the props stanza I ended up using. [crashplan_service] TIME_PREFIX = ^\[ BREAK_ONLY_BEFORE = (\r\n)?\[\d{2}\.\d{2}\.\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{3} MAX_TIMESTAMP_LOOKAHEAD = 21 MAX_EVENTS = 1000 NO_BINARY_CHECK = true TIME_FORMAT = %m.%d.%y %H:%M:%S.%3N This allowed Splunk to ingest Crashplan configuration output (yay 600+ lines of xml), into a single event. ... View more

@somesoni2 I tried that out and it someone worked but then I found a number of multiline events that didn't play nicely with that. I ended up using EVENT_BREAKER ... View more

And I found the answer, I needed to change the props.conf entry for that source from [WinEventLog:Microsoft-Windows-Windows Defender/Operational] to [source::WinEventLog:Microsoft-Windows-Windows Defender/Operational] Now everything is working. I knew it was a "duh" issue. ... View more

@martynoconnor So I've removed most of the EVALs and have a single static value one configured. Here is the btool props list for the sourcetype, notice the single eval (EVAL-brenden) from TA-windefender/local/props.conf. /opt/splunk/bin$ ./splunk btool props list "WinEventLog:Microsoft-Windows-Windows Defender/Operational" --debug /opt/splunk/etc/apps/TA-windefender/local/props.conf [WinEventLog:Microsoft-Windows-Windows Defender/Operational] /opt/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True /opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True /opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true /opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE = /opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True /opt/splunk/etc/system/default/props.conf CHARSET = UTF-8 /opt/splunk/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml /opt/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000 /opt/splunk/etc/apps/TA-windefender/local/props.conf EVAL-brenden = "brenden" /opt/splunk/etc/system/default/props.conf HEADER_MODE = /opt/splunk/etc/system/default/props.conf LEARN_MODEL = true /opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true /opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100 /opt/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000 /opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000 /opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2 /opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600 /opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800 /opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256 /opt/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128 /opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER = /opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER = /opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE = /opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing /opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full /opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner /opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer /opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none /opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard /opt/splunk/etc/system/default/props.conf SHOULD_LINEMERGE = True /opt/splunk/etc/system/default/props.conf TRANSFORMS = /opt/splunk/etc/system/default/props.conf TRUNCATE = 10000 /opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false /opt/splunk/etc/system/default/props.conf maxDist = 100 /opt/splunk/etc/system/default/props.conf priority = /opt/splunk/etc/apps/Splunk_TA_windows/default/props.conf rename = wineventlog /opt/splunk/etc/system/default/props.conf sourcetype = My props.conf looks like so: [WinEventLog:Microsoft-Windows-Windows Defender/Operational] EVAL-brenden = "brenden" ... View more

That would be helpful now wouldn't it. ... View more

Looks like there is an issue comment on github that has the complete EVAL. ... View more

So this was for a drill down to look at an event and the prior events leading up to it. A little token evals and I came up with the following <eval token="tok_threshold_time">"-". tostring($row.threshold$*4) ."m"</eval> <eval token="tok_earliest">relative_time($row.recentTime$, $tok_threshold_time$)</eval> <eval token="tok_latest">strptime($row.recentTime$,"%m/%d/%Y %H:%M:%S")</eval> <link target="_blank">search?q=index=$row.index$%20sourcetype=$row.sourcetype$%20earliest=$tok_earliest$%20latest=$row.recentTime$</link> Maybe not the most elegant but it works. ... View more

Good point, thanks. ... View more

@douglashurd Does the 3.6.x fixed version also include the missing eventtypes and tags that I mentioned in this question? ... View more

Here is my effective props for the cisco:estreamer sourcetype. $ ./splunk cmd btool props list cisco:estreamer [cisco:estreamer:data] ADD_EXTRA_TIME_FIELDS = True ANNOTATE_PUNCT = True AUTO_KV_JSON = true BREAK_ONLY_BEFORE = BREAK_ONLY_BEFORE_DATE = True CHARSET = UTF-8 DATETIME_CONFIG = /etc/datetime.xml DEPTH_LIMIT = 1000 FIELDALIAS-estreamer_app = app_proto AS app FIELDALIAS-estreamer_bytes_in = src_bytes AS bytes_in FIELDALIAS-estreamer_bytes_out = dest_bytes AS bytes_out FIELDALIAS-estreamer_category = malware_event_type AS malware_type FIELDALIAS-estreamer_connection_id = connection_counter AS connection_id FIELDALIAS-estreamer_date = event_sec AS date FIELDALIAS-estreamer_dest = dest_ip AS dest FIELDALIAS-estreamer_dest_interface = iface_egress AS dest_interface FIELDALIAS-estreamer_dest_zone = sec_zone_egress AS dest_zone FIELDALIAS-estreamer_dvc = sensor AS dvc FIELDALIAS-estreamer_file_hash = sha256 AS file_hash FIELDALIAS-estreamer_first_pkt_sec_1 = connection_second AS first_pkt_sec FIELDALIAS-estreamer_first_pkt_sec_2 = connection_sec AS first_pkt_sec FIELDALIAS-estreamer_flow_id = connection_id AS flow_id FIELDALIAS-estreamer_instance_id = connection_instance_id AS instance_id FIELDALIAS-estreamer_intrusion_signature = msg AS signature FIELDALIAS-estreamer_malware_signature = detection AS signature FIELDALIAS-estreamer_packets_in = src_pkts AS packets_in FIELDALIAS-estreamer_packets_out = dest_pkts AS packets_out FIELDALIAS-estreamer_rule = fw_rule AS rule FIELDALIAS-estreamer_severity = priority AS severity FIELDALIAS-estreamer_src = src_ip AS src FIELDALIAS-estreamer_src_interface = iface_ingress AS src_interface FIELDALIAS-estreamer_src_zone = sec_zone_ingress AS src_zone FIELDALIAS-estreamer_tcp_flag = tcp_flags AS tcp_flag FIELDALIAS-estreamer_url = uri AS url FIELDALIAS-estreamer_vlan = vlan_id AS vlan HEADER_MODE = LEARN_MODEL = true LEARN_SOURCETYPE = true LINE_BREAKER_LOOKBEHIND = 100 LOOKUP-estreamer_file_action = file_actions file_action OUTPUT action LOOKUP-estreamer_fw_action = fw_actions fw_rule_action OUTPUT action LOOKUP-estreamer_severities = severities impact,priority OUTPUT severity LOOKUP-estreamer_sources = sources source OUTPUT vendor, product, ids_type LOOKUP-estreamer_transport = ip_protos ip_proto OUTPUT transport MATCH_LIMIT = 100000 MAX_DAYS_AGO = 2000 MAX_DAYS_HENCE = 2 MAX_DIFF_SECS_AGO = 3600 MAX_DIFF_SECS_HENCE = 604800 MAX_EVENTS = 256 MAX_TIMESTAMP_LOOKAHEAD = 128 MUST_BREAK_AFTER = MUST_NOT_BREAK_AFTER = MUST_NOT_BREAK_BEFORE = SEGMENTATION = indexing SEGMENTATION-all = full SEGMENTATION-inner = inner SEGMENTATION-outer = outer SEGMENTATION-raw = none SEGMENTATION-standard = standard SHOULD_LINEMERGE = false TIME_PREFIX = event_sec= TRANSFORMS = TRUNCATE = 0 detect_trailing_nulls = false maxDist = 100 priority = sourcetype = [cisco:estreamer:log] ADD_EXTRA_TIME_FIELDS = True ANNOTATE_PUNCT = True AUTO_KV_JSON = true BREAK_ONLY_BEFORE = BREAK_ONLY_BEFORE_DATE = True CHARSET = UTF-8 DATETIME_CONFIG = /etc/datetime.xml DEPTH_LIMIT = 1000 EXTRACT-encore_log_fields = ^(?P<timestamp>\d+\-\d+\-\d+\s+\d+:\d+:\d+,\d+)\s+(?P<class>[^ ]+)\s+(?P<severity>\w+)\s+(?P<message>.+) HEADER_MODE = LEARN_MODEL = true LEARN_SOURCETYPE = true LINE_BREAKER_LOOKBEHIND = 100 MATCH_LIMIT = 100000 MAX_DAYS_AGO = 2000 MAX_DAYS_HENCE = 2 MAX_DIFF_SECS_AGO = 3600 MAX_DIFF_SECS_HENCE = 604800 MAX_EVENTS = 256 MAX_TIMESTAMP_LOOKAHEAD = 128 MUST_BREAK_AFTER = MUST_NOT_BREAK_AFTER = MUST_NOT_BREAK_BEFORE = SEGMENTATION = indexing SEGMENTATION-all = full SEGMENTATION-inner = inner SEGMENTATION-outer = outer SEGMENTATION-raw = none SEGMENTATION-standard = standard SHOULD_LINEMERGE = True TRANSFORMS = TRUNCATE = 10000 detect_trailing_nulls = false maxDist = 100 priority = sourcetype = And here is my inputs for the app (configured as per the directions): [monitor:///opt/splunk/etc/apps/TA-eStreamer/data] _rcvbuf = 1572864 crcSalt = <SOURCE> disabled = false host = firesight index = estreamer source = encore sourcetype = cisco:estreamer:data [script:///opt/splunk/etc/apps/TA-eStreamer/bin/splencore.sh clean] _rcvbuf = 1572864 disabled = 0 host = bbmosplunk index = default interval = 900 source = encore sourcetype = cisco:estreamer:clean start_by_shell = false [script:///opt/splunk/etc/apps/TA-eStreamer/bin/splencore.sh start] _rcvbuf = 1572864 disabled = 0 host = bbmosplunk index = default interval = 120 source = encore sourcetype = cisco:estreamer:log start_by_shell = false [script:///opt/splunk/etc/apps/TA-eStreamer/bin/splencore.sh status] _rcvbuf = 1572864 disabled = 0 host = bbmosplunk index = default interval = 30 source = encore sourcetype = cisco:estreamer:status start_by_shell = false ... View more

Thanks for info araitz, that is what I figured I needed to do, now to get that setup. As for the sonicwall I have it setup and logging to splunk over syslog and I did see your post about field extractions on it for 3.x. I'll see if I can get it updated to 4.x and post back about it. ... View more