I am working with some WAF logs that provide a correlation from sourceIP to city_name, country_name, latitude, and longitude, but not state information. I am currently only looking at the United States. I need to somehow enrich the data to help filter user logins by time and distance between last login or if there are parallel logins going on. Please advise where I might start. Thank you ... View more

I created this PART 2 as the previous thread is getting long. Recap: I am trying to monitor login behavior to an online application using WAF logs. The userID may be associated with multiple sourceIP(s). I use the following query to create a reference point, historic_login_list.csv. (I run this for last 90 days but not include current day) index=waf sourcetype=waf_logs " key words" | stats count by userID sourceIP GeoLoc | dedup userID sourceIP| outputlookup append=f historic_login_list.csv I use the following query to compare results to the reference list. index=waf sourcetype=waf_logs " key words" | fillnull value=NULL userID | stats count by userID sourceIP GeoLoc _time| dedup]userID sourceIP| search NOT [|inputlookup historic_login_list.csv |fields userID sourceIP GeoLoc] I use a fill null in the comparative query as it picks errors, and I added _time for investigative purposes. I have scrubbed the historic list csv to discard data that is unusual or not a normal reference point. Does anyone know of a better way to do this??? and get the results that I am after... I appreciate any suggestions to improve or make more efficient. Thank you for your review. ... View more

So I am using this stanza in my .../local props.conf [mscs:storage:blob] SHOULD_LINEMERGE = true LINE_BREAKER: (,[\r\n]+\s+){ TRUNCATE = 0 KV_MODE = json This works fine to parse my events like this "time": "2018-07-09T16:14:21.3832528Z", "resourceId": "blah", "operationName": "blah blah", "category": "blah", "resultType": "blah", "resultSignature": "blah", "durationMs": 1234, "callerIpAddress": "x.x.x.x", "correlationId": "x-x-x-x-x", "identity": {"blah blah":{"blah"}}, "level": "blah", "location": "blah", "properties": {"statusCode":"OK","serviceRequestId":"x-x-x"} } ] } OR "time": "2018-07-09T16:14:21.3832528Z", "resourceId": "blah", "operationName": "blah blah", "category": "blah", "resultType": "blah", "resultSignature": "blah", "durationMs": 1234, "callerIpAddress": "x.x.x.x", "correlationId": "x-x-x-x-x", "identity": {"blah blah":{"blah"}}, "level": "blah", "location": "blah", "properties": {"statusCode":"OK","serviceRequestId":"x-x-x"} } , { Does anyone know how to drop the brackets and commas? Thank you ... View more

I have install the Splunk_TA_microsoft-cloudservices APP on a heavy forwarder which routes to multiple indexers in a "distributed search peer" configuration. The app is working and I can find the events in my search head for example, index=azure sourcetype = mscs:storage:blob The files that are sent to the blob storage are in json format. The problem is that Splunk is not parsing it correctly. I believe I need to add SHOULD_LINEMERGE somewhere . I was thinking that I need to add to "props.conf" on the fwdr in /opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/local When I look in .../default I can see a props.conf but there is no stanza for sourcetype = mscs:storage:blob . Do I create a props.conf in .../local and enter a stanza for sourcetype = mscs:storage:blob and add SHOULD_LINEMERGE = true? Ideally I want the breaks to occur on the timestamps. Please advise. Thank you ... View more