I'm trying to validate if we have a large amount of data duplication. Whenever I run the dedup _raw command the number of results is typically halved but based on the onprem set up I don't see how the data could be ingested twice (our typical data flow syslog to a HF that is then ingested by an indexer cluster). If I run index=sample|transaction _raw I only get a small number of results where the raw results are combined.  How can I confirm how big my data duplication problem might be? ... View more

I've been able to deploy universal forwarders to dozens of Windows servers that run IIS logs. I have created a dedicated index and I have pushed an app (used to be Splunk supported, they have since moved to a different app package) to said forwarders. The forwarders are set to send the data to our indexer cluster. To cover my bases for the different versions I have included several different monitor stanzas in the inputs.conf file:     [monitor://C:\inetpub\logs\...\W3S*\*.log] disabled = false sourcetype = ms:iis:auto index=iis [monitor://C:\inetpub\logs\*\W3S*\*.log] disabled = false sourcetype = ms:iis:auto index=iis [monitor://C:\Program Files\Microsoft\Exchange Server\V*\Logging\Ews] disabled = false sourcetype = ms:iis:auto index=iis     When deployed to the dozens of servers, I'm not seeing any data come back up or even any path watches coming back when searching the logs coming back from the universal forwarders. As a test I have added several files to a dedicated server and kept playing around with the monitor stanzas with no luck. When opening the inputs.conf locally on that server in notepad, the text looked merged so I added some spaces and line breaks. Restarted the service, I can path watches added but still nothing coming in. Even when specifying a path to a file, nothing comes in:     [monitor://C:\Test\logs\LogFiles\W3SVC1\u_ex221010.log] disabled = false sourcetype = ms:iis:auto index=iis     For something that seems so simple, where am I going wrong? ... View more

In our Splunk environment, we currently ingest Azure AD logs and we have three different sourcetypes: azure:aad:signin azure:aad:audit azure:aad:user There no missing events and the ingested data is very rich. However, I don't see any way within the Splunk ingested Azure signin data to to filter by authentication method (Single-factor vs multi-factor). This is something that can be done via Azure Active Directory, Monitoring, Sign-in logs but I do not see any reference to it in my Splunk data (I do see a lot of conditional access enforcement and the other primary fields, but not any of the secondary fields that could be used for filtering in Azure):   ... View more

We use Splunk dashboards with searches that refresh on regular intervals as screens to monitor in an operations center. Does anyone have experience with having new results (i.e. rows) light up, flash, do something else eye catching to grab attention? ... View more

I was looking to implement a search described in this article: threathunting-spl/Detecting_Beaconing.md at master · inodee/threathunting-spl · GitHub TLDR: The above link shows a search that allows for a data source like firewall connections data to be used to identify connections that are suspiciously uniform in terms of interval. It uses both streamstats and eventstats to calculate the standard deviation for the time intervals for connections between unique combinations of src and dst IPs.  The issue is that the data I would be using is enormous - I'm looking to do the above search on 24 hours worth of data but it fails due to memory limits. What I have in place so far, I do have an accelerated data model for the filtered firewall data, but I dont know how to combine tstats and the search in the above link. Summary indexing wouldnt work since I would still like the stats calculations over the larger time frame (i.e. doing the search every hour for the last hour worth of data might miss whether a connection truly does have a low standard deviation).  Has anyone successfully combined tstats with streamevents/eventstats or built a search that works around just how resource intensive the search is? ... View more

Our DNS logs are sent via syslog to a HF through an Epilog agent. The EpiLog agent reads the dns log file line by line and each line is sent as a separate event to the HF,  looking something like this: Dec 24 04:05:11 192.####### MSDNSLog 0 12/24/2021 12:02:06 AM 04B4 PACKET 000000####### UDP Rcv 142.####### 3f94 R Q [8281 DR SERVFAIL] PTR (2)87in-addr(4)arpa(0) Dec 24 04:05:11 192.####### MSDNSLog 0 UDP response info at 000000EE456861F0 Dec 24 04:05:11 192.####### MSDNSLog 0 Socket = 1244 Dec 24 04:05:11 192.####### MSDNSLog 0 Remote addr 142.1#######, port 53 Dec 24 04:05:11 192.#######MSDNSLog 0 Time Query=1220313, Queued=0, Expire=0 Dec 24 04:05:11 192.####### MSDNSLog 0 Buf length = 0x0fa0 (4000) Dec 24 04:05:11 192.####### MSDNSLog 0 Msg length = 0x0037 (55) Dec 24 04:05:11 192.####### MSDNSLog 0 Message: Dec 24 04:05:11 192.####### MSDNSLog 0 XID 0x3f94 Dec 24 04:05:11 192.####### MSDNSLog 0 Flags 0x8182 Dec 24 04:05:11 192.####### MSDNSLog 0 QR 1 (RESPONSE) Dec 24 04:05:11 192.####### MSDNSLog 0 OPCODE 0 (QUERY) Dec 24 04:05:11 192.####### MSDNSLog 0 AA 0 Dec 24 04:05:11 192.####### MSDNSLog 0 TC 0 Dec 24 04:05:11 192.####### MSDNSLog 0 RD 1 Dec 24 04:05:11 192.####### MSDNSLog 0 RA 1 Dec 24 04:05:11 192.####### MSDNSLog 0 Z 0 Dec 24 04:05:11 192.####### MSDNSLog 0 CD 0 Dec 24 04:05:11 192.####### MSDNSLog 0 AD 0 Dec 24 04:05:11 192.#######MSDNSLog 0 RCODE 2 (SERVFAIL) Dec 24 04:05:11 192.####### MSDNSLog 0 QCOUNT 1 Dec 24 04:05:11 192.1####### MSDNSLog 0 ACOUNT 0 Dec 24 04:05:11 192.#######  MSDNSLog 0 NSCOUNT 0 Dec 24 04:05:11 192.1###### MSDNSLog 0 ARCOUNT 1 Dec 24 04:05:11 192.1##### MSDNSLog 0 QUESTION SECTION: Dec 24 04:05:11 192.1##### MSDNSLog 0 Offset = 0x000c, RR count = 0 So originally each of those lines was indexed as a separate event in Splunk. I played around with the props.conf file for that specific sourcetype and set  the parameters as follows: SHOULD_LINEMERGE=TRUE TIME_PREFIX to match Dec 24 04:05:11 192.###### MSDNSLog 0 TIME_FORMAT=%m/%d/%Y %l:%M:%S %p BREAK_ONLY_BEFORE=PACKET (Every event starts with a line that contains packet) LINE_BREAKER = ([\r\n]+) TRUNCATE=0 MAX_EVENTS=500000 (I've seen some  events be very long) MAX_TIMESTAMP_LOOKAHEAD=100 SEDCMD-null = regex to get rid of  Dec 24 04:05:11 192.####### MSDNSLog 0 at the beginning of every line Based on my understanding (and I played around with Add Data on a searchhead and the above parameters, where it works), the following should happen: The lines are broken on each new line, then they are merged, with each new event being formed when a line has PACKET in it, timestamp is extracted and then the MSDNSLOG stuff at the beginning of each line is removed.  However, I'm not seeing the timestamp being extracted properly and some (not all)of the DNS events get split like below into separate events: What could I be missing to get all events merged correctly? Please keep in mind that using sysmon/network tap/stream is not an option at the moment so I stuck with trying to the data ingested properly using the conf files.   ... View more