I'm using heavy forwarder to take logs in from a cloud ESA appliance. The logs sending over every 5 minutes via scp (deleting old files every 2 hours after modtime stops) work fine, line by line and time stamps all good. For some reason splunk is randomly ingesting some events by grabbing from a random place in the file, giving it a time stamp, and calling it an event. For example, I most commonly get 1 - 3 characters, like ile, or id. Sometimes I get the middle to the end of an event. I don't understand why it's doing this, there is no line breaking (it's disabled) and I have enabled crcsalt by source. I'm using splunk 6.4.6 on the indexers, and 6.6.1 on the heavy forwarder (started with 6.4.6). If I upload the file to my dev box, it's fine; for some reason the monitor feature of splunk is having issues. I also ingest WSA (but with universal forwarder) over scp, and I don't have these issues. I have put the ESA app on the heavy forwader and search heads. Also tried just indexers and search heads, only input on heavy forwarder. None this changed anything. This is all virtualized Linux, Ubuntu 64bit LTS servers. props.conf: CHARSET = utf-8 MAX_TIMESTAMP_LOOKAHEAD = 24 NO_BINARY_CHECK = true SHOULD_LINEMERGE = false TRUNCATE = 250000 TIME_PREFIX = \w{3}\s TRANSFORMS-drop=esa_header_drop transforms.conf [esa_header_drop] REGEX=^.*Info:\s(Begin\sLogfile|Logfile\srolled\sover|End\sLogfile|Version\:\s|Time\soffset\sfrom\sUTC\:).* DEST_KEY=queue FORMAT=nullQueue I log amp, mail, gui, auth, http, via scp. They all have the props above, individually configured. Sample event (good and bad) from one, full auth file: le Tue Jun 6 15:12:22 2017 Info: A publickey authentication attempt by the user ***** from 0.0.0.0 failed using an SSH connection. Tue Jun 6 15:10:40 2017 Info: logout:10.0.0.1 user:- session:blahgfregre Tue Jun 6 15:09:28 2017 Info: logout:10.0.0.25 user:- blaggj4iogjio3 Tue Jun 6 15:09:20 2017 Info: A publickey authentication attempt by the user ***** from 0.0.0.0 failed using an SSH connection. Below is a screenshot of search finding one event, that gets mangled and duplicated (using the suggestion break before regex). Checking the file, there is only one event on one line, no idea why splunk is doing this. ... View more

I'd like to create a field at search time, we'll call it internal_ip. I can already filter by CIDR block and get the results I want, but I need more depth than that for a lookup I'd like to do. The problem is with my IDS logs, depending on the directional flow, the internal IP can be either src_ip or dest_ip (this particular search will never return both internal IP addresses). I'd like a query to search both src_ip and dest_ip fields for a specific private network IP block, and assign that value to my new field. How do I accomplish this? ... View more

I'd like to sanitize host names during search time in Splunk (IDS alerts), so users don't receive a hyperlink to the bad sites and the alerts don't get snagged by the mail filter. I'd like to keep the inline table results of the alerts for mobile users and general convenience. For example, www.badsite.com is automatically hyperlinked in e-mail clients, so I'd like to prepend hxxp:// to the field so the sent alert message doesn't get hyperlink or dropped. By specifying a bogus protocol both fat mail client and mail scanner will pass the inline table without issue. I found something about appending in the community, but that just shows a blank field during search for me (even when I assign httphostname without a period). eval http.hostname=http.hostname."hxxp://" | In theory, I'd like this to prepend and actually work: eval http.hostname="hxxp://".http.hostname | I'm using Splunk 6.3.2 in a clustered Linux environment, any help is appreciated. ... View more

I have a complex distributed environment, I'll try to stick to the root of my concerns. Basically I have site 1 and site 2. Site 2 just forwards directly to site 1 now via a forwarder. However I would like to have site 2 forward to a site 2 indexer. The problem I see, I will be forwarding site 2 index to site 1 index (I won't go into details, it is what needs to be done). What happens if site 2 index sends the exact same data as site 1 index is already getting? I imagine there will be a slight overlap with some of the logs during this transition, however I worry about the implications of this duplicity. ... View more

I have two indexes for ids (suricata) and proxy (Cisco WSA), I'd like to correlate when splunk finds an IDS alert and pull info from the proxy logs which include the username and internal IP. Looking over previous answers, this is what I came up with to start (just a base, my alert syntax is complex but I know what I need for that component). I was hoping for a singular event, and being able to format a table for the splunk e-mail alert. However, when I do transaction, the events stay with their respective indexes. So, below would show one of the two tabled fields and one always blank. index="ironport" OR index="ids" sourcetype="cisco:wsa:squid" OR sourcetype=suri| rename http.hostname AS cs_url_host | transaction cs_url_host maxpause=30s | table cs_url_host,cs_user The next step is the time issue. Proxy logs are behind x minutes, whereas IDS is almost immediate. I'd need to wait a few minutes before joining the indexes. I've looked at join, but I haven't figured out how to take the matching hostname IDS alert to the hostname of the proxy logs. Join doesn't seem to work very well either (often giving me no results). Haven't gotten append to work, but open to it. ... View more