Hello, when using syslog-ng with splunk you can control the index and sourcetype etc in the app inputs.conf that is pushed to your forwarder. of course this implies you are using a splunk enterprise on prem deployment server to push apps to your endpoints forwarders. for example firewall sends its log to syslog-ng syslog-ng.conf has filters configured for each type of device sending logs, it checks through those filters until it finds a match for the log received from the firewall. lets say it is using subnet to identify this is a firewall. once it matches the filter it then checks the log rule which tells it which destination rule to use to place the log in the folder structure and naming convention that has been configured in the destination rule for this device type lets say it puts the log in "/home/syslog/logs/firewall/$HOST/$YEAR-$MONTH-$DAY-firewall.log" you will note that in the path that is configured in the destination rule for this log the hostname will be the 5th folder in the path so in your inputs.conf file for whichever app you use for this devicetypes logs you would need to tell it that the 5th position is where the hostname is. splunk will then be able to find the logs using that name with the host=host name example in my example the syslog_firewall_inputs\local\ inputs.conf file would look like this Firewall Logs sent to syslog-ng [monitor:///home/syslog/logs/firewall//.log] host_segment = 5 sourcetype = fgt_log index = firewall disabled = false note that the sourcetype was one we created ourselves for our environment so be sure all settings are for your own environment. as you can see it is in this file where you can specify which index and which sourcetype splunk UF will associate with the log. I hope this is helpful. your description was not 100% clear so I am hoping this is what you need ... View more

I think you are looking for something like this : index=<your_index> | stats latest(_time) as latest by source host | eval "Latest Log Time"=strftime(latest,"%I:%M:%S %p") | fields- latest Above will be much slower query so you can also use the query given by @FrankVl for faster approach. In addition to that | tstats latest(_time) as latest where index=<your_index> by source,host | eval "Latest Log Time"=strftime(latest,"%I:%M:%S %p") | fields- latest You need to put <your_index> as your index name . let me know if this helps! ... View more

Thank you. I understand it, so far, I was on the safe side, use the whole add-on everywhere, I did not feel the Force in me to select which file needed, which doesn't. (And guide says Install the add-on...) Regards, István ... View more

Well, the highlight command doesn't support regular expressions or wildcards or anything, so that will be difficult. If the number of possible message texts is limited, you could create a macro that contains a highlight command with all possible message strings behind it. ... View more

That's because when no modifications have been made to it yet, the file is called splunk-launch.conf.default You can copy it to a file called splunk-launch.conf and then make the required modifications. ... View more

If the syslog server goes down, then of course the situation is the same as when the indexer goes down. Point is: indexers will require a restart every once in a while (deploying new config, updating splunk etc.). And that can take a few minutes, causing immediate data loss (especially when sending syslog over UDP). A separate syslog server can use a lightweight, high performance syslog daemon (rsyslog, syslog-ng) to receive the data and write it to disk. Then you can for instance install a Universal Forwarder on that same box to ingest the data into splunk. That UF can then also auto load balance across all your indexers (in case you have a larger setup with multiple (clustered) indexers). If you send it all directly to an indexer, that indexer will have to deal with all the search load for that data. The syslog daemon also may require a restart for config changes, but typically syslog daemons restart in a matter of seconds. For even further improved high availability (and data distribution) you could use a pool of syslog servers and put a load balancer in front of them. Another massive advantage of using a syslog daemon to first receive the data is that these daemons offer powerful (yet fairly simple) configuration options to recognize where data came from and distinguish different types of syslog data and then write that to separate files / folders, which makes it much easier to write splunk input configurations and determine the right host and sourcetype values to set for instance. ... View more

it looks like you have broken (incomplete events)? You'd need host extraction transforms for each different format (so basically for each proper sourcetype as per the original TA I would say). ... View more

Is this your search head props.conf? You should paste the indexer props If this is your indexer props, then you need to set your base configs which tells Splunk how to identify the timestamp and linebreak ... View more

eventtype is shared globally and dashboard sits in search app, still I am getting same error. eventtype is created in different app and dashboard in different. ... View more

Look at the _internal index for the metrics.log from your indexers, specifically for group=tcpin_connections. And then filter that for connections from your HFs. Alternatively: do the same for the metrics.log from your HFs and look at the tcpout_connections metrics. ... View more

The example I gave is indeed only the base of your search. As I already mentioned: from this you need to create 2 alert searches: one that filters for previous below 90 and current >90. And another alert search that filters for previous >90 and current <90. PS: you might want to convert your "answer" into a comment to my answer 🙂 ... View more

Hi ddrillic, Even if the master goes down the forwarder will continue to send data to the indexer that it was sending earlier. There would be no data loss unless the indexer to which the forwarder was sending data goes down. After the master is restored again it will continue sending data to all indexers. You can refer this doc: https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Whathappenswhenamasternodegoesdown ... View more

I used the appendcols command which did not work. ... View more

Try: REGEX = Title[\r\n]\s*([^\r\n]*)[\s\S]*?Rule[\r\n]*\s*([^\r\n]*)([\r\n]*\s*Ident[\r\n]*\s*([^\r\n]*)[\s\S]*?Ident[\r\n]*\s*([^\r\n]*))?[\s\S]*?Result[\r\n]*\s*([^\r\n]*) FORMAT = Title::$1 Rule::$2 CCE::$4 DISA::$5 Result::$6 It's fairly efficient. ... View more

@micahkemp They are both in the $SPLUNK_HOME/etc/apps//local folder. It's a single server instance. ... View more

02-07-2018 07:45:56.718 -0500 WARN FileClassifierManager - The file '/var/log/openvpn.log' is invalid. Reason: binary 02-07-2018 07:45:56.718 -0500 INFO TailReader - Ignoring file '/var/log/openvpn.log' due to: binary Well there we go, never occurred to check logs on the forwarder itself. So now I have a new mystery. The file is definitely not a binary. [2.4.2-RELEASE][root@pvlpfs01.local]/var/log: file openvpn.log openvpn.log: ASCII text [2.4.2-RELEASE][root@pvlpfs01.local]/var/log: hexdump -C openvpn.log | less 00000000 4a 61 6e 20 32 34 20 32 33 3a 33 32 3a 32 31 20 |Jan 24 23:32:21 | 00000010 70 76 6c 70 66 73 30 31 20 6f 70 65 6e 76 70 6e |pvlpfs01 openvpn| 00000020 5b 32 32 38 38 31 5d 3a 20 65 76 65 6e 74 5f 77 |[22881]: event_w| the /// doesn't hurt anything, just defines the path all the way from root. That was added to see if it made a difference with the problem I was having. In any case im going to add a stanza in props.conf [openvpn] NO_BINARY_CHECK = true ... View more

To better utilize server resources, you could also look into enabling multiple pipelines on a single splunk instance. Or simply replacing 16 big servers with 32 smaller servers or something. But that all depends a bit on how flexible you are in replacing servers (for virtuals that might be easier then when it is running on bare metal). ... View more

@493669 Thanks for your analysis. It was helpful. I'll look into it. ... View more

try this: | savedsearch Cycle_11 | append [| savedsearch Cycle_10]|eval "Field_1"=coalesce('Field_1(CYCLE 11)','Field_1(CYCLE 10)')|stats values(*) as * BY Field_1|table ID Name * ... View more

Doesn't your organization have some process for getting properly signed certs, by a CA that is trusted by all the user's OS/browser? ... View more

| metadata type=hosts will give you much more than just your forwarders. It will give every known value of the host field, so that could also be all kinds of network devices that send over syslog or something, where the original device's name ends up in the host field. ... View more

You have stored the props.conf/transforms.conf under <sample_app_1>/default or <sample_app_1>/local ? these are the search paths of splunk where to look for these files. ... View more

I'd expect that to be visible in _audit index. Just try creating one and changing / deleting it, to see what it looks like in _audit, so you can build some alerts on that. Edit: Had a quick look at it and cannot find obvious traces in _audit. _internal does show some clues though, events with method=POST for changes of reports/alerts and events with method=DELETE for deletes of alerts/reports. ... View more

hey @zacksoft You can use strftime(X,Y) to convert in a specified time format in Y and strptime(X,Y) to convert the same in epoch time. have a look at this doc http://docs.splunk.com/Documentation/Splunk/7.0.2/SearchReference/DateandTimeFunctions#strftime.28X.2CY.29 For time format, you can have a look at this doc https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Commontimeformatvariables Try this run anywhere search for more: | gentimes start=-1 end=20 | eval Endtime=strftime(endtime,"%d/%m/%Y %H:%M:%S"),Starttime=strftime(starttime,"%d/%m/%Y %H:%M:%S"),Starthuman=strptime(starthuman,"%a %b %d %H:%M:%S %Y"),Endhuman=strptime(endhuman,"%a %b %d %H:%M:%S %Y") | table starttime Starttime endtime Endtime starthuman Starthuman endhuman Endhuman let me know if this helps! ... View more