This answer highly depends on the XML configuration you apply to Sysmon. With our configuration now, we're seeing an average of 20-25GB/day across ~6,000 endpoints. So you're looking at maybe 4-10GB /day for 1000 hosts. Sometimes our ingest will spike (like during a patch day) and we'll hit 60-70GB a day during peaks. My suggestion is to start with a minimal configuration instead of enabling everything from the beginning. Add events onto it and be prepared to blacklist or remove a configuration after you've deployed it. Sysmon can be really finicky until you get it to a point where everyone is comfortable with the type of data you're getting vs the amount of licensing being consumed. Also be aware that depending on the type of events you bring in, you could end up pulling in plaintext passwords from scripts or command line switches into Splunk. You can mask these with a SEDCMD in props.conf on the forwarder side. ... View more

I think the "ep" means "endpoint". The way I'm understanding it is that the "ep*" indexes are for services on the endpoint that could be from Windows, but are not necessarily from Windows itself. So you could have multiple threat intelligence components (sysmon, Bit9, etc) on the endpoint feeding your epintel index, but these are not Windows Security logs. I believe the same can stand for "epav" -- these logs could come from a built-in Windows Service like Defender, but you could also populate it with another source like SEP or McAfee (or all three if you hate yourself). Meaning you can have multiple sources outside of Windows reporting to these indexes, and they can be for the same overall task (av or intel). In our environment at least, these types of sources have much shorter retention periods, and I care less about the longevity of the data than I do say the OS Security Events like Login/Logoff/Membership changes, etc. So keeping the high-volume (and expensive) endpoint (ep) indexes separate from the more important events allows you to retain the security auditing events in Splunk for longer than you would normally want the endpoint logs. Then, searching becomes a little easier as you can just specify "ep*" or "os*" for those types of logs without bringing in all OS logging in one search. ... View more

What (if any) errors are there in mongod.log? Have you changed a SSL cert recently? ... View more

That's pretty strange. However, the default path for kvstore is $SPLUNK_DB/kvstore. Perhaps you can verify a couple of things. Where the KV Store is configured to reside: /opt/splunk/bin/splunk btool server list | grep kvstore If it returns dbPath=$SPLUNK_DB/kvstore, then verify your $SPLUNK_DB path is truly var/lib/splunk: cat /opt/splunk/etc/splunk-launch.conf | grep SPLUNK_DB Hopefully this points you in the right direction. I am on 7.0.1 and have not modified my default SPLUNK_DB path, and I am seeing /opt/splunk/var/lib/splunk/kvstore/ on both Windows and Linux installations. You aren't on Splunk Cloud by chance, are you? ... View more

Then I suppose the only thing left to do is to validate you're receiving DHCP logs on that server :). Take a look at that server's Event Viewer and see if it is populating the DHCP log. ... View more

Just to clarify -- in your indexes.conf you have a windows index and a wineventlog index specified? To me, it sounds like the forwarder is reporting to _internal fine as you mention seeing data in that index, but it sounds like there's nothing reporting to windows/wineventlog because that index is not correctly configured. I would consider setting a value for lastChanceIndex in indexes.conf (maybe use 'main', or create a new index for this). That way if you have data that's trying to write to an index that doesn't exist, you can check your lastchance index for the data to validate. ... View more

Personally I haven't seen much of an impact whether it's 4 or 160. I just started exceeding my page file size which caused the KV Store to crash so that's why I started off in the memory department. I feel like that particular gauge is a little misleading with the values it's configured to report as "bad". ... View more

Do you have an index named "windows" or "wineventlog"? ... View more

I brought this down by adding more memory to my Search Head. I was seeing ~160 page faults per op and now i'm down to about 4 (32GB->64GB RAM). I agree there isn't a lot of information surrounding this topic. It appears the kv_store attempts to write to the page file on disk if it can't utilize memory, so theoretically either increasing your page file/swap size or RAM should help bring this value down. ... View more

Hi Doug! Yes, the 6.2.2.1-73 patch resolved the FIPS issues we were having, and Sourcefire IDS events started coming in right away when it was applied. I posted an answer a bit ago and noted this patch resolves the issue. Thanks for checking in! ... View more

Yeah we've read the Ops guide and our dependencies appear to be in order. Our FMC admin spoke with Cisco and it sounds like there isn't a supported way to get eStreamer to work with FMC when FIPS is enabled on FMC. One of my other users mentioned that FMC is attempting to use Blowfish during the connection after seeing this in the debug log (which is not a FIPS-approved cipher). Sounds like it would be a pain to disable FIPS on FMC to validate Support's statement so we are not going that route at the moment. ... View more

So we patched to 6.2.0.2 and we're experiencing the same message when I run ./splencore test. But, I did notice that if I just ran ./splencore start that I get some more info as far as errors are concerned: [splunk@server-01 bin]$ ./splencore.sh start 2017-08-28 08:32:49,378 estreamer.client INFO eNcore version: 3.0.0 2017-08-28 08:32:49,379 estreamer.client INFO Python version: 2.7.5 (default, May 3 2017, 07:55:04) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-14)] 2017-08-28 08:32:49,379 estreamer.client INFO Platform version: Linux-3.10.0-693.1.1.el7.x86_64-x86_64-with-redhat-7.4-Maipo 2017-08-28 08:32:49,379 estreamer.client INFO Starting client (pid=9331). 2017-08-28 08:32:49,379 estreamer.client INFO Sha256: 8dd9941c993687f478a28a236567b49763d6bf30e23a43ec125a3391495201c7 2017-08-28 08:32:49,379 Diagnostics INFO Check certificate 2017-08-28 08:32:49,379 Diagnostics INFO Creating connection 2017-08-28 08:32:49,379 estreamer.connection INFO Connecting to fmc:8302 2017-08-28 08:32:49,379 estreamer.connection INFO Using TLS v1.2 2017-08-28 08:32:49,624 Diagnostics INFO Creating request message 2017-08-28 08:32:49,624 Diagnostics INFO Request message=0001000200000008ffffffff48900061 2017-08-28 08:32:49,625 Diagnostics INFO Sending request message 2017-08-28 08:32:49,626 Diagnostics INFO Receiving response message 2017-08-28 08:32:49,626 Diagnostics ERROR The FMC eStreamer server has closed the connection. There are a number of possible causes which may show above in the error log.\n\nIf you see no errors then this could be that:\n * the server is shutting down\n * there has been a client authentication failure (please check that your outbound IP address matches that associated with your certificate - note that if your device is subject to NAT then the certificate IP must match the upstream NAT IP)\n * there is a problem with the server. If you are running FMC v6.0, you may need to install "Sourcefire 3D Defense Center S3 Hotfix AZ 6.1.0.3-1"\n 2017-08-28 08:32:49,626 estreamer.client ERROR ConnectionClosedException: Connection closed 2017-08-28 08:32:49,626 estreamer.client INFO Stopping... 2017-08-28 08:32:49,627 estreamer.monitor INFO Stopping Monitor. 2017-08-28 08:32:49,627 estreamer.client INFO Goodbye 2017-08-28 08:32:49,648 Service ERROR OSError: \nTraceback (most recent call last):\n File "./estreamer/service.py", line 179, in main\n self.start( reprocessPkcs12 = args.pkcs12 )\n File "./estreamer/service.py", line 148, in start\n self._posix()\n File "./estreamer/service.py", line 90, in _posix\n self._loop()\n File "./estreamer/service.py", line 67, in _loop\n if not condition.isTrue():\n File "/opt/splunk/etc/apps/TA-eStreamer/bin/encore/estreamer/condition/splunk.py", line 33, in isTrue\n 'status' ] )\n File "/usr/lib64/python2.7/subprocess.py", line 568, in check_output\n process = Popen(stdout=PIPE, *popenargs, **kwargs)\n File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__\n errread, errwrite)\n File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child\n raise child_exception\nOSError: [Errno 2] No such file or directory\n Support initially thought it was FIPS causing the issue, but we ended up not having FIPS enabled on the FMC side. We are running FIPS_MODE on the Splunk Heavy Forwarder that we are trying to get this app working on. ... View more

Thanks! Will report back after patching 🙂 ... View more

Thanks Doug! We are running FMC 6.2.0.1 and are running into an issue: When testing eNcore: [splunk@server-01 bin]$ ./splencore.sh test 2017-08-03T08:38:19.199042 Diagnostics INFO Checking that configFilepath (estreamer.conf) exists 2017-08-03 08:38:19,205 Diagnostics INFO Check certificate 2017-08-03 08:38:19,206 Diagnostics INFO Creating connection 2017-08-03 08:38:19,206 estreamer.connection INFO Connecting to fmc:8302 2017-08-03 08:38:19,206 estreamer.connection INFO Using TLS v1.2 2017-08-03 08:38:19,392 Diagnostics INFO Creating request message 2017-08-03 08:38:19,392 Diagnostics INFO Request message=0001000200000008ffffffff48900061 2017-08-03 08:38:19,392 Diagnostics INFO Sending request message 2017-08-03 08:38:19,393 Diagnostics INFO Receiving response message 2017-08-03 08:38:19,402 Diagnostics ERROR The FMC eStreamer server has closed the connection. There are a number of possible causes which may show above in the error log. And from Defense Center we're seeing this: Aug 03 2017 08:38:20 fmc SF-IMS[4760]: [4760] Event Streamer:ConnectionHandler [INFO] Removed host entry for pid: 22071 Aug 03 2017 08:38:20 fmc SF-IMS[4760]: [4760] Event Streamer:ConnectionHandler [INFO] Child with pid 22071 exited with status 134 Aug 03 2017 08:38:19 fmc SF-IMS[22071]: [22071] EventStreamer child(10.0.0.1):ConnectionHandler [INFO] Matched Certificate CN:10.0.0.1 to 10.0.0.1 (IPv4) Aug 03 2017 08:38:19 fmc SF-IMS[22071]: [22071] EventStreamer child(10.0.0.1):ConnectionHandler [INFO] Resolved CN 10.0.0.1 to 10.0.0.1 Aug 03 2017 08:38:19 fmc SF-IMS[4760]: [4760] Event Streamer:ConnectionHandler [INFO] Added 10.0.0.1(22071) to host table Aug 03 2017 08:38:19 fmc SF-IMS[4760]: [4760] Event Streamer:ConnectionHandler [INFO] Added 10.0.0.1 to host table Aug 03 2017 08:38:19 fmc SF-IMS[4760]: [4760] Event Streamer:ConnectionHandler [INFO] Accepted IPv4 connection from 10.0.0.1:50408/tcp We're reaching out to support to try an identify what exit code 134 is -- but we're not sure how to proceed from here. Is the log this error is referencing at /bin/encore/estreamer.log? There isn't any additional info here. Thanks! ... View more

Have you reviewed the doc on migration? https://docs.splunk.com/Documentation/Splunk/6.6.2/Installation/MigrateaSplunkinstance ... View more

I think what you'll want to do is create a transforms.conf for this sourcetype that creates the fields for you based on your regex. You will want to define what constitutes a field name (after a comma, before a colon) and what constitutes a field value (after the colon and before the comma). Then use FORMAT = $1::$2 to have Splunk figure out the field and values for you when you search. See this thread for an example: https://answers.splunk.com/answers/150530/how-to-autofill-field-name-with-field-extraction-or-dynamic-rex-statement.html ... View more

We are beginning to roll out a UF install as we deploy Windows 10 on our workstations. So ideally we'd pull with the UF. ... View more

I think somesoni2 has a better option with a syslog-ng relay, but you can use the built in syslog routing queue to send data from the Meraki host from Splunk out to S3; look into the _SYSLOG_ROUTING value for DEST_KEY in transforms. Then configure an outputs.conf for your S3 host. http://docs.splunk.com/Documentation/SplunkCloud/6.6.0/Forwarding/Routeandfilterdatad ... View more

Awesome! Have a great weekend 🙂 ... View more

Have you tried coalesce? | eval combined=coalesce(lcuser, SlotID) ... View more

I want to note that I heard from my SE and this change is only for taking down a single peer, but will be expanded to rolling-restarts in the future. ... View more