topic Re: Sourcefire Encore data ingestion issue in Getting Data In

Sourcefire Encore data ingestion issue

vik_splunk — Tue, 15 Jun 2021 19:13:24 GMT

Hi All,

We have recently upgraded from 7.2.6 to 8.1.3 Splunk and since then, we have been having issues with Sourcefire ingestion from FMC.

Splunk and sourcefire version - prior to upgrade - 7.2.6 and 3.0.0

Splunk and sourcefire version - Post upgrade - 8.1.3 and 4.6.0

TA used - https://splunkbase.splunk.com/app/3662/

What we've attempted so far

3.0.0 with compatibility enabled for Python 2.x - Errors out with Connection reset by peer
estreamer.subscriber ERROR error: \nTraceback (most recent call last):\n File "/opt/splunk/etc/apps/TA-eStreamer/bin/encore/estreamer/subscriber.py", line 198, in start\n self.connection.connect()\n File "/opt/splunk/etc/apps/TA-eStreamer/bin/encore/estreamer/connection.py", line 80, in connect\n self.socket.connect( ( host, port ) )\n File "/opt/splunk/lib/python2.7/ssl.py", line 864, in connect\n self._real_connect(addr, False)\n File "/opt/splunk/lib/python2.7/ssl.py", line 855, in _real_connect\n self.do_handshake()\n File "/opt/splunk/lib/python2.7/ssl.py", line 828, in do_handshake\n self._sslobj.do_handshake()\nerror: [Errno 104] Connection reset by peer\n
4.6.0 upgraded TA in 8.1.3 - Connection succeeds and collects logs for a while but then, we are met with the errors "Invalid JSON in settings file" followed by Subscriberparser is dead, message - We also found this bug reference, similar to the error -
https://quickview.cloudapps.cisco.com/quickview/bug/CSCvy06369
4.6.0 upgraded TA with compatibility enabled for 2.x - Same as above. Connection succeeds but eventually, stops collection after a while and errors out with the same message as present in 2.
Fresh install of 4.6.0 followed by fresh config. Connects fine to FMC but errors out as below
"Error state. Clearing queue"

In a nutshell, what used to be a stable stream of logs from FMC is completely broken/fragmented. In all cases, able to use the splencore test to establish successful connection and have restarted the service but no luck.

We have been through all articles in community and as well, all suggested troubleshooting but no luck. Any advice on getting this working is much appreciated.

@douglashurd - Can you please advise. Thanks!

Re: Sourcefire Encore data ingestion issue

douglashurd — Tue, 15 Jun 2021 21:56:45 GMT

Thanks for the message! We're looking at it. Appreciate all the details.

Doug

Re: Sourcefire Encore data ingestion issue

vik_splunk — Thu, 17 Jun 2021 08:41:31 GMT

Hi @douglashurd just wanted to follow up and check if you were able to find anything of interest/ a possible fix?

Meanwhile, we managed to test in a different environment with FMC 6.4.4 and that seems to be stable in comparison.

Versions used - Splunk 8.1.3 , Python 3 , Add on ver. 4.6.0 and FMC 6.4.4

Interestingly, we do see the same error as seen earlier but the key difference is that it does not result in abrupt stoppage of logs. Seems to offer the same pre-upgrade stability.

As can be seen, the estreamer.log stopped at 9:39 EDT yesterday with the same error as mentioned in the bug report. However, subsequent listing of the data folder shows the continuous inflow of logs.

2021-06-16 09:33:32,517 Monitor INFO Running. 697900 handled; average rate 86.8 ev/sec;
2021-06-16 09:35:31,612 Monitor INFO Running. 697900 handled; average rate 85.52 ev/sec;
2021-06-16 09:37:31,434 Monitor INFO Running. 698000 handled; average rate 84.29 ev/sec;
2021-06-16 09:39:31,593 Monitor INFO Running. 698000 handled; average rate 83.09 ev/sec;
2021-06-16 09:39:32,450 Service ERROR [no message or attrs]: Invalid JSON in settings file

pwd
/opt/splunk/etc/apps/TA-eStreamer/bin/encore/data/splunk
ls -ltr
Jun 17 04:36 encore.1623902980.log

Re: Sourcefire Encore data ingestion issue

elee_splunk — Mon, 21 Jun 2021 17:02:01 GMT

Hi @douglashurd ,

I am also seeing the same issue with the latest TA on Splunk 8.2. Seems like it happens every day or every other day I get the same error and it stops pulling logs.

2021-06-21 08:10:12,573 Monitor INFO Running. 62720000 handled; average rate 1371.82 ev/sec;
2021-06-21 08:12:11,573 Monitor INFO Running. 62860500 handled; average rate 1371.3 ev/sec;
2021-06-21 08:14:11,827 Monitor INFO Running. 63002100 handled; average rate 1370.8 ev/sec;
2021-06-21 08:16:12,215 Monitor INFO Running. 63148100 handled; average rate 1370.4 ev/sec;
2021-06-21 08:16:43,508 Service ERROR [no message or attrs]: Invalid JSON in settings file

Re: Sourcefire Encore data ingestion issue

vik_splunk — Tue, 22 Jun 2021 09:15:26 GMT

Error not withstanding, it seems to be stable as before following our upgrade to FMC 6.4.4.

Appears to have been by the bug noted earlier in this ticket.

To conclude, the error for Invalid JSON.. is still appearing for the new FMC version as well. It continues to ingest logs anyway.

Marking this as the solution @douglashurd . Unless you see any fix that is required to the TA, my issue is fixed now. Thanks!

Re: Sourcefire Encore data ingestion issue

vik_splunk — Wed, 23 Jun 2021 13:31:36 GMT

Appears that I spoke too soon @douglashurd

It was fine for two days and now, we are back to square one unfortunately with the below errors

This is still an issue and will require a fix. Please advise.

2021-06-23 09:23:53,924 Connection INFO Connecting to <IP>
2021-06-23 09:23:53,926 Connection INFO Using TLS v1.2
2021-06-23 09:23:53,926 Transformer INFO Starting process.
2021-06-23 09:23:53,927 Monitor INFO Starting Monitor.
2021-06-23 09:23:53,927 Decorator INFO Starting process.
2021-06-23 09:23:53,928 Transformer DEBUG Transformer
2021-06-23 09:23:53,928 Decorator DEBUG Decorator
2021-06-23 09:23:53,928 Writer INFO Starting process.
2021-06-23 09:23:53,929 Writer DEBUG Writer
2021-06-23 09:23:53,929 Monitor INFO Starting. 0 handled; average rate 0 ev/sec;
2021-06-23 09:25:54,081 Controller INFO Process subscriberParser is dead.
2021-06-23 09:25:54,081 Monitor INFO Running. 0 handled; average rate 0 ev/sec;
2021-06-23 09:25:54,133 Controller INFO Stopping...
2021-06-23 09:25:54,134 Controller INFO Process 7091 (Process-1) exit code: 1
2021-06-23 09:25:54,134 Decorator INFO Stop message received
2021-06-23 09:25:54,140 Decorator INFO Error state. Clearing queue
2021-06-23 09:25:54,141 Decorator INFO Exiting
2021-06-23 09:25:54,141 Controller INFO Process 7092 (Process-2) exit code: 0
2021-06-23 09:25:54,146 Transformer INFO Stop message received
2021-06-23 09:25:54,152 Transformer INFO Error state. Clearing queue
2021-06-23 09:25:54,152 Transformer INFO Exiting
2021-06-23 09:25:54,152 Controller INFO Process 7093 (Process-3) exit code: 0
2021-06-23 09:25:54,157 Writer INFO Stop message received
2021-06-23 09:25:54,163 Writer INFO Error state. Clearing queue
2021-06-23 09:25:54,163 Writer INFO Exiting
2021-06-23 09:25:54,163 Controller INFO Process 7096 (Process-4) exit code: 0
2021-06-23 09:25:54,163 Monitor INFO Stopping Monitor.
2021-06-23 09:25:54,333 Controller INFO Goodbye

Re: Sourcefire Encore data ingestion issue

elee_splunk — Wed, 23 Jun 2021 15:18:59 GMT

Same here, happening to me on a daily basis

Re: Sourcefire Encore data ingestion issue

vik_splunk — Thu, 24 Jun 2021 19:21:15 GMT

Hmmm.. Just putting it out there. Not sure if it contributes to the issue @elee_splunk

What Linux version are you running Splunk on the specific machine?

The noticeable difference in our environment is as below.

FMC	RHEL	Splunk	estreamer TA	customizations if any	Status
6.6.4	6.x	8.1.3	4.6.0	sourcetype slightly changed	Completely broken without a semblance of stability
6.6.4	7.x	8.1.3	4.6.0	default sourcetype	Pretty stable

@douglashurd thoughts?

Re: Sourcefire Encore data ingestion issue

elee_splunk — Thu, 24 Jun 2021 19:57:59 GMT

FMC	Ubuntu	Splunk	estreamer TA	customizations if any	Status
6.6.4	20.04.2 LTS	8.2	4.6.0	default sourcetype	Intermittent

I have to stop the estreamer service and start it back up to get it going at least once every day or 2 days. When that doesn't work I have to reboot the whole server.

Re: Sourcefire Encore data ingestion issue

AlexS — Tue, 29 Jun 2021 13:03:40 GMT

Hi Douglas,

it seems I´m facing the same issue.

Version 4.0.9 suddenly stopped working, and ran into the following error:
Decorator ERROR [no message or attrs]: 'View' object has no attribute '_view__isHex'...

As I did not get it to work again (restart,reboots...), I tried the new 4.6.0 release.

In the beginning it looked fine (besides missing status and clean options in splencore.sh that I added again) , logs were written and ingested into Splunk.
After almost 3 hours the error "Service ERROR [no message or attrs]: Invalid JSON in settings file" appeared in the estreamer.log. But logs where still written and input to splunk worked - except estreamer.log: monitor logs stopped and no other message was written.

Suddenly more then 7 hours after the "invalid JSON in settings file" error, fmc logs stopped, too.

After restart of the client, it started working again. The "invalid JSON.." error already reappeared after 2 hours. This time I did not wait for fmc logs to stop and directly restarted the client. Now since 5 hours, no error but according to the other feedback in here, it´s only a matter of time.

Did you already identify any issues?

Thanks,
Alex

Re: Sourcefire Encore data ingestion issue

elee_splunk — Tue, 29 Jun 2021 18:44:48 GMT

@douglashurd @vik_splunk

I've switched back to my old HF and its been very stable.

FMC	Ubuntu	Splunk	estreamer TA	customizations if any	Status
6.6.4	18.04.5 LTS	7.2.10	3.6.8	default sourcetype	Stable

Re: Sourcefire Encore data ingestion issue

douglashurd — Tue, 29 Jun 2021 20:56:14 GMT

Please email a link to this thread to encore-community@cisco.com

Thanks,

Doug

Re: Sourcefire Encore data ingestion issue

vik_splunk — Wed, 30 Jun 2021 09:51:23 GMT

Just did, @douglashurd . Thanks!

Re: Sourcefire Encore data ingestion issue

vik_splunk — Wed, 30 Jun 2021 09:53:44 GMT

Hi @elee_splunk That's good to hear.

Appears Python 2 script seems to offer stability in your case. In our environment, it's a bit puzzling in the fact that we have one forwarder ingesting logs without issues in the new version while the prod environment seems to throw errors.

Re: Sourcefire Encore data ingestion issue

skhademd — Wed, 30 Jun 2021 15:22:16 GMT

Hi,

We believe this is a bug in the splencore.sh script, specifically with the clean command, in a prior update we modified the location of the ingest directory, this script was not updated to reflect the new location which we believe is causing the error you are seeing. Please perform the following

1) Disable the clean script

2) Modify the splencore.sh to point the proper ingest directory

3) Upgrade the TA to 4.6.1, which includes the modification above

If problems continue please open a ticket with TAC support so we can collect additional trouble shoot files

Thanks

Seyed

Re: Sourcefire Encore data ingestion issue

elee_splunk — Wed, 30 Jun 2021 15:27:35 GMT

Will try that. I don't believe we have access to the 4.6.1 version of the TA yet. Doesn't look like it's published to splunkbase.

Re: Sourcefire Encore data ingestion issue

vik_splunk — Wed, 30 Jun 2021 16:10:13 GMT

Thanks @skhademd

The new TA is available now. Will validate and confirm.

Re: Sourcefire Encore data ingestion issue

vik_splunk — Wed, 30 Jun 2021 18:46:45 GMT

@skhademd @douglashurd

Have installed 4.6.1 afresh and configured from scratch to connect to our FMCs(with DEBUG logging enabled. Collection has resumed in Splunk. However, here are the initial observations.

1)Data collection is always 10-15 minutes behind current time.

2)Checking estreamer logs, the events per second is consistently on the decrease. Not sure if it's on the path to stoppage of collection

2021-06-30 14:19:58,631 Monitor INFO Running. 103400 handled; average rate 25.34 ev/sec;
2021-06-30 14:20:08,633 Receiver DEBUG FMC sent no data
2021-06-30 14:20:23,145 Receiver DEBUG FMC sent no data
2021-06-30 14:20:33,658 Receiver DEBUG FMC sent no data
2021-06-30 14:20:47,175 Receiver DEBUG FMC sent no data
2021-06-30 14:20:57,684 Receiver DEBUG FMC sent no data
2021-06-30 14:21:08,190 Receiver DEBUG FMC sent no data
2021-06-30 14:21:18,699 Receiver DEBUG FMC sent no data
2021-06-30 14:21:29,209 Receiver DEBUG FMC sent no data
2021-06-30 14:21:36,240 Receiver DEBUG Got null message.
2021-06-30 14:21:47,252 Receiver DEBUG FMC sent no data
2021-06-30 14:21:57,764 Receiver DEBUG FMC sent no data
2021-06-30 14:21:58,274 Monitor INFO Running. 103400 handled; average rate 24.62 ev/sec;
2021-06-30 14:22:08,267 Receiver DEBUG FMC sent no data
2021-06-30 14:22:18,778 Receiver DEBUG FMC sent no data
2021-06-30 14:22:29,289 Receiver DEBUG FMC sent no data
2021-06-30 14:22:36,301 Receiver DEBUG Got null message.
2021-06-30 14:22:46,312 Receiver DEBUG FMC sent no data
2021-06-30 14:22:56,822 Receiver DEBUG FMC sent no data
2021-06-30 14:23:14,347 Receiver DEBUG FMC sent no data
2021-06-30 14:23:29,368 Receiver DEBUG FMC sent no data
2021-06-30 14:23:43,386 Receiver DEBUG FMC sent no data
2021-06-30 14:23:47,402 Monitor INFO Running. 103400 handled; average rate 23.93 ev/sec;
2021-06-30 14:23:57,406 Receiver DEBUG FMC sent no data
2021-06-30 14:24:07,915 Receiver DEBUG FMC sent no data
2021-06-30 14:24:22,436 Receiver DEBUG FMC sent no data
2021-06-30 14:24:32,946 Receiver DEBUG FMC sent no data
2021-06-30 14:24:43,454 Receiver DEBUG FMC sent no data
2021-06-30 14:24:53,959 Receiver DEBUG FMC sent no data
2021-06-30 14:25:04,471 Receiver DEBUG FMC sent no data
2021-06-30 14:25:11,492 Receiver DEBUG Got null message.
2021-06-30 14:25:21,498 Receiver DEBUG FMC sent no data
2021-06-30 14:25:32,006 Receiver DEBUG FMC sent no data
2021-06-30 14:25:42,518 Receiver DEBUG FMC sent no data
2021-06-30 14:25:53,028 Receiver DEBUG FMC sent no data
2021-06-30 14:25:53,542 Monitor INFO Running. 103400 handled; average rate 23.29 ev/sec;
2021-06-30 14:26:03,540 Receiver DEBUG FMC sent no data
2021-06-30 14:26:11,564 Receiver DEBUG Got null message.
2021-06-30 14:26:21,563 Receiver DEBUG FMC sent no data
2021-06-30 14:26:32,076 Receiver DEBUG FMC sent no data
2021-06-30 14:26:42,584 Receiver DEBUG FMC sent no data
2021-06-30 14:26:53,089 Receiver DEBUG FMC sent no data
2021-06-30 14:27:03,601 Receiver DEBUG FMC sent no data
2021-06-30 14:27:11,626 Receiver DEBUG Got null message.
2021-06-30 14:27:21,636 Receiver DEBUG FMC sent no data
2021-06-30 14:27:32,149 Receiver DEBUG FMC sent no data
2021-06-30 14:27:49,673 Receiver DEBUG FMC sent no data
2021-06-30 14:28:00,183 Receiver DEBUG FMC sent no data
2021-06-30 14:28:00,697 Monitor INFO Running. 103400 handled; average rate 22.67 ev/sec;
2021-06-30 14:28:10,688 Receiver DEBUG FMC sent no data
2021-06-30 14:28:21,193 Receiver DEBUG FMC sent no data
2021-06-30 14:28:31,701 Receiver DEBUG FMC sent no data

3)Clean up script has an issue with absolute vs relative path I suspect , as I notice this error in Splunk internal logs.

06-30-2021 14:17:27.087 -0400 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-eStreamer/bin/splencore.sh clean" find: `./encore/data': No such file or directory

Re: Sourcefire Encore data ingestion issue

vik_splunk — Thu, 01 Jul 2021 08:15:43 GMT

@skhademd @douglashurd Unfortunately, the new version of 4.6.1 didn't help either. The same error is back, as always, slightly later in the same day.

In parallel, we have had our Network admins raise a ticket with Cisco. Cisco has acknowledged the problem and raised a bug as can be seen below. I will e-mail details of the case to the e-mail address shared by Doug.

We have a bug open for this, but it looks like there is no root cause found yet:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvy79722

Logs as shown below.

2021-06-30 18:21:51,735 Monitor INFO Running. 104600 handled; average rate 5.62 ev/sec;
2021-06-30 18:22:19,262 Receiver DEBUG FMC sent no data
2021-06-30 18:22:41,791 Receiver DEBUG FMC sent no data
2021-06-30 18:22:52,297 Receiver DEBUG FMC sent no data
2021-06-30 18:23:02,798 Receiver DEBUG FMC sent no data
2021-06-30 18:23:13,307 Receiver DEBUG FMC sent no data
2021-06-30 18:23:23,836 Receiver DEBUG FMC sent no data
2021-06-30 18:23:34,343 Receiver DEBUG FMC sent no data
2021-06-30 18:23:52,872 Receiver DEBUG FMC sent no data
2021-06-30 18:23:53,384 Monitor INFO Running. 104600 handled; average rate 5.59 ev/sec;
2021-06-30 18:24:11,898 Receiver DEBUG FMC sent no data
2021-06-30 18:24:22,400 Receiver DEBUG FMC sent no data
2021-06-30 18:24:32,907 Receiver DEBUG FMC sent no data
2021-06-30 18:24:43,413 Receiver DEBUG FMC sent no data
2021-06-30 18:24:53,929 Receiver DEBUG FMC sent no data
2021-06-30 18:25:00,952 Receiver DEBUG Got null message.
2021-06-30 18:25:15,971 Receiver DEBUG FMC sent no data
2021-06-30 18:25:26,472 Receiver DEBUG FMC sent no data
2021-06-30 18:25:45,003 Receiver DEBUG FMC sent no data
2021-06-30 18:25:51,028 Monitor INFO Running. 104700 handled; average rate 5.56 ev/sec;
2021-06-30 18:26:11,040 Receiver DEBUG FMC sent no data
2021-06-30 18:26:21,550 Receiver DEBUG FMC sent no data
2021-06-30 18:26:32,058 Receiver DEBUG FMC sent no data
2021-06-30 18:26:42,569 Receiver DEBUG FMC sent no data
2021-06-30 18:26:53,076 Receiver DEBUG FMC sent no data
2021-06-30 18:27:00,099 Receiver DEBUG Got null message.
2021-06-30 18:27:10,099 Receiver DEBUG FMC sent no data
2021-06-30 18:27:20,612 Receiver DEBUG FMC sent no data
2021-06-30 18:27:31,123 Receiver DEBUG FMC sent no data
2021-06-30 18:27:41,635 Receiver DEBUG FMC sent no data
2021-06-30 18:27:46,163 Monitor INFO Running. 104700 handled; average rate 5.52 ev/sec;
2021-06-30 18:27:56,154 Receiver DEBUG FMC sent no data
2021-06-30 18:28:10,174 Receiver DEBUG FMC sent no data
2021-06-30 18:28:25,192 Receiver DEBUG FMC sent no data
2021-06-30 18:28:41,212 Receiver DEBUG FMC sent no data
2021-06-30 18:29:05,236 Receiver DEBUG FMC sent no data
2021-06-30 18:29:15,746 Receiver DEBUG FMC sent no data
2021-06-30 18:29:32,268 Receiver DEBUG FMC sent no data
2021-06-30 18:29:45,291 Receiver DEBUG FMC sent no data
2021-06-30 18:29:55,802 Receiver DEBUG FMC sent no data
2021-06-30 18:29:56,312 Monitor INFO Running. 104700 handled; average rate 5.49 ev/sec;
2021-06-30 18:30:06,304 Receiver DEBUG FMC sent no data
2021-06-30 18:30:17,330 Receiver DEBUG FMC sent no data
2021-06-30 18:30:34,346 Receiver DEBUG FMC sent no data
2021-06-30 18:30:47,081 Service ERROR [no message or attrs]: Invalid JSON in settings file

Re: Sourcefire Encore data ingestion issue

AlexS — Thu, 01 Jul 2021 13:53:57 GMT

I compared the new 4.6.1 version and for me issue is also not fixed. The changes necessary for the cleaning to run, I already implemented before.

For now I added another script input to call splencore.sh with a stop statement every x hours, so the service gets restarted regularly. The error appears after random time. Last time error appeared after 45 minutes, but the time before it was running more than 10 hours.
And still same behavior, logging continues to work after the error is logged but stops some time later.