All Apps and Add-ons

Import from Splunk Tenable data takes long

benlc
Path Finder

Everything works fine with the import. But it takes a long time to import all my data via API into Splunk.
Per 15min approximitly 1000 new Events after the scan is finished. So if I have about 600'000 Scan-Events it takes almost a week.
Is this normal? Where can I improve it?

Any idea? I can't find any errors in the log.
thanks for helping.
Ben

0 Karma

nickhills
Ultra Champion

Importing scans from Security Center is time consuming, but in theory, once you are up to date, its only the delta your importing on each run - unless your saying you have 600,00 results per scan?

If you don't need to import all events you can change the window from which Splunk will read from the API.

From time-time the Nessus scripts fall over, and I have to restart them - When I do so, i tend to bring the window forward (until just before it stopped) This keeps the delay down.

Another problem I have is that importing large numbers of events with the same time stamp (because that's how nessus does it), I get the following at search:

[indexerName] Events may not be returned in sub-second order due to search memory limits configured in limits.conf:[search]:max_rawsize_perchunk. See search.log for more information.

I suspect this has similar performance implications at index time too, which may well be contributing to the slow import. times.

Sadly, in my experience this is normal, and I have not found a way to improve it. (yet)

If my comment helps, please give it a thumbs up!
0 Karma

benlc
Path Finder

Thanks for your answer and very sorry for my delay.
We don't use the security center, we use the api of the Nessus Professional v6.
Do you have any idea why we get all the Events every time..? Is there some kind of option?

The second problem does not occur, I guess because we only get 1000 Events per 15 min ;-).

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!