topic Re: File Eating in Getting Data In

File Eating

hartfoml — Wed, 07 Dec 2011 17:41:29 GMT

I am eating NESSUS.V1 files from our Nessus contiues monitoring system

Nessus puts the output from the scan in XML format in the v1 files and the indivitule system info is put in a one event format when it is read into Splunk indexer.

The problem is some of the events are more than 257 lines and Splunk is truncating the events at 257.

Then I lose some of the event integrity and I have to go look at the very next event to get the rest of the data.

How can I increase the number of lines for this source or source type?

Re: File Eating

Ayn — Wed, 07 Dec 2011 17:46:41 GMT

Change MAX_EVENTS for your sourcetype in props.conf.

MAX_EVENTS = <integer>
* Specifies the maximum number of input lines to add to any event.
* Splunk breaks after the specified number of lines are read.
* Defaults to 256 (lines).

Re: File Eating

Ayn — Wed, 07 Dec 2011 17:47:46 GMT

Also, I would be very interested to hear what you're doing and what you want to do with Nessus reports in Splunk. I'm in the process of creating an app for Nessus and some other vulnerability scanners so I'm very thankful for all input!

Re: File Eating

hartfoml — Wed, 07 Dec 2011 17:49:11 GMT

thanks Ayn,

I found this post after I asked the question

http://splunk-base.splunk.com/answers/6764/events-chunked-into-256-lines

Thanks again

Re: File Eating

hartfoml — Wed, 07 Dec 2011 18:11:53 GMT

Ayn,

Do you think I should modify the props.conf in the universal forwarder app "/etc/deployment-apps/appName/Local" or the indexer "etc/system/local" ?

Re: File Eating

Ayn — Wed, 07 Dec 2011 18:18:19 GMT

Line breaking is done on the indexer, so that's where the props.conf changes should go.

Re: File Eating

gkanapathy — Wed, 07 Dec 2011 18:48:31 GMT

Well, line breaking hapens whereever the parsing occurs. With a Universal Forwarder, it is indeed on the indexer, but details are: http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

Re: File Eating

Ayn — Wed, 07 Dec 2011 20:35:08 GMT

Well in this case it was either on a UF or an indexer, so...