Getting Data In
Highlighted

Custom Sourcetype not indexing - Apache TomCat

Path Finder

I have created what I believe to be a custom sourcetype for Apache TomCat logs (which are customised). But when I add an input for a single file and try it (Via Splunk Web), I see no entires in my new index that i'm trying to index it to..

My props.conf:

[Apache-TomCat]
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
REPORT-Apache-TomCat = Apache-TomCat
TRANSFORMS-comment = comment

My transforms.conf:

[comment]
REGEX = \#.*
DEST_KEY = queue
FORMAT = nullQueue

[Apache-TomCat]
FIELDS="date", "time", "c-ip", "x-H(remoteUser)", "cs-method", "cs-uri", "sc-status", "time-taken", "x-H(requestedSessionId)", "x-P(inFrame)", "x-P(eventSource)", "x-P(eventParam)", "x-P(eventShift)", "x-P(rcounter)", "x-P(scrollPositions)", "x-P(objFocusId)", "x-P(__navigator_index)", "x-R(username)", "x-S(int_user_id)
DELIMS = " "

I looked to see if the log had been indexed, by going to the splunk web, clicking manage, then indexes and looking at the count of events for my index which is 0! 😞

Are there any error logs which might help tell me what the problem is?

0 Karma
Highlighted

Re: Custom Sourcetype not indexing - Apache TomCat

Splunk Employee
Splunk Employee

AccentureQBETA, can you post your inputs.conf?

0 Karma
Highlighted

Re: Custom Sourcetype not indexing - Apache TomCat

Path Finder

I have it working now. Or so it appears so far...

I'll need to try some searches (which I'm quiet new at still) to see if the fields have extracted.

Do you know if this method of field-extraction is index-time or search-time?

0 Karma
Highlighted

Re: Custom Sourcetype not indexing - Apache TomCat

Path Finder

I did a data input from Manage > Data Inputs, rather than Home > Add Data and it's working. I think it might have been the host regex I had.

The file is called: NTPA1111_filename.txt

I have the host regex as: [NTPA][0-9]*

So I expect the Host to be NTPA1111

If the Host Regex fails, does anyone know what happenes? where the error is logged?

I've done a search for index=test
which returns 16667 events, but there should be 21k+ events 😞 Why has this happened?

My index, which the file is being processed into is called test.

View solution in original post

0 Karma
Highlighted

Re: Custom Sourcetype not indexing - Apache TomCat

Path Finder

Inputs.config:

[default]
host = NTXA1528

All my files are from: C:\Program Files\Splunk\etc\system\local

0 Karma
Highlighted

Re: Custom Sourcetype not indexing - Apache TomCat

Splunk Employee
Splunk Employee

AccentureABETA, that's not the inputs.conf I was referring to. But if it's working now, then there's no need to look at it. I was just trying to see if you actually specified an index that your Apache-TomCat events would go to. I'll assume you selected your new index from the dropdown when you were adding the data source?

As for your host regex.... I think the proper regex would be NTPA[0-9]* The one you have will match "any character in the set of N, T, P, or A, and then any number of digits.

My regex-foo is fairly mediocre though, so if I'm wrong someone will come along behind me and correct me.

0 Karma
Highlighted

Re: Custom Sourcetype not indexing - Apache TomCat

Path Finder

I think you are right about the regex. I'll update that and tryit again later.

The right number of events are not being parsed though.

I'll cut my file down to 10 lines including the first three lines of the header (all beginging with #) and see if the sourcetype/datainput gets 10 events.

0 Karma
Highlighted

Re: Custom Sourcetype not indexing - Apache TomCat

Path Finder

10 worked fine

0 Karma
Highlighted

Re: Custom Sourcetype not indexing - Apache TomCat

Splunk Employee
Splunk Employee

As for the disparity between the number of lines in the file and the number of events that Splunk sees: I'd check your line breaking. Splunk may be reading multiple lines of the file as a single event.

0 Karma
Highlighted

Re: Custom Sourcetype not indexing - Apache TomCat

Path Finder

What's the best way to check this?

0 Karma