Issue to manage single event with Splunk input data script.

[Nieucel]

Hello,

I connect a bat script as input data to my Splunk instance.
This script reads a folder of websphere log files, for each websphere log file, it processes
a conversion of TIMESTAMP: French TIMESTAMP (with accents) to 'Standard' TIMESTAMP.
e.g.: event converted:

[03/01/2011 08:01:37 AM]Servlet.Engine.Transports : 8730 [error] HTTP0402E Invalid Request

The script display all converted events on the STDOUT.
My issue is that Splunk manage all events from one file as only one multi-line event.(so, I get as many events than file number.)

e.g.:
script stdout for 1 file read:

[12/13/2010 07:13:56 AM] 140.94.3.120 "GET /home.jsp HTTP/1.1" 200 -
[03/01/2011 08:01:37 AM]Servlet.Engine.Transports : 8730 [error] HTTP0402E Invalid Request 1
[03/05/2011 08:59:08 AM]Servlet.Engine.Transports : 9179 [error] HTTP0402E Invalid Request 1
[03/10/2011 09:06:46 AM]Servlet.Engine.Transports : 9243 [error] HTTP0402E Invalid Request 1
[03/15/2011 07:15:00 AM] 140.94.3.120 "GET /home.jsp HTTP/1.1" 200 -
[03/20/2011 07:15:03 AM] 140.94.2.182 "HEAD /home.jsp HTTP/1.1" 200 -
[04/01/2011 08:01:37 AM]Servlet.Engine.Transports : 8730 [error] HTTP0402E Invalid Request 1
[04/05/2011 08:59:08 AM]Servlet.Engine.Transports : 9179 [error] HTTP0402E Invalid Request 1
[04/10/2011 09:06:46 AM]Servlet.Engine.Transports : 9243 [error] HTTP0402E Invalid Request
[04/15/2011 07:15:00 AM] 140.94.3.120 "GET /home.jsp HTTP/1.1" 200 -
[04/20/2011 07:15:03 AM] 140.94.2.182 "HEAD /home.jsp HTTP/1.1" 200 -

In Splunk all these lines are managed as 1 event.

My Splunk configuration for the script:
Command : D:\Product\Splunk\bin\scripts\FOR_TDA_TLS\readFolder.bat D:\Appli\LOGS\TDA_TLS

interval: 20

source type: websphere_activity

app: TDA

index: websphere

What do you think about this?
Is it a problem with the script output? (End of line for example)
Or problem with the Splunk configuration? (source type)

Regards

[Nieucel]

Thank you very much for the quick response.
Now, with the props.conf configuration, my script is well working.

Just a remark:
I have to go through this script, because the event 'French' style with this kind of timestamp format:
[lun. déc. 13, 2010 07:13:56 AM] 140.94.3.120 "GET /home.jsp HTTP/1.1" 200 -
Are not well managed by Splunk.

regards

[BobM]

You need to tell splunk that this scripted input is not a multiline event.
I would add the following to a props.conf file in the app.

[websphere_activity]
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false

The linebreaker should be the default but it doesn't hurt to redefine it.

It is worth noting that you could probably do without the script altogether and index the files directly. Splunk is good at detecting foreign languages and adjusting accordingly though you can tell it what the format is with the following.

TIME_FORMAT = <strptime-style format>
CHARSET = <string>

by the way, it is worth noting this is an index time change so it will not fix any data already in your index.