Getting Data In

Forwarded log data is not being parsed correctly. Is there something wrong with my current configuration setup?

shailesh030
Path Finder

Hi!,

I have a splunk setup in which log files are being forwarded by an universal forwarder to an indexer and a search head is being used to perform the search
I have keeping the configuration files in etc/apps/app123/local in searchhead and indexer respectively.
Following are the contents of my configuration files:

On the universal forwarder:
inputs.conf (in apps/local)

[monitor:///home/abc/appLogs.txt]
sourcetype = applogs
blacklist = .(gz)$
index=main

On the search head:
props.conf

[applogs]
REPORT-parse_server=applogs
KV_MODE=none

transforms.conf

[applogs]
DELIMS = "~"
FIELDS = "Text","device_name","domain_name","OperationName","txn_id","time_stamp","FAULT","FaultCode"

On the indexer:
props.conf

[applogs]
TIMESTAMP_FIELDS = StartTimeStamp,ExitTimeStamp,App1StartTimeStamp
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TIME_FORMAT=%b %d %H:%M:%S
TIME_PREFIX=^
TZ=UTC
TRUNCATE=300000
pulldown_type = 1

In the search head UI raw data, I can see the events being indexed with the correct sourcetype but they are not being mapped to fields given in transforms.conf

I have ran btool against each of the config files & no issues were found. The config files are only in apps & none in system/local so it can't be a precedence issue.
I also tried by putting all configurations (props + transforms) into props.conf and keeping them in etc/app/local in searchhead and indexer.

I am not able to figure out what am I missing or where am I going wrong.
Any help will be highly appreciated.

0 Karma

RicoSuave
Builder

Have you tried setting KV_MODE = AUTO? Setting it to NONE disables field extraction for that sourcetype.

0 Karma

shailesh030
Path Finder

I changed props.conf on the search head to KV_MODE=AUTO, restarted splunkd but it still doesn't extract the fields.

0 Karma

RicoSuave
Builder

Oh, try DELIMS = ","

0 Karma

shailesh030
Path Finder

But my log data is delimited by tilda "~" . Nevertheless, I tried changed the ~ to "," in DELIMS in transforms.conf and it still didn't work.

0 Karma

RicoSuave
Builder

Can you post a sample of the applogs events?

0 Karma

shailesh030
Path Finder

Thanks Joetron .. here are some of the applogs events. Each event is in one line

Aug 4 07:02:43 ABC-XY12345-Default [XYZ][123][xsltmsg][info] #ABCD-IN#~XY12345~ALPHA~GCP~55403201~2014-08-04 07:02:43~FAULT~12345
Aug 4 07:02:44 ABC-XY22345-Default [XYZ][123][xsltmsg][info] #ABCD-IN#~XY22345~ALPHA~GCP~65403201~2014-08-04 07:02:44~FAULT~12346
Aug 4 07:02:45 ABC-XY32345-Default [XYZ][123][xsltmsg][info] #ABCD-IN#~XY32345~ALPHA~GCP~75403201~2014-08-04 07:02:45~FAULT~12347
Aug 4 07:02:46 ABC-XY42345-Default [XYZ][123][xsltmsg][info] #ABCD-IN#~XY42345~ALPHA~GCP~85403201~2014-08-04 07:02:46~FAULT~12348

0 Karma
Get Updates on the Splunk Community!

What the End of Support for Splunk Add-on Builder Means for You

Hello Splunk Community! We want to share an important update regarding the future of the Splunk Add-on Builder ...

Solve, Learn, Repeat: New Puzzle Channel Now Live

Welcome to the Splunk Puzzle PlaygroundIf you are anything like me, you love to solve problems, and what ...

Building Reliable Asset and Identity Frameworks in Splunk ES

 Accurate asset and identity resolution is the backbone of security operations. Without it, alerts are ...