Getting Data In

Forwarded log data is not being parsed correctly. Is there something wrong with my current configuration setup?

shailesh030
Path Finder

Hi!,

I have a splunk setup in which log files are being forwarded by an universal forwarder to an indexer and a search head is being used to perform the search
I have keeping the configuration files in etc/apps/app123/local in searchhead and indexer respectively.
Following are the contents of my configuration files:

On the universal forwarder:
inputs.conf (in apps/local)

[monitor:///home/abc/appLogs.txt]
sourcetype = applogs
blacklist = .(gz)$
index=main

On the search head:
props.conf

[applogs]
REPORT-parse_server=applogs
KV_MODE=none

transforms.conf

[applogs]
DELIMS = "~"
FIELDS = "Text","device_name","domain_name","OperationName","txn_id","time_stamp","FAULT","FaultCode"

On the indexer:
props.conf

[applogs]
TIMESTAMP_FIELDS = StartTimeStamp,ExitTimeStamp,App1StartTimeStamp
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TIME_FORMAT=%b %d %H:%M:%S
TIME_PREFIX=^
TZ=UTC
TRUNCATE=300000
pulldown_type = 1

In the search head UI raw data, I can see the events being indexed with the correct sourcetype but they are not being mapped to fields given in transforms.conf

I have ran btool against each of the config files & no issues were found. The config files are only in apps & none in system/local so it can't be a precedence issue.
I also tried by putting all configurations (props + transforms) into props.conf and keeping them in etc/app/local in searchhead and indexer.

I am not able to figure out what am I missing or where am I going wrong.
Any help will be highly appreciated.

0 Karma

RicoSuave
Builder

Have you tried setting KV_MODE = AUTO? Setting it to NONE disables field extraction for that sourcetype.

0 Karma

shailesh030
Path Finder

I changed props.conf on the search head to KV_MODE=AUTO, restarted splunkd but it still doesn't extract the fields.

0 Karma

RicoSuave
Builder

Oh, try DELIMS = ","

0 Karma

shailesh030
Path Finder

But my log data is delimited by tilda "~" . Nevertheless, I tried changed the ~ to "," in DELIMS in transforms.conf and it still didn't work.

0 Karma

RicoSuave
Builder

Can you post a sample of the applogs events?

0 Karma

shailesh030
Path Finder

Thanks Joetron .. here are some of the applogs events. Each event is in one line

Aug 4 07:02:43 ABC-XY12345-Default [XYZ][123][xsltmsg][info] #ABCD-IN#~XY12345~ALPHA~GCP~55403201~2014-08-04 07:02:43~FAULT~12345
Aug 4 07:02:44 ABC-XY22345-Default [XYZ][123][xsltmsg][info] #ABCD-IN#~XY22345~ALPHA~GCP~65403201~2014-08-04 07:02:44~FAULT~12346
Aug 4 07:02:45 ABC-XY32345-Default [XYZ][123][xsltmsg][info] #ABCD-IN#~XY32345~ALPHA~GCP~75403201~2014-08-04 07:02:45~FAULT~12347
Aug 4 07:02:46 ABC-XY42345-Default [XYZ][123][xsltmsg][info] #ABCD-IN#~XY42345~ALPHA~GCP~85403201~2014-08-04 07:02:46~FAULT~12348

0 Karma
Get Updates on the Splunk Community!

Learn Splunk Insider Insights, Do More With Gen AI, & Find 20+ New Use Cases You Can ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Buttercup Games: Further Dashboarding Techniques (Part 7)

This series of blogs assumes you have already completed the Splunk Enterprise Search Tutorial as it uses the ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

What are Community Office Hours? Community Office Hours is an interactive 60-minute Zoom series where ...