Getting Data In

Forwarded log data is not being parsed correctly. Is there something wrong with my current configuration setup?

shailesh030
Path Finder

Hi!,

I have a splunk setup in which log files are being forwarded by an universal forwarder to an indexer and a search head is being used to perform the search
I have keeping the configuration files in etc/apps/app123/local in searchhead and indexer respectively.
Following are the contents of my configuration files:

On the universal forwarder:
inputs.conf (in apps/local)

[monitor:///home/abc/appLogs.txt]
sourcetype = applogs
blacklist = .(gz)$
index=main

On the search head:
props.conf

[applogs]
REPORT-parse_server=applogs
KV_MODE=none

transforms.conf

[applogs]
DELIMS = "~"
FIELDS = "Text","device_name","domain_name","OperationName","txn_id","time_stamp","FAULT","FaultCode"

On the indexer:
props.conf

[applogs]
TIMESTAMP_FIELDS = StartTimeStamp,ExitTimeStamp,App1StartTimeStamp
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TIME_FORMAT=%b %d %H:%M:%S
TIME_PREFIX=^
TZ=UTC
TRUNCATE=300000
pulldown_type = 1

In the search head UI raw data, I can see the events being indexed with the correct sourcetype but they are not being mapped to fields given in transforms.conf

I have ran btool against each of the config files & no issues were found. The config files are only in apps & none in system/local so it can't be a precedence issue.
I also tried by putting all configurations (props + transforms) into props.conf and keeping them in etc/app/local in searchhead and indexer.

I am not able to figure out what am I missing or where am I going wrong.
Any help will be highly appreciated.

0 Karma

RicoSuave
Builder

Have you tried setting KV_MODE = AUTO? Setting it to NONE disables field extraction for that sourcetype.

0 Karma

shailesh030
Path Finder

I changed props.conf on the search head to KV_MODE=AUTO, restarted splunkd but it still doesn't extract the fields.

0 Karma

RicoSuave
Builder

Oh, try DELIMS = ","

0 Karma

shailesh030
Path Finder

But my log data is delimited by tilda "~" . Nevertheless, I tried changed the ~ to "," in DELIMS in transforms.conf and it still didn't work.

0 Karma

RicoSuave
Builder

Can you post a sample of the applogs events?

0 Karma

shailesh030
Path Finder

Thanks Joetron .. here are some of the applogs events. Each event is in one line

Aug 4 07:02:43 ABC-XY12345-Default [XYZ][123][xsltmsg][info] #ABCD-IN#~XY12345~ALPHA~GCP~55403201~2014-08-04 07:02:43~FAULT~12345
Aug 4 07:02:44 ABC-XY22345-Default [XYZ][123][xsltmsg][info] #ABCD-IN#~XY22345~ALPHA~GCP~65403201~2014-08-04 07:02:44~FAULT~12346
Aug 4 07:02:45 ABC-XY32345-Default [XYZ][123][xsltmsg][info] #ABCD-IN#~XY32345~ALPHA~GCP~75403201~2014-08-04 07:02:45~FAULT~12347
Aug 4 07:02:46 ABC-XY42345-Default [XYZ][123][xsltmsg][info] #ABCD-IN#~XY42345~ALPHA~GCP~85403201~2014-08-04 07:02:46~FAULT~12348

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...