Getting Data In

monitor stanza in inputs.conf not working, data is not getting indexed

krishnarajapant
Path Finder

Hi Experts,

Below is my inputs.conf stanza which we have in forwarders server in the below path. SplnkFwdrinputs is the app name which we have created in deployment server and pushed to the forwarder by defining the serverclass

/opt/splunkforwarder/etc/apps/SplnkFwdrinputs/local

[monitor:///opt/softwareag/profiles/IS/workspace/temp/splunkStorage]
disabled = false
sourcetype = TMF_XML
index = WMTMF
crcSalt = <SOURCE>

We have created the index as well in the indexer server and we could see the index in the indexer server with name WMTMF.

we have all .xml files in the /opt/softwareag/profiles/IS/workspace/temp/splunkStorage path but we are not able to see any data in the indexer

we have defined the stanza in props.conf of indexer server to handle the xml files which is working as expected, same has been tested with manual upload.

we could see all the xml files when we give the command ./splunk monitor list inputs

Not sure what am I missing here?

-Krishna Rajapantula

Tags (2)
0 Karma

tskinnerivsec
Contributor

Sorry, meant, I saw no issues with props, not new.

0 Karma

krishnarajapant
Path Finder

Hi,

Yes, event i have noticed that and changed the index name in inputs.conf with lower case, still no luck.

can you please see the above props.conf , is there any issue with that [TMF_XML] stanza?

-Krishna Rajapantula

0 Karma

tskinnerivsec
Contributor

I also just noticed, your index name is upper case, which could be causing issues. Per splunkdocs:

  1. To create a new index, enter:

A name for the index. User-defined index names must consist of only numbers, lowercase letters, underscores, and hyphens. They cannot begin with an underscore or hyphen, or contain the word "kvstore".

http://docs.splunk.com/Documentation/Splunk/6.2.5/Indexer/Setupmultipleindexes

you may want to change that, edit your file monitor, and see if that helps.

tskinnerivsec
Contributor

You need to see what props and transforms have been applied to that sourcetype on the indexers. If the data is showing up in main, then the file monitor itself is working and communication to the indexers is ok. A configuration on the indexers may be getting applied to the sourcetype and overriding the index it is being directed to.

on the indexers try this from the /splunk/bin directory

./splunk btool list props > props_list.txt

then, look through that file for a line that says
[TMF_XML]

The stanzas below that line will indicate what is being applied to that sourcetype. A clue to your issue may rest
in one of those configuration stanzas.

0 Karma

krishnarajapant
Path Finder

Hi,

I have below for [TMF_XML] stanza in props.conf in indexer server

[TMF_XML]
ANNOTATE_PUNCT = True
AUTO_KV_JSON = true
BREAK_ONLY_BEFORE = \
BREAK_ONLY_BEFORE_DATE = false
CHARSET = AUTO
DATETIME_CONFIG = \etc\datetime.xml
HEADER_MODE =
KV_MODE = xml
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 256
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
NO_BINARY_CHECK = true
REPORT-xmlext = xml-extr
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = True
TRANSFORMS =
TRUNCATE = 10000
category = Custom
detect_trailing_nulls = auto
disabled = false
maxDist = 100
priority =
pulldown_type = true
sourcetype =

-Krishna Rajapantula

0 Karma

tskinnerivsec
Contributor

I saw new issues with the props.conf. The inputs.conf file is not the only thing that needed to get changed. Did you recreate the index in lower case? If not, create the new index.

0 Karma

krishnarajapant
Path Finder

Hi,
I have created new index with lowercase with name wmtmf1 and updated the inputs.conf, still it is same.

can you please tell me the issues with props.conf

Above result is the output of ./splunk btool list props > props_list.txt

-Krishna Rajapantula.

0 Karma

krishnarajapant
Path Finder

Hi tskinnerivsec,

Thanks for your responses.

I have tried all the steps mentioned, still no luck.

surprisingly if i'm not mentioning then it is getting indexed in default index

So I believe it is only problem with the indexes other than default "main" index.

I have created those index manually before pushing this data , not sure where is the issue?

-Krishna Rajapantula.

0 Karma

tskinnerivsec
Contributor

Have you performed a tcp dump from the source host to see if the forwarder is attempting to send the data to your indexer on tcp port 9997 ? Also, is any eventdata at all being indexed from this universal forwarder?

0 Karma

tlelle_splunk
Splunk Employee
Splunk Employee

You can take out

[tcpout-server://indexerip:9997]

from your outputs.conf, assuming that you have an actual IP listed in the line

[tcpout:default-autolb-group]
server = indexerip:9997

also try the command ./splunk list forward-server to ensure that the connection to your indexer is active.

If you make the changes and the server is still showing inactive, ensure that port 9997 is open on your indexer.

0 Karma

krishnarajapant
Path Finder

Hi tlelle,

Yes correct, I have mentioned actual ip in that stanza instead of "indexerip"

I have removed the [tcpout-server://indexerip:9997] stanza from outputs.conf

I have one forwarder indexerip:9997 with active list in my forwarder servr.

Still i'm not able to see any data in my indexer

-Krishna Rajapantula

0 Karma

tlelle_splunk
Splunk Employee
Splunk Employee

Have you previously indexed these files into another index (say, maybe a test index) and have since switched the index that you want to contain the data? If so, have the source files that you are trying to index changed?

It's a stretch, but maybe try changing your crcsalt?

Have you had any success testing out your data with a oneshot command? http://docs.splunk.com/Documentation/Splunk/6.2.5/Data/MonitorfilesanddirectoriesusingtheCLI#CLI_com...

0 Karma

krishnarajapant
Path Finder

Hi tlelle,

Thanks for your responses.

I have tried all the steps mentioned, still no luck.

surprisingly if i'm not mentioning then it is getting indexed in default index

So I believe it is only problem with the indexes other than default "main" index.

I have created those index manually before pushing this data , not sure where is the issue?

-Krishna Rajapantula.

0 Karma

jensonthottian
Contributor

Did you try restarting splunkd?

0 Karma

tlelle_splunk
Splunk Employee
Splunk Employee

What does your outputs.conf look like? Were you already able to onboard data previously from this machine?

0 Karma

krishnarajapant
Path Finder

Hi tlelle,

Thanks for your response.

outputs.conf is in /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/local

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = indexerip:9997

[tcpout-server://indexerip:9997]

No first time i'm trying to push data from this machine.

even I have tried the same in another environment as well, same issue, not sure what i'm missing.

-Krishna Rajapantula

0 Karma

krishnarajapant
Path Finder

Thanks jensonthottian for your response.

Yes I did that..Still no luck.

-Krishna Rajapantula

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...