Monitor file system on universal forwarder remote host does not forward data expected.

[tkropp]

We have successfully created and deployed an application.

We are currently attempting to consume json data written to a file system on universal forwarder.

1) we created - to
/var/log/github_api/

2) we placed some test json files in there.
(named .json, and .txt, as well as no file extention).

[monitor:///var/log/github_api/]
index=github_api
ignoreOlderThan=1d
host_segment = 3
sourcetype=json

3) used deploy server to push out configs.

Result: Only getting some files but not all.

===============Begin TSHOOT==============

splunkd.log information.


03-06-2013 15:04:39.782 -0500 INFO DeployedApplication - Refreshed app: ise_git_phlow_inputs for service class: syslog from archive: /opt/splunkforwarder/var/run/syslog/ise_git_phlow_inputs-1362598641.bundle
03-06-2013 15:04:40.083 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor:///var/log/github_api/.


Seems to be liking the config.

=========== Next

Looks like we are getting no events into the newly created index.

```
03-06-2013 15:04:39.782 -0500 INFO DeployedApplication - Refreshed app: ise_git_phlow_inputs for service class: syslog from archive: /opt/splunkforwarder/var/run/syslog/ise_git_phlow_inputs-1362598641.bundle
03-06-2013 15:04:40.083 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor:///var/log/github_api/.

```


github_api
500,000
None
1
0
N/A
N/A
/data/hotwarm-indexes/github_api/db
ise_all_indexer_base
Enabled | Disable Delete


====== NEXT UP

Another update. looking into the indexer splunkd.logs for anything relevant

03-06-2013 14:09:52.945 -0500 INFO HotDBManager - idx=github_api Setting hot mgr params: maxHotSpanSecs=7776000 snapBucketTimespans=false maxHotBuckets=3 maxDataSizeBytes=786432000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
03-06-2013 14:09:52.945 -0500 INFO databasePartitionPolicy - idx=github_api Initialized with params='[300,60,188697600,,,,786432000,5,true,500000,5,5,false,3,0,_blocksignature,7776000,1000000,0,3,77760000,2592000,131072,25,0,15,0,0,-1,18446744073709551615,2592000,true,60000,300000,false]' isSlave=false needApplyDeleteJournal=false
03-06-2013 14:09:52.945 -0500 INFO DatabaseDirectoryManager - Writing a bucket manifest in hotWarmPath='/data/hotwarm-indexes/github_api/db'. Reason='Refreshing manifest.'
03-06-2013 14:09:52.946 -0500 INFO databasePartitionPolicy - openDatabases complete currentId=0 idx=github_api
[root@ic-spk01 splunk]#

====== Another update

Main indexer - suspicious.....

So this could have something to do with it:

03-06-2013 15:35:40.563 -0500 WARN DateParserVerbose - Accepted time (Fri Jan 25 10:50:12 2013) is suspiciously far away from the previous event's time (Mon Jan 28 16:31:36 2013), but still accepted because it was extracted by the same pattern. Context: source::/var/log/github_api/test2.txt|host::github_api|json|remoteport::58235

====== FINAL ANSWER?

So as part of the troubleshooting effort we did the following:

1) copied the existing sample JSON data file and made a replica

cp -r notifications test1.txt

cp -r test1.txt test.json

cp -r test1.txt test2.txt

Upon indexing?

on Indexer (splunkd.log)

03-06-2013 15:35:40.563 -0500 WARN DateParserVerbose - Accepted time (Sat Jan 26 09:33:46 2013) is suspiciously far away from the previous event's time (Thu Jan 31 10:11:27 2013), but still accepted because it was extracted by the same pattern. Context: source::/var/log/github_api/test2.txt|host::github_api|json|remoteport::58235
03-06-2013 15:35:40.563 -0500 WARN DateParserVerbose - Accepted time (Fri Jan 25 10:50:12 2013) is suspiciously far away from the previous event's time (Mon Jan 28 16:31:36 2013), but still accepted because it was extracted by the same pattern. Context: source::/var/log/github_api/test2.txt|host::github_api|json|remoteport::58235
03-06-2013 15:35:40.817 -0500 INFO databasePartitionPolicy - idx=github_api Creating hot bucket=hot_v1_1, given event timestamped=1354488490
03-06-2013 15:35:40.817 -0500 INFO DatabaseDirectoryManager - Writing a bucket manifest in hotWarmPath='/data/hotwarm-indexes/github_api/db'. Reason='Bucket directory structure changed.'

on Universal forwarder (splunkd.log)

03-06-2013 15:25:29.740 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor:///var/log/github_api/.

[tkropp]

For us the resolution seemed to be two things.

1) Make data appear in the file system (new data, so generate some json and plop it in the directory.)

2) modify the inputs.conf on the Universal forwarder to comment out ignoreOlderThan (the date time in data was going well back to last year)

[monitor:///var/log/github_api/]
index=github_api
->#ignoreOlderThan=1d
host_segment = 3
sourcetype=json

3) adjust a conflicting indexing configuration, we had the new index being defined in two places.
a. in the main indexer configuration we push to indexers.
b. in the default configuration specific to the new applications context (/default/indexes.conf)

we retained b. only for the application specific index.

Solved by removing "ignoreOlderThan=1d".

Improved index configuration duplication by retain settings for indexes.conf in app specific configuration

Update::: 20130307 - Able to add additional test data today.