Setup Splunk monitoring to watch a directory. Files started coming in but with the timestamp not being parsed correctly. I adjusted by Settings > Data > Source Type then I cloned the default json and clicked Advanced and set the timestamp to this `%d-%m-%Y%H:%M:%S` for the field systemTime. (I even tried adding surrounding quotes at one point)
"systemTime" : "22-01-2019_15:05:01",
"fieldType" : "XXX-XXX",
"fieldLocation" : "XXX1",
"fieldCommand" : "XXXXXX",
"kernalName" : "Linux",
"nodeName" : "x86_64",
"kernalRelease" : "4.15.0-43-generic",
"kernalVersion" : "#46~16.04.1-Ubuntu SMP Fri Dec 7 13:31:08 UTC 2018",
"machine" : "x86_64",
"processor" : "x86_64",
"hardwarePlatform" : "x86_64",
"operatingSystem" : "GNU/Linux",
"timeup" : " 15:05:01 up 8 days, 4:48, 2 users, load average: 0.35, 0.40, 0.31",
"soft1Version" : "XXXXX",
"soft2Version" : "XXXXXXXX"
I noticed the files stopped coming in so I checked
index=_internal source=*/splunkd.log OR source=*\\splunkd.log | search *system* log_level=ERROR and found errors like
ERROR JsonLineBreaker - JSON StreamId:3524616290329204733 had parsing error:Unexpected character while looking for value: '\\'.
Despite the files not being ingested, when I go to Settings > Data Inputs > Files & Directories the file count for that directory continues to rise.
It seems to be that if I remove the timestamp part, the file does get correctly processed but _time becomes 1979...
Please try with below configuration in props.conf for your new sourcetype.
[yourSourcetype] INDEXED_EXTRACTIONS=JSON KV_MODE = none TIMESTAMP_FIELDS=systemTime TIME_FORMAT=%d-%m-%Y_%H:%M:%S
Where do I place the props.conf file? I tried making one in $SPLUNK_HOME/etc/system/local and it wants me to be root to create the file. Will this cause any permissions problems? Thank you!
Does your splunk instance running as root ? If not then it should not prompt you to create file as root. You need to create file with same user as splunk is running.
You can create this props.conf in
$SPLUNK_HOME/etc/system/local or if you have any custom app then
None of the events are showing. I created props.conf in
/opt/splunk/etc/system/local as root and saved it as
system_json1. I made sure to restart Splunk after this. However, I do not see this new source type via the GUI. Even the old events that had the time incorrectly processed disappeared.
How are you ingesting data into Splunk ? And configuration which you recently created will apply to new data only, it will not apply to data which is already ingested.
I am ingesting the data into Splunk by using Settings > Add Data > Monitor > Files & Directories. I can change the source type for that data by Settings > Data Inputs > Files & Directories. I clicked on the directory that is causing me problems and changed the source type to
props.conf file and the previous data also disappeared with that.
To be clear, Splunk is still showing that the number of files for that directory increment. For whatever reason, the files are not being processed.
Yes, I am currently on the Test Instance. The data source was new and that is why I am just now addressing the incorrect time formatting. I do not know of a way to change the sourcetype using Monitoring without it affecting both the old and new data as it only allows you to specify one.