inputs.conf [http://.....] ... View more

@snigdhasaxena I did see those docs, it does not really explain why I'm seeing this behaviour. I have the [queue] maxSize = 500KB by default and the persistent queue set to 1GB, which is larger then the size of the Queue. I don't see the queueSize stanza in the documentation for inputs.conf. I'm assuming it might be a depreciated setting but will put it in my conf anyway and test to see if it makes a difference. ... View more

So yes, from what I can read out of the doc persistent queues will be filled up when the processing pipeline get's filled it, which is something that happens if the data is being streamed in faster that the pipeline can process it but also when the output is failing (like when there is loss of connectivity to the indexer) which is my scenario. Not 100% sure at this point. Will try to find some more info on this. ... View more

OK, I eventually got to the bottom of this. The regex that I have put in there is fine , so is the remaining format of the xml file. I initially discovered that if I create a new sourcetype (and not use "syslog") and get it to use the datetime.xml then it works, so it must have been something in the syslog props definition. I eventually realised it was the MAX_TIMESTAMP_LOOKAHEAD which is set to 30 for syslog but for any new sourcetype (which will inherit the default settings) it will be set to 128. The timestamp in the log example that I have ends on the 43rd character of the event. ... View more

hmm, interesting, without spending too much time looking into why it's giving you .0000 you can just do a round command on the Total eval to get rid of those , so: | eval Total=round(response-submit,0) ... View more

whats is your string? ... View more

Yes, I know that the capture group of the event breaker gets removed but this happens before I even specify the event breaker. So if I take a file and try to upload it to Splunk via the web interface, the preview part that lets you define the line breaker and all other index-time props settings already show the content of the file with no empty lines. ... View more

Here is something that will make this task a lot easier. You can use a REST API call of the following format: https://xx.xx.xx.xx:8089/services/admin/inputstatus/TailingProcessor:FileStatus you can run that as a curl command from CLI or just put that in the web browser. Replace the xxx with the IP address of the host where the Universal Forwarder is installed (or the indexer if its the indexer that is monitoring files directly). This REST API call will present a page that will show you each file that falls under the monitor: stanza and what is the status on it's read. Here is an examply: /opt/splunk/var/log/splunk/metrics.log.2 file position 25000134 file size 25000134 parent $SPLUNK_HOME/var/log/splunk percent 100.00 type finished reading Pretty self-explanatory. If the file has not been fully read yet the percentage will be less then 100. if there is a reason why Splunk hasn't started reading the file (like CRC) the reason will be stated there. Really good for troubleshooting. ... View more

Nice one, thank you. ... View more

I'm just going to leave the question open for now to get some more attention and see how other people dealt with this issue and will eventually vote for an answer. ... View more

Logs below: Oracle Audit logs: Apr 10 00:00:02 XXXXXXX crit Oracle Audit[56361294]: LENGTH: "250" SESSIONID:[8] "24754003" ENTRYID:[1] "1" STATEMENT:[1] "1" USERID:[10] "XXXXXXX" USERHOST:[8] "XXXXXX" ACTION:[3] "100" RETURNCODE:[1] "0" COMMENT$TEXT:[20] "Authenticated by: OS" OS$USERID:[6] "XXXXXXX" DBID:[10] "1731194098" PRIV$USED:[1] "5" SAP db data: Apr 11 16:35:41 XXXXXXXXXX HDB_SYSTEMDB[41189]: 2018-04-11 16:35:41;nameserver;hananode01;DGH;00;30001;SYSTEMDB;;;0;0;Security_Audit;INFO;SELECT;SYSTEM;SYS;M_CONNECTIONS;;;;;SUCCESSFUL;;;;;;;SELECT CONNECTION_ID, STATEMENT_ID, START_MVCC_TIMESTAMP FROM M_ACTIVE_STATEMENTS WHERE (STATEMENT_STATUS = 'ACTIVE' OR STATEMENT_STATUS = 'SUSPENDED') AND START_MVCC_TIMESTAMP > 0 AND SECONDS_BETWEEN(LAST_EXECUTED_TIME, CURRENT_TIMESTAMP) >= ? AND HOST = ? AND PORT = ?,(10, 'XXXXXXX', 30001);125219;;;;;; Cisco logs: Apr 11 16:21:09 XXXXXXXXXXX: 2018 Apr 11 14:21:09 UTC: %AUTHPRIV-6-SYSTEM_MSG: START: ssh pid=22378 from=::ffff:10.167.218.233 - dcos-xinetd[3253] Windows Snare: Apr 9 02:00:01 XXXXXXXXXXXXXXXXX MSWinEventLog 4 Security 4220128 Mon Apr 09 02:00:00 2018 4688 Microsoft-Windows-Security-Auditing -\- N/A Success Audit XXXXXXXXXXXXXXXXXX Process Creation A new process has been created. Creator Subject: Security ID: S-1-5-18 Account Name: XXXXXXXXXXXXXX Account Domain: XXXXXXXXX Logon ID: 0x3E7 Target Subject: Security ID: S-1-0-0 Account Name: - Account Domain: - Logon ID: 0x0 Process Information: New Process ID: 0x6b1c New Process Name: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe Token Elevation Type: TokenElevationTypeDefault (1) Creator Process ID: 0xad54 Process Command Line: Token Elevation Type indicates the type of token that was assigned to the new process in accordance with User Account Control policy. Type 1 is a full token with no privileges removed or groups disabled. A full token is only used if User Account Control is disabled or if the user is the built-in Administrator account or a service account. Type 2 is an elevated token with no privileges removed or groups disabled. An elevated token is used when User Account Control is enabled and the user chooses to start the program using Run as administrator. An elevated token is also used when an application is configured to always require administrative privilege or to always require maximum privilege, and the user is a member of the Administrators group. Type 3 is a limited token with administrative privileges removed and administrative groups disabled. The limited token is used when User Account Control is enabled, the application does not require administrative privilege, and the user does not choose to start the program using Run as administrator. 4220127 ... View more

Yes, that is pretty similar to what we figured out as the most appropriate solution. We were also thinking of getting splunk to extract _time as the timestamp on the event of the original device (not the syslog timestamp that is added , as we saw some delays there) then calculate the difference between _indextime and _time and round it (probably rounding to 30 min as I think the client has devices in India where then have half an hour shift time zones. What we then thought was to export the list of hosts and their time offset into a CSV and based on it add the following stanza to props.conf: [host::Host_name_1] TZ = timezone that matches the offset. That way Splunk would adjust the _time at index time but as you said , we could also create a new field that would be equal to _time + calculated offset and do any timecharts based on that and not the regular timestamp. ... View more

I'll try to provide some anonymized samples. Syslog - we have several syslog server and devices sending data to syslogs from all over the world. The syslog server have correct timestamps configured but the logs that a syslog server will receive might not always be from a device that physically also sits in the same time zone. ... View more

sorry I was away for a couple of weeks. When I have a chance I will look at it but I think you might be right, this might work. ... View more

Why would you want to keep your karma points? ... View more