- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Explain this transform
v4.3 sles 11.1
can you explain for me this transform
[csafields]
REGEX = ^[^\|]+\|([^\|]+)\|([^\|]+)\|([^\|]+)\|([^\|]+)\|([^\|]+)\|([^\|]+)
FORMAT = nbtname::$1 ip::$2 ruleid::$3 code::$4 remotetime::$5 alert::$6
i get the FORMAT part (which doesnt work correctly with my event data), but what is this regex? what does the \| mean?
here's the raw event data
2012-04-05 15:38:29 syslog.prod.org [UDP: [10.219.0.134]:1086->[10.222.1.253]]:
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (124678725) 14 days, 10:19:47.25 SNMPv2-MIB::snmpTrapOID.0 = OID: CSAMC-SNMPv2-MIB::csaTraps.1 CSAMC-SNMPv2-MIB::eventID = INTEGER: 10329635 CSAMC-SNMPv2-MIB::ruleID = Wrong Type (should be INTEGER): NULL CSAMC-SNMPv2-MIB::hostName = STRING: "hostB.prod.org" CSAMC-SNMPv2-MIB::eventTime = STRING: "2012-04-05 15:38:29.617" CSAMC-SNMPv2-MIB::severityCode = INTEGER: 2 CSAMC-SNMPv2-MIB::eventCode = INTEGER: 164 CSAMC-SNMPv2-MIB::processName = STRING: "<remote application>" CSAMC-SNMPv2-MIB::fileName = Wrong Type (should be OCTET STRING): NULL CSAMC-SNMPv2-MIB::sourceIPAddress = Wrong Type (should be IpAddress): NULL CSAMC-SNMPv2-MIB::destinationIPAddress = Wrong Type (should be IpAddress): NULL CSAMC-SNMPv2-MIB::eventText = STRING: "The process '<remote application>' has triggered too many log records in the last few minutes. Further messages will be logged at a decreased rate for 10 minutes." CSAMC-SNMPv2-MIB::hostID = INTEGER: 209 CSAMC-SNMPv2-MIB::currentHostIPAddress = Wrong Type (should be IpAddress): STRING: "10.132.194.158" CSAMC-SNMPv2-MIB::hostOSType = STRING: "W" CSAMC-SNMPv2-MIB::sourcePort = Wrong Type (should be INTEGER): NULL CSAMC-SNMPv2-MIB::destinationPort = Wrong Type (should be INTEGER): NULL CSAMC-SNMPv2-MIB::eventType = STRING: "Administrative" CSAMC-SNMPv2-MIB::ruleDescription = Wrong Type (should be OCTET STRING): NULLCSAMC-SNMPv2-MIB::ruleModuleID = Wrong Type (should be INTEGER): NULL CSAMC-SNMPv2-MIB::ruleModuleName = Wrong Type (should be OCTET STRING): NULL CSAMC-SNMPv2-MIB::buttonCode = Wrong Type (should be INTEGER): NULL CSAMC-SNMPv2-MIB::userName = STRING: "myDOMAIN\\WSecGat_Px" CSAMC-SNMPv2-MIB::flags = INTEGER: 0
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (124679186) 14 days, 10:19:51.86 SNMPv2-MIB::snmpTrapOID.0 = OID: CSAMC-SNMPv2-MIB::csaTraps.1 CSAMC-SNMPv2-MIB::eventID = INTEGER: 10329637 CSAMC-SNMPv2-MIB::ruleID = INTEGER: 1374 CSAMC-SNMPv2-MIB::hostName = STRING: "hostA.prod.org" CSAMC-SNMPv2-MIB::eventTime = STRING: "2012-04-05 15:38:29.999" CSAMC-SNMPv2-MIB::severityCode = INTEGER: 2 CSAMC-SNMPv2-MIB::eventCode = INTEGER: 179 CSAMC-SNMPv2-MIB::processName = Wrong Type (should be OCTET STRING): NULL CSAMC-SNMPv2-MIB::fileName = Wrong Type (should be OCTET STRING): NULL CSAMC-SNMPv2-MIB::sourceIPAddress = Wrong Type (should be IpAddress): NULL CSAMC-SNMPv2-MIB::destinationIPAddress = Wrong Type (should be IpAddress): NULL CSAMC-SNMPv2-MIB::eventText = STRING: "The 'Service Control Manager' service logged event code 7036 into the system event log: The LiveUpdate service entered the running state. " CSAMC-SNMPv2-MIB::hostID = INTEGER: 2206 CSAMC-SNMPv2-MIB::currentHostIPAddress = Wrong Type (should be IpAddress): STRING: "10.10.10.10" CSAMC-SNMPv2-MIB::hostOSType = STRING: "W" CSAMC-SNMPv2-MIB::sourcePort = Wrong Type (should be INTEGER): NULL CSAMC-SNMPv2-MIB::destinationPort = Wrong Type (should be INTEGER): NULL CSAMC-SNMPv2-MIB::eventType = STRING: "NT Event log"CSAMC-SNMPv2-MIB::ruleDescription = "" CSAMC-SNMPv2-MIB::ruleModuleID = INTEGER: 280 CSAMC-SNMPv2-MIB::ruleModuleName = STRING: "CSA Service Monitoring" CSAMC-SNMPv2-MIB::buttonCode = Wrong Type (should be INTEGER): NULL CSAMC-SNMPv2-MIB::userName = Wrong Type (should be OCTET STRING): NULL CSAMC-SNMPv2-MIB::flags = INTEGER:
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Just wrote a massive reply and it doesn't fit in the box! Read my re-re-edited answer
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I don't think this is the transform that's applied to the sample data you provided, for that reason - I don't see any pipes in there either.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
sorry, i am on 4.3.1, my keyboard didnt get the .1 in there.
and hmmm, i dont see any pipe chars in my raw data that can be used for the transform, however, when i do a open search on my csa index i get lots of fields on the left with data (albeit the values for each field have more data in them then needed, but it is extracting fields somehow). the delimiter in my case looks like it should be a tab \t or \s{2,10}
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Also, if you are on v4.3 still you should consider upgrading to v4.3.1. There were some reasonably major issues in 4.3 that are resolved in 4.3.1; http://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/4.3.1
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
the \ is an escape character so the regex is saying;
^[^\|]+
<-- while character is not | keep eating up all characters. \|
means | the character and not OR.
Ok, so if you look at this part;
([^\|]+)\|([^\|]+)\|([^\|]+)
Each part within brackets is a "group". The $1, $2 refers to group 1, group 2 etc. So everything within those brackets is what will be assigned to that group. Lets take the above snippet as an example. Lets say the data looked like this;
blah blah | foo | monkey!
Then the above regex would assign "blah blah " (note the whitespace caught at the end too) to the fieldname for $1. " foo " to the fieldname for $2 and so on. Anything outside of brackets is just to progress through an event, it may be that the last 3 sets of values enclosed by |'s are interesting so you skip through the previous ones without grouping.
EDIT AGAIN:
Hmm, just re-read your comment.. There must be a pipe character or something differentiating. Could you perhaps paste some example data? (Best way would be to edit your original question to paste it clearly as code)
EDIT EDIT AGAIN:
How well is the data being extracted? I just tried it in a test instance and found that Splunk automatically extracted some values (although not very good ones, just lots of Wrong and INTEGER). If thats what you're seeing then thats just Splunk trying to be clever and pull out Key/Value pairs. If you are sure something is extracting it then try the command, ./splunk cmd btool props list
and also try transforms instead of props too to check for any other extractions that might be taking effect that look like they fit. (Saves looking through all the config files). If you stick a --debug on the end of that command it will also tell you what app is applying it
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
i'll look into this further as suggested, but i did search all other transforms and props files looking for CSA related entries and only found related items in the CSA app folder, etc. and yes, the extractions i have do have "Wrong" and "INTEGER" included in the values, which leads me to believe i need to write my own transform regex/format to map the event data to variables. should not be too hard to do.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Ah right, look at my edited answer for a fuller explanation (although give me a moment if you look now 🙂 )
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
but how does the regex actually group real matches for the FORMAT statement? the groupings have a \| between thme, but my data has no | char so how does the groupings fall into matches?
