All Apps and Add-ons

Cannot get sourcetypes to change

shawnfreynolds
Explorer

I am sending paloalto logs to a syslog server which then sets the index to "pan_logs" and the sourcetype to "pan_log" and forwards them onto our indexer/search head. I am able to see the logs on the indexer with the source type of pan_log and the index of "pan_logs" but not able to see the new sourcetypes. it appears like the transforms to change the sourcetypes to there respected values are not changing. looking for help.

0 Karma

mikaelbje
Motivator

Hi,

I just resolved this for a customer where I migrated to a syslog-ng server for Syslog reception instead of logging directly to Splunk.

To troubleshoot I added the following to the destination stanza in syslog-ng to output all the variables so that I could see for myself what variables were populated.


template("PROGRAM=$PROGRAM\nPID=$PID\nBSDTAG=$BSDTAG\nTAG=$TAG\nTAGS=$TAGS\nFACILITY=$FACILITY\nFACILITY_NUM=$FACILITY_NUM\nLEVEL=$LEVEL\nLEVEL_NUM=$LEVEL_NUM\nPRI=$PRI\nPRIORITY=$PRIORITY\nFULLHOST=$FULLHOST\nFULLHOST_FROM=$FULLHOST_FROM\nHOST=$HOST\nHOST_FROM=$HOST_FROM\nLOGHOST=$LOGHOST\nMSGHDR=$MSGHDR\nMSGID=$MSGID\nMSGONLY=$MSGONLY\nMSG=$MSG\nMESSAGE=$MESSAGE\nSOURCE=$SOURCE\nSOURCEIP=$SOURCEIP\nSOURCE_IP=$SOURCE_IP\nSEQNUM=$SEQNUM\nUNIXTIME=$UNIXTIME\nFULLDATE=$FULLDATE\nISODATE=$ISODATE\nDATE=$DATE\nSTAMP=$STAMP\nTZ=$TZ\nTZOFFSET=$TZOFFSET\nSEC=$SEC\nMIN=$MIN\nHOUR=$HOUR\nHOUR12=$HOUR12\nDAY=$DAY\nWEEK=$WEEK\nWEEK_DAY=$WEEK_DAY\nWEEK_DAY_ABBREV=$WEEK_DAY_ABBREV\nWEEK_DAY_NAME=$WEEK_DAY_NAME\nMONTH=$MONTH\nMONTH_ABBREV=$MONTH_ABBREV\nMONTH_NAME=$MONTH_NAME\nMONTH_WEEK=$MONTH_WEEK\nYEAR=$YEAR\nYEAR_DAY=$YEAR_DAY
\n");

It turned out that my issue was due to the fact that syslog-ng adds hostnames to the event. To fix this I added the following to the source stanza for the reception of Palo Alto logs:

udp(port(515) flags(no-hostname));

Which makes it look like this:


source remote_pan_log {
udp(port(515) flags(no-hostname)
);
}

The logs now showed up with all the necessary info I needed. I correlated the values of the variables with the syslog messages I had in the Splunk server before the migration and created the following magic formula:

destination df_splunk_pan_log {
  file("/var/log/splunk/pan_log/$HOST/syslog"
  template("$DATE $MSGHDR$MSGONLY\n") template_escape(no)
  );
}

The pan_log, pan_traffic and pan_threat sourcetype renames as well as the lookups now work like before and I didn't have to change anything in the Splunk App for Palo Alto.

0 Karma

Ayn
Legend

I was debugging something very similar to this at a customer just the other day. Turns out the sourcetype renaming transforms rely on there being 3 CSV fields before the log type field (THREAT, TRAFFIC, ...) whereas the customer's logs only had 2 (received_time and serial_number). You might want to check if this is the case in your setup as well. If so, the fix is to remove one of the groups in the transform in transforms.conf, so for instance the traffic log transform would change from this:

[pan_traffic]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,TRAFFIC,)
FORMAT = sourcetype::pan_traffic

to this:

[pan_traffic]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,TRAFFIC,)
FORMAT = sourcetype::pan_traffic

shawnfreynolds
Explorer

yep the stanza's/program is installed on the indexer --
Do I need to actually install anything on the forwarder? Also, forwarder is running 5.x code which is forwarding to my indexer on 6.x code... could that be a problem?

0 Karma

Ayn
Legend

Oh, sorry, I confused the stanza name in transforms.conf with the one in props.conf.

Where are you doing this btw, on the indexer? Because that's where you should be doing it 🙂 Also you need to restart Splunk after doing any changes to these transforms.

0 Karma

shawnfreynolds
Explorer

Thats what it came with in the default install -- i believe the the FORMAT=sourcetype::pan_traffic is to change sourcetype to pan_traffic if it hits the regex?

0 Karma

Ayn
Legend

It seems you modified the stanza - right now the stanza sourcetype is "pan_traffic", but it should be "pan_logs" because that's the sourcetype with the events that need to have their sourcetype changed.

0 Karma

shawnfreynolds
Explorer

My logs look like this:
Jan 10 14:54:13 host 14:54:13,001801006057,TRAFFIC,end,1,2014/01/10 14:54:12,1.1.1.1,2.2.2.2,0.0.0.0,0.0.0.0,Permit Into DMZ,,,ping,vsys1,Inside,ANS Test,ae1.1,ae1.2,syslog.domain.com,2014/01/10 14:54:12,240135,1,0,0,0,0,0x4000,icmp,allow,204,102,102,2,2014/01/10 14:54:06,0,any,0,6350272,0x0,United States,10.0.0.0-10.255.255.255,0,1,1

transforms.conf for 'traffic', located in
\etc\apps\SplunkforPaloAltoNetworks\local
[pan_traffic]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,TRAFFIC,)
FORMAT = sourcetype::pan_traffic

still no sourcetype getting changed.

0 Karma

shawnfreynolds
Explorer

I was thinking the same thing by changing that regex in the transforms.conf and it still doesn't seem to change my sourcetype. I wasn't sure if there was something 'special' that I had to do because it was being received via a forward instead of directly off the appliance.

0 Karma

Ayn
Legend

Also, your modified transforms should go into a transforms.conf in the app's local directory instead of default!

0 Karma

Ayn
Legend

Also if this is the same problem you're experiencing you're going to want to change the field extractions as well, because they're delimiter based and will be thrown off if the number/order of fields in the extraction definitions do not match what's in your log data. Specifically the first field that is called "future_use1" had to be removed in the setup I encountered this problem in.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...