Ingest Processor doesn't route logs where I want

[derekmkll]

I am fairly new to Splunk. I am trying to reduce ingestion size of logs that are coming from the Palo Alto Cloud connections. A lot of values come back null or are useless for our applications. I have tried to use the premade Palo Alto filter for log ingestion but it has no effect on pan:threat:cloud or pan:traffic:cloud sourcetypes. I tried eval commands to change the sourcetypes to their noncloud counterparts (pan:threat/pan:traffic) but they still come out unchanged. When I made my own pipeline it does change the data (removing the unused fields) and then sends it off to the default indexer where it is no longer showing under the "Palo Alto Networks FLS LF" source. I assume that since I have converted it to a json splunk doesn't auto recognize it as a Palo Alto log and is indexing it in a different index. I will set up an S3 bucket for the raw logs after this.

Traffic Log Pipeline:

$pipeline = | from $source

    | flatten _raw

    | fields - DestinationDeviceCategory, DestinationDeviceMac, DestinationDeviceOSFamily, DestinationDeviceOSVersion, SourceDeviceProfile, ChunksReceived, ChunksTotal, SDWANSite, LinkChangeCount, DestinationDeviceHost, DestinationUUID, SDWANClusterType, SDWANPolicyName, VirtualSystemName, SourceEDL, _raw, ContainerName, IMSI, SDWANDeviceType, NSSAINetworkSliceType, LinkSwitches, EndpointSerialNumber, DestinationEDL, Tunnel, DestinationDeviceVendor, DestinationDeviceModel, DestinationDynamicAddressGroup, ChunksSent, SourceDeviceVendor, ContainerNameSpace, GPHostID, SourceDeviceOSVersion, NSSAINetworkSliceDifferentiator, SourceDeviceModel, SourceDeviceOSFamily, EndpointAssociationID, SourceDeviceMac, SourceDeviceCategory, SourceDynamicAddressGroup, ParentSessionID, SourceDeviceHost, SDWANCluster, DestinationDeviceProfile, SourceUUID, IMEI, ContainerID, DynamicUserGroupName, HASessionOwner

    | select _time, RepeatCount, SessionStartTime, DestinationPort, HTTP2Connection, sourcetype, DGHierarchyLevel1, DestinationLocation, NATDestination, _kind, PacketsSent, LogSetting, VirtualLocation, index, _forwarder_channel_info, SourceUser, Rule, FromZone, Application, SourceLocation, SessionDuration, Protocol, DestinationUser, PacketsTotal, DGHierarchyLevel2, DGHierarchyLevel3, DGHierarchyLevel4, DestinationAddress, ToZone, ActionSource, DeviceName, SequenceNo, RuleUUID, source, Bytes, SessionEndReason, host, NATDestinationPort, PacketsReceived, DeviceSN, ConfigVersion, ParentStarttime, BytesSent, SessionID, Subtype, NATSourcePort, Action, InboundInterface, SourcePort, URLCategory, OutboundInterface, SourceAddress, TimeGenerated, TimeGeneratedHighResolution, LogType, TimeReceived, BytesReceived, NATSource

    | into $destination;

[henry_collins]

The main issue is not the field filtering itself, but where the pipeline is running and how Splunk is classifying the event afterward. Changing sourcetype in a pipeline won’t necessarily make Splunk treat it as a Palo Alto CIM/add-on event again, especially after modifying _raw.

A better approach is to keep the original sourcetype and remove unwanted fields before indexing, rather than rebuilding the event structure. Also check that the destination index and source type mappings are still tied to the Palo Alto add-on inputs.

Your custom pipeline proves the filtering works, so focus on preserving the metadata (sourcetype, source, index routing) instead of converting the event. You may also want to compare _meta values before and after the pipeline to see what is being lost.