Use field extractor with a search?

[AlexeySh]

Hello,

I am wandering to know if there is a way to apply a field extractor not to a source type but to a search.
I’d like to employ a delimited-based field extraction only for specific condition. Like

Sourcetype=xxx fied_1=abcd

Thanks for the help.

Regards,
Alex.

[gaurav_maniar]

Hi,

you can use 'rex' command with your query to extract fields at search time and provides fields extraction as well. The only limitation is, it does not provide any delimiter based extraction, you have to write the regex.

rex command reference - https://docs.splunk.com/Documentation/Splunk/7.2.0/SearchReference/Rex

Please accept the answer, it it solves your problem.

[AlexeySh]

Hello @gaurav_maniar,

If we want to extract all fields, the rex command became too complicated and doesn’t work properly (or it could be our lack of skills as well). That’s why we want to use delimited-based field extraction.

[gaurav_maniar]

As your data is already delimited, writing a field extraction with rex command will be very easy and it will work properly if your regex is correct without any problem.

If you go with filed extractor, it will directly apply it to sourcetype and as of now no delimited field extraction is available with rex command.

We can help, if you provide some sample logs

[AlexeySh]

We will really appreciate the help.

Here’s some simple logs, I just modified some private information, like customer ID or domain name.

Nov 2 12:50:14 sc-1456400473-logforwardercomp-5bce225a53cefb004074a882-59fm5sn logforwarder[24] analytics,agent_data,,AgentTimelineEvent,hash,2018-11-02T12:49:45.267329700Z,2018-11-02T12:50:08.656Z,2018-11-02T12:49:45.267329700Z,60,,TrapsAgent,1111111173,6857076101111111111,coreop-f-prodb2-mnmauto123123123123-1234.prod.brz,2.0.6,70,1,6aaaaaaaaaaaa5da86ada7b4c6b01504,1,0,6.1.7601,1,123.123.123.123,wks123,abcdef.fr,,,5.0.3.38921,36-4887,0,7777777777aaaaaaaaaa157092d94eb18c2a73a0a49beeaaaaaaaaaaa30e86a2,dll,,2018-11-02T12:49:45.267329700Z,comdlg32.dll,\?\C:\Windows\SysWOW64\,485888,"{""contentVersion"":""36-4887"",""result"":""Benign"",""trusted"":""None"",""publishers"":[""Microsoft Windows""],""resultId"":0,""trustedId"":0}",0,0,16159

---

Nov 2 08:59:06 sc-1456400473-logforwardercomp-5bce225a53cefb004074a882-59fm5sn logforwarder[24] threat,threat,,AgentSecurityEvent,2018-11-02T08:16:15.144216600Z,2018-11-02T08:58:55.998Z,2018-11-02T08:16:15.144216600Z,60,,TrapsAgent,1111111173,6857076101111111111,coreop-f-prodb2-mnmauto123123123123-1234.prod.brz,2.0.6,70,1,6aaaaaaaaaaaa5da86ada7b4c6b01504,1,0,6.1.7601,1,123.123.123.123,wks456,abcdef.fr,0,2,5.0.3.38921,36-4887,0,a1866535ef474c2f869865f09x111111,COMPONENT_EPM_J01,ExploitModules,CYSTATUS_JIT_EXCEPTION,,reported,0,,,0,0,"[""CreateProcessA"",""2""]",0,-1,0,"[{""pid"":6952,""parentId"":2724,""exeFileIdx"":0,""userIdx"":0,""commandLine"":""\""C:\Users\user_1234\AppData\Abcdabcd\aaaaaaaaaaaaaa\firefox.exe\"" ""}]","[{""rawFullPath"":""C:\Users\user_1234\AppData\Abcdabcd\aaaaaaaaaaaaaa\firefox.exe"",""fileName"":""firefox.exe"",""sha256"":""70225F14A28007815B0410B1F41F7EA6A16B6329FD69F7EC0638A1A1A1A1A1A1"",""fileSize"":531408,""signers"":[""Mozilla Corporation""]}]","[{""userName"":""user_1234"",""userDomain"":""abcdef.fr""}]",[],Memory Corruption Exploit

---

The log’s format is described on Paloalto website.

Thanks for the help!

[kamlesh_vaghela]

@AlexeySh

Can you please share more information like sample events and expected results??

[AlexeySh]

Well, basically it’s a Paloalto Traps logs. You can find its log format on Paloalto website. As you can see there are 4 log types and they are slightly different, 1-2 fields more or 1-2 fields less. So you can’t apply field extraction to sourcetype directly, you have to know logs format as well (‘recordType’, the first field).

Unfortunately our Traps logs come to Splunk in a pretty messy format: we have some additional information in the beginning of each event. So we decided to create an independent index and sourcetype for it. By using rex transformation we can extract a “real” value of ‘recordType’ field. But once we have it, we’d like to just use a delimited-based field extraction by comma to extract all other fields for each log type.