topic Re: rex expression does not work in curl in Splunk Search

rex expression does not work in curl

Sukisen1981 — Wed, 01 May 2019 08:31:55 GMT

I have a simple search on a text pad, like this index=text|rex field=_raw "ApplicationRegistry-(?<text>.*)" max_match=0 |table source,sourcetype,text
This works in web UI but does not work with curlcurl -ku admin:admin https://192.168.1.4:8089/servicesNS/admin/search/jobs/export --data-urlencode search=“search index%3Dtext%7Crex%20field%3D_raw%20"ApplicationRegistry-(%3F.*)"%20max_match%3D0%20%7C%20table%20host%2Csource%2Csourcetype%2Ctext” -d output_mode=json
However, if i remove the rex expression and just table source,host,sourcetype the below command works

curl -ku admin:admin https://192.168.1.4:8089/servicesNS/admin/search/search/jobs/export -d search=“search index%3Dtext%20%7C%20table%20host%2Csource%2Csourcetype” -d output_mode=json

Why is curl failing with rex? i receive an error - The system cannot find the file specified.
I have looked at many answers here, replace -d with this or that etc. etc. but ot does not work

Re: rex expression does not work in curl

harsmarvania57 — Wed, 01 May 2019 09:32:12 GMT

Hi,

Can you please try below command

curl -ku admin:admin https://192.168.1.4:8089/servicesNS/admin/search/search/jobs/export -d search="search index=text|rex field=_raw \"ApplicationRegistry-(?<text>.*)\" max_match=0 |table source,sourcetype,text" -d output_mode=json

Re: rex expression does not work in curl

Sukisen1981 — Wed, 01 May 2019 10:00:58 GMT

Tried and received the same error message. I can see that you are trying to escape the regular expression, I guess its all on how to escape / treat the regex as a regex

Re: rex expression does not work in curl

Sukisen1981 — Wed, 01 May 2019 10:01:32 GMT

BTW does the splunk version play a role? I am on 6.6.3

Re: rex expression does not work in curl

harsmarvania57 — Wed, 01 May 2019 10:16:30 GMT

Might be, I am testing in my lab environment and above command which I posted is working fine on Splunk 7.1.2

Re: rex expression does not work in curl

Sukisen1981 — Wed, 01 May 2019 10:54:09 GMT

the thing is the customer is on 6.6.3 and I can't just tell them to upgrade to 7.x version...
As far as an ideal situation goes my and your curl commands should work just fine but it is not 😞

Re: rex expression does not work in curl

chris_barrett — Wed, 01 May 2019 12:13:37 GMT

Try the example found here and see if it works - https://docs.splunk.com/Documentation/Splunk/7.2.6/RESTREF/RESTsearch#search.2Fjobs.2Fexport

I do wonder if the asterisk not being escaped/encoded in your test is the issue?

Re: rex expression does not work in curl

Sukisen1981 — Wed, 30 Sep 2020 00:24:05 GMT

Hi @chris_barrett and @harsmarvania57 . To make things more generic I have now used the audit command so that we can all replicate the issue. So, this works -
curl -ku admin:admin https://192.168.1.5:8089/servicesNS/admin/search/search/jobs/export -d search=“search index="_audit"|table source,host,text” -d output_mode=json

In the splunk web UI this works -
index="_audit"|rex field=_raw "Audit:(?.*)" |table source,host,text
But if i use curl to fetch the rex, it fails. So this does not work

 curl -ku admin:admin https://192.168.1.5:8089/servicesNS/admin/search/search/jobs/export -d search=“index="_audit"|rex field=_raw "Audit:(?<text>.*)" |table source,host,text" |table source,host,text” -d output_mode=json

I know its about escaping/writing the regex inside the curl command, but what is the way to do that?
I did refer to this answer here - https://answers.splunk.com/answers/495305/why-is-rex-field-not-producing-results-when-used-i.html
But this does not work, curious to know if you guys try the _audit index with the rex as stated above are you receiving results from the curl command?

Re: rex expression does not work in curl

codebuilder — Wed, 01 May 2019 19:35:41 GMT

Replace your opening and closing quotation marks with single ticks to define the contents as a literal string.

 curl -ku admin:admin https://192.168.1.4:8089/servicesNS/admin/search/search/jobs/export -d search='search index=text|rex field=_raw \"ApplicationRegistry-(?<text>.*)\" max_match=0 |table source,sourcetype,text' -d output_mode=json

Then it should work for you.

Re: rex expression does not work in curl

Sukisen1981 — Thu, 02 May 2019 06:54:18 GMT

does not work. I receive the same message in CMD

Re: rex expression does not work in curl

DavidHourani — Thu, 02 May 2019 07:56:45 GMT

Hi @Sukisen1981,

This is working for me :

curl -ku admin:admin https://127.0.0.1:8089/servicesNS/admin/search/search/jobs/export -d search='search index="_audit" user=* |head 1 | rex field=_raw "user=(?<text>[^\s,]*)" | table source text'  -d output_mode=json

and this as well :

curl -ku admin:admin https://127.0.0.1:8089/servicesNS/admin/search/search/jobs/export -d search='search index="_audit" user=* |head 1 |rex field=_raw "Audit:(?<text>.*)" | table source text'  -d output_mode=json

Could you please try and let me know if it works for you. Also which version of Splunk are you currently
using ? It could be a problem specific to your current version.

Cheers,
David

Re: rex expression does not work in curl

harsmarvania57 — Thu, 02 May 2019 08:14:19 GMT

Maybe noob question and not relevant but still asking, are you using curl on Linux or on Windows ?

Re: rex expression does not work in curl

Sukisen1981 — Thu, 02 May 2019 11:01:09 GMT

Receive error 'head' is not recognized as an internal or external command, in both cases,once again I am on splunk 6.63, is that having any effect?
It is getting a bit irritating now, for the moment we are managing without the rex fields but we really wanted to have the regex fields to be extracted through splunk before using the API JSON output in the downstream UI system...

Re: rex expression does not work in curl

DavidHourani — Thu, 02 May 2019 11:17:37 GMT

I'm testing on the same version now, working on linux.. What distro are you using ? you can get rid of the head 1 if you like it's just for reducing the number of events. Could you try it on your splunk GUI first, run the command with the exact user you are using for the curl command see what it does and if it gives results. If it's working on GUI and you're using admin for CLI it should work 🙂
Also could be a problem with the path for your curl command if you're using windows..

Re: rex expression does not work in curl

Sukisen1981 — Thu, 02 May 2019 16:17:23 GMT

Hi @DavidHourani - I am on windows so I need to replace the single quote after the first appearance of the word 'search' with double quotes,but that does not still solve the original error message.
It is a problem with the path in windows for sure - the message is The system cannot find the file specified. But what is the solution :)?
The rex works in the UI , does not really matter if the rex returns results or not, it should not error out in the UI and that's not happening in the UI

Re: rex expression does not work in curl

Sukisen1981 — Thu, 02 May 2019 16:17:48 GMT

I am using windows

Re: rex expression does not work in curl

DavidHourani — Thu, 02 May 2019 16:41:36 GMT

what you're facing is a windows related problem with Curl, not a Splunk problem, what you could do is write a small script that ads the query in a variable and send the variable instead of typing the search 🙂

Re: rex expression does not work in curl

Sukisen1981 — Fri, 03 May 2019 17:46:25 GMT

Hi @DavidHourani - Is there any pointer to the script you are referring to? I realize that this is a windows specific issue , but alas I can not ask the client to use Linux just for this issue.
I have posted the same question on stack today, awaiting answers from from bash experts over there and also trying to install cygwin.
On this not being a splunk issue, I disagree - No where in the curl docs that splunk mention something like this can occur whilst using windows, and well, a major product vendor can not provide features that give issues on one of the most widely used OS in the world.
But that is besides the point, thanks for your reply I am now investigating other alternatives to make this work on windows. Once again, many thanks

Re: rex expression does not work in curl

DavidHourani — Sat, 04 May 2019 08:32:24 GMT

Hi @Sukisen1981,
You're welcome, yeah I agree on your point of view about Windows/Splunk issues. Problem is that not all commands used for Linux and then re-adapted for Windows are integrated in the same way so escaping characters doesn't always work as you'd expect it. Cygwin might solve the problem for you as well (Let me know if it does 🙂 ).

As for the scripts, have a look here :
https://linuxhint.com/curl_bash_examples/
The idea is to put the whole Splunk query in a variable and reuse the variable name in the curl command.

Re: rex expression does not work in curl

Sukisen1981 — Sat, 11 May 2019 08:23:43 GMT

Hi @DavidHourani -
Cygwin works and ultimately of course I wanted to eventually use the REST api edn points on 8089 , wrote a small python script of 5-6 lines and it works.
Many thanks for your help , I am gonna up vote a couple of your comments since you really digged into this