Splunk Search

Suggestions please on how to configure content against a csv updated daily

reswob4
Builder

I'll state my problem first, then some of the posts, apps, and documents I've looked at already....

In AD, we have an OU whose membership changes on a daily basis. I need to create content that compares user activity against the members of that OU.

In our environment, our search head and two indexers are on Linux and our Heavy Forwarder is on a Windows server.

Some of the things I have already done and problems I'm having:

I have a powershell script that I can set up as a daily task on the Heavy Forwarder that can either create a csv with the members of that OU OR have the powershell send each member into Splunk as an event to be entered into the index.

I used the following reference to create a lookup table (http://docs.splunk.com/Documentation/Splunk/6.1.4/PivotTutorial/AddlookupfilestoSplunk) from the output of the aforementioned script, but I'm not sure of the best way to update the table daily since the script runs on the Heavy Forwarder and the search head is on Linux. (OK so I could create a SMB/CIFS share). And when I manually updated the csv file, the subsequent searched didn't pick up the new user name. So I'm not sure if I've created the lookup correctly.

I've looked at the following answers:

http://answers.splunk.com/answers/151962/how-to-configure-scripted-inputs-and-check-if-they-are-runn...

http://answers.splunk.com/answers/149349/indexing-csv-file-that-changes-daily.html

http://answers.splunk.com/answers/169215/how-to-create-a-correlation-using-splunk-data-and.html

because I think they each contain part of the answer I need, but I just haven't put all the parts together in the correct order yet.

I also looked at the Splunk App for Active Directory and the Splunk App for PowerShell. The thought was that if I got the splunk app for AD working, then I could do a direct comparison in my search or if I got the PowerShell app to work then I could directly dump the script output to the local folder. But I'm not sure if I can run them on my search head (which is Linux if you remember).

So I figured that while I'm still doing trial and error, I could at least post my question to see if anyone else has a recipe for how they solved this problem and if I'm missing something.

Thanks.

0 Karma
1 Solution

reswob4
Builder
0 Karma

reswob4
Builder

Sorry, this was answered in this post.

Somehow I created two posts....

http://answers.splunk.com/answers/184463/why-is-my-search-to-match-events-from-a-lookup-not.html

0 Karma

halr9000
Motivator

While I think going the ldapsearch route will make more sense, I'll answer your other question as well.

You can certainly configure the Windows forwarder to pick up a CSV file. Doing so would create events in Splunk--not a lookup table. To do this, you would follow the steps in the indexed field extraction section of the Getting Data In book which deals with "files with headers". Since the forwarder doesn't have a UI, you'll need to edit the inputs.conf and props.conf files. If you are new to this, The easiest route would be to quickly install Splunk on your workstation (you can set the services to manual so they don't run all the time if you want). Then in splunkweb, go to add a new file input, and it'll walk you through the steps of creating the right props and input settings. You can then put those files on the forwarder with very little modification needed (such as the path to the CSV file).

Once you have that part figured out, just have the forwarder watch the file, and it'll update the file as soon as your script writes it out. Seconds later, you can search that data in Splunk. Now, these won't be lookup tables. That may or may not matter for your scenario. Next step for you would be learning the right search commands to compare these fields and so on.

0 Karma

reswob4
Builder

Great. Now, I've seen a post about creating a lookup table from events. If I'm reading the two correctly, I can use your suggestion here to update the lookup table on a daily basis.

(sidenote, the Windows server is being used as a Heavy Forwarder, not a UF. note to self: moar coffee before posting)

So I still can't seem to get any results from my lookups. I deleted what I created through the web and tried directly creating the files via the example in the Knowledge Manager Manual - Configure Field Lookups and the following post:

http://answers.splunk.com/answers/171990/not-getting-field-automatically-from-lookup-table.html

but I still can't seem to get any results from the search.

file: inactive_users.csv

SamAccountName,name

username1,"fullname, full1"

username2,"secondname, full2"

username3,"thirdname, full3"

etc (but lines are single spaced)

transform.conf:

[inactive_users]
filename=inactive_users.csv
max_matches=1
min_matches=1
default_match=Unknown

props.conf:

[user_monitoring]
LOOKUP-inactiveusers = inactive_users SamAccountName OUTPUTNEW name AS fullname

search:

sourcetype="user_monitoring" | lookup inactive_users SamAccountName OUTPUTNEW name | table SamAccountName, name

results:

Nothing

I put the files in /etc/system/lookups, /etc/system/local and in /etc/apps/search/lookups, /etc/apps/search/local

For the transforms.conf file I tried with and without the full patch to inactive_users.csv

Not sure what I'm doing wrong here....

Thanks

0 Karma

halr9000
Motivator

I suggest going with the SA-ldapsearch app (Splunk Support for Active Directory) which I think you have already considered. I am 99% sure that works on any of our supported platforms. I've also sent feedback to our docs team to clarify that this is the case. Then you can do everything from Splunk. You would create a search using the ldapsearch command and either populate a lookup table using the outputlookup command, or include your user activity comparison in the search pipeline to use the realtime data from AD.

0 Karma

halr9000
Motivator

There's an edit button on the top right under the little gear icon. But I went ahead and updated the URL for you.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...