All language subtitles for 006 Using OSINT Tools (OBJ 2.1)

Instructor: In this lesson,

I'm gonna show you how to use a couple of the open source

intelligence tools specifically.

We're gonna take a quick look at metagoofil,

the Harvester and Recon NG.

Now first let's look at metagoofil.

Let's say for example,

I want to search Udemy's website for any Word,

PowerPoint and PDF files,

and I wanna limit that search to the first 50 documents

found, download up to 25 of those files,

put them into a working directory,

and then I can save those 25 files in there and output the

results to a file that tells me everything we found.

I can do that pretty easily by using metagoofil.

Now first though,

we have to install metagoofil because it's not installed by

default in our version of Cali Linux,

which at the time of this recording is 2021.4A.

To do that,

you can try to just run metagoofil and will ask if you wanna

install it or you can use pseudo app,

get install metagoofil.

I'm gonna do it the easy way and just type in metagoofil

first, now, if it's installed,

it would give me the usage and syntax of what I would have

to do.

If it's not,

it's gonna gimme an error message and in this case it's not

installed, but Cali Linux is smart to say,

Would you like me to install it?

In this case,

I'm gonna say yes and let them do all the hard work of

installing it for me,

you do need to put in your password for the pseudo user,

which is the root user,

and the case of the default virtual machine that comes from

cali.org.

It's just gonna be KALI or Kali.

It's gonna go through, it's gonna install, it's gonna say,

Would you like to install it?

Say yes, and there we go.

It's downloading it and off it goes, installing it,

and in about another 10 seconds, boom, we're done.

Okay, we're back to the command prompt.

So at this point we can now use this tool.

Now, just to bring myself back up to the top of the screen,

I'm just gonna type in the word clear and in a Linux

terminal that just brings you right back to the top of your

screen.

Now what I wanna do is go ahead and do that Udemy search

that I was talking about.

We're gonna go ahead and type in metagoofil,

then dash D for the domain Udemy.com is the domain.

We're gonna be searching dash T for the file type,

and then the file types we wanna search for.

I'm gonna do a doc, a, docx, a PowerPoint,

or PPT, a PPTX,

which is the newer version of PowerPoint,

and let's go ahead and throw PDF in there for good measure.

Then we're gonna do dash L for limiting the number we want,

and I'm just gonna go ahead and make it up to 50 of each.

And then I'm gonna do dash N,

which is the number I wanna download,

and I'm gonna do up to 25.

And then I wanna put dash O for my output directory.

And in this case,

I'm just gonna go ahead and call it Udemy files and it'll

make a new directory in the directory I'm currently in

called Udemy files.

And then dash F and dash F is gonna be to say I want to

create a file that gives me the results in one combined

HTML file.

Now once I do this,

I'm just gonna go ahead and hit enter and it's gonna go off

and start that search.

You could see here that it is gonna go ahead and download

those files.

It's gonna save them into that new file folder that I just

created, which is called Udemy files.

It created the folder and now we're out there searching for

50 doc files and we're gonna wait 30 seconds and then it's

gonna try the next search,

and it's gonna continue to do this through each of those

five file types.

Now I'm gonna go ahead and fast forward through this search

because it does take a little bit of time to do all the

searching and we're gonna come back once the search is

complete, Okay, our search is complete, or in this case,

our search actually failed.

Now why did it fail?

Well,

Google actually will block you if you're making too many

requests because it detects that you're a bot,

and in this case it did just that.

To be able to overcome this,

you can actually install something like tour into your Linux

machine and then you'll route all your traffic through tour.

So you're coming from different IP addresses between each

and every one of your searches.

But for the purposes of this demonstration,

this gives us the idea of what we were trying to do.

Now, notice here at the top, we did search

not just udemy.com,

but all of its Sub domains as well.

So we found that they were actually trying to find some

files and the about.udemy.com site,

the investors.udemy.com site and other things like that.

All right, now that we did that,

I'm gonna go ahead and clear my screen to bring me to the

top again.

Okay, once we clear the screen,

I'm just gonna do the LS command,

which will list out the files and you'll see there there is

a text file with the HTML links based on what it found from

all the metadata it was searching,

and you'll also see there is one called Udemy Files,

which is a folder, and if I go into the Udemy files,

you're gonna see that we do have a couple of files here that

were downloaded as part of our reconnaissance

using metagoofil

and we can go through those files to look at what we found.

The next tool we're gonna use is known as the Harvester,

and the harvester is spelled as one word with a capital H.

Now the harvester is a wonderful tool and it's used to

gather emails, subdomains, hosts, employee names,

email addresses, PGP key entries,

open ports and service banners off of the servers.

Now again,

the Harvester is another command line tool and it's pretty

easy to use once you learn the syntax.

It's very similar to what we just used with metagoofil.

Now when we wanna use this command,

we can just type in the harvester and then hit enter and

it's gonna tell us how to use it.

Here you can see the usage on the screen.

So to use the tool,

we're just gonna go ahead and type in the harvester dash D,

the domain name that we wanna search.

In this case,

I'm gonna use udemy.com dash L to limit the number of

searches we wanna return.

I'm gonna use five as the number of things I wanna return,

and then I want to go ahead and enter in dash B and this

search engine I want to use, for instance,

I'm gonna use Google.

I could just as easily use something like LinkedIn or Bing

or something else if I wanted to as well.

Once you're done with that,

go ahead and hit enter and it's gonna run off and run that

command.

You'll see here we see the banner for the harvester,

we see the fact that it's going in, searching Google,

we're getting zero results, no IP's, no emails,

and one host that was found with two IP addresses.

Now, why is that?

Well, again,

I just did the scan from this computer when I was using

metagoofil,

and so Google at this point is already blocking me because

they think that I am a bot or I'm doing something bad,

and so therefore they're blocking me.

Again, if you go ahead and change your IP addresses,

you go ahead and do something like Tour.

It's gonna keep you from having this same problem.

Let me go ahead and clear the screen.

The next thing we're gonna go ahead and do is work with

Recon NG.

Now, Recon NG is a great tool,

but it is a little bit more complicated than the two I just

showed you.

So to start up Recon NG,

you just type in Recon NG and hit enter.

When you do that,

it's gonna load up and now you're in a special command

prompt within the recon NG environment.

Notice here we're in Recon NG and we are in the default

workspace.

Now, recon NG is a wonderful web reconnaissance

framework and it

works a lot like meta exploit does for exploits,

and this social engineer toolkit works for social

engineering.

It brings a lot of tools and a lot of capabilities into one

place, but because of that,

it is a little bit more complicated.

Now also,

the other thing to note is a lot of the things you're gonna

find online for recon NG are gonna be wrong and they're not

gonna work.

The reason for that is there was a switch in a lot of the

syntax going between version four and version five.

You could see here I'm operating with version 5.1.2,

which is the latest at the time of this recording.

So all the things I'm gonna show you now are based

on that version.

These will not work in the older version four,

and the old version four commands will not always work in

the newer version five.

So keep that in mind.

If you're looking at any tutorials online or YouTube videos,

if they're older than about 2020,

you may run into some issues there.

Now to use this,

we're gonna launch into Recon NG just like we did here and

now I like to personally set up my own workspace instead of

using the default workspace.

This gives me almost a place to store and save all my

different commands and tools and information that I find in

one area.

And so to do this,

we are gonna use the workspaces command.

Now, if you don't know any of the commands in recon NG yet,

which you probably don't,

you might wanna first type in help.

When you type in help,

it'll list out all the commands you can use,

and you'll see there that we have one called workspaces at

the bottom of the list,

and this is used to manage workspaces.

A workspace is just a defined area to keep your different

information from different engagements into different

buckets so they're not mixing between clients.

So what I'm gonna do is I'm just gonna type in the word

workspaces, and then from workspaces, if I hit enter,

it's gonna tell me I didn't give enough syntax because we

didn't know what to do with it.

Yet here we can do a create list load or remove.

In our case, we can see if there's any workspaces already,

and there shouldn't be because this is a brand new

installation, but by doing that,

we do workspaces space list and hit enter.

And you see we only have the default one.

I'm gonna go ahead and create one called Dion,

and to do that, we're just gonna type in workspaces, create,

and then I'm gonna use the folder name of Dion.

Now, if I go ahead and do workspaces list,

you're gonna see two Default and Dion.

All right.

Now that we've done that,

we can go ahead and select that workspace to work in it.

And in this case, because I just created that workspace,

it automatically put me into the Dion Workspace.

But let's say I had another one because I was gonna do

workspaces create, Udemy.

Now I'm gonna have three different workspaces.

Oh, I actually type that wrong,

so make sure you type it correctly.

Everything in Linux is case sensitive and obviously spelling

sensitive.

I forgot the S on workspace and that's why I got that error.

So here we go.

I listed it out and you see there are three workspaces.

Now, Dion, Udemy, and default.

When you create a new workspace,

by default it moves you into that workspace.

So you'll notice when I created Dion right next to recon NG,

there was this thing that said bracket, Dion bracket,

and that told me that I was in the Dion space,

and you could see that just below the first table where I

entered workspaces create Dion,

it moved me into that Dion workspace.

Then when I created the Udemy workspace,

it then changed me from Dion into Udemy.

But let's say I wanted to go back into Dion,

how would I do that?

Well, to do that,

we're just gonna type in workspaces load,

and then the name of the workspace.

In my case, it's Dion.

Now you can see I am back into Recon NG inside the

workspace Dion.

All right,

the next thing we have to do is we have to have some modules

installed to be able to do some functionality inside of

Recon NG.

Now, by default,

there is no modules installed when you first get Recon NG,

and this is a brand new install,

and if you just loaded up your virtual machine,

you have a brand new install as well.

And so these are things we have to work through.

So what we're gonna do is we're actually gonna go into the

module section, which is another one of those key words.

Again, if you get lost at any time,

just type in help and you'll notice there we have a modules

command and it says that it interfaces with

installed modules.

So to check if there's any installed modules,

I'm just gonna type in modules and then I would use the

command search.

Now, if you don't know any of the commands for modules,

again, just type in modules and hit enter.

There is your different options.

You can load, reload,

or search.

Search is essentially like listing,

but you can list all of them.

If you just type in search and hit enter,

or you can actually search for a keyword because there are

many modules and maybe you just want a specific module for a

specific use case.

If I go ahead and hit search and hit enter,

you're gonna see that I have no modules found and it has a

red error there.

That's because I haven't installed anything yet.

And so now we need to go and find some modules.

How do you do that?

Well, looking back up at the help area,

you'll see a command called Marketplace.

This interface is with the module marketplace,

and this is where you can search for and download a single

module or all the modules.

Now, some modules you're gonna find are gonna require you

to get an API key to be able to associate with the service.

For example, there's a module to be able to search Twitter,

and Twitter requires you to register with them as a

developer to get an API key so you can then make that API

connection from Recon NG into Twitter and start searching

their stuff.

Now, to make things easy for our demonstration,

we're not gonna go into that and I'm gonna pick a module

that does not require API keys.

If you're gonna use Recon NG for real, go online,

look at the Recon NG manual.

You'll be able to walk through how to do all of those things

because there's a lot of capability in this tool

for the exam.

You do not need to know how to use Recon NG.

I just wanted to show you so you can get comfortable

with it,

so you can use it in the real world a little bit and then

take it from there.

Now, what we wanna do first is we wanna find a module

that we can use.

The one I'm gonna use is known as recon slash domain

contacts slash WHOIS POCs.

I know this one doesn't require an API,

and the purpose of this is for us to be able to use this to

look up the WHOIS data for different domain names.

Now, to do this,

we're just gonna use the command marketplace and then we're

gonna use the term install.

Now, just like before,

if you don't know how to do something,

you can simply type in marketplace and enter and it will

give you the syntax,

and then you're gonna put the one that you're looking for.

In my case, I know exactly which one I'm looking for.

It's recon slash domains,

dash contacts slash WHOIS underscore POCs.

Hit enter and it's gonna go ahead and install that module.

You'll see right here,

it installed the module and then it reloaded it.

If I wanna validate that actually happened,

I can do that by using the module search command like I did

before to show there was no modules found.

So let's go ahead and do that.

And now instead of no modules being found,

I see that I have one module found.

It's under the recon category,

and it is the one I just installed.

Now, if you wanted to install all of the modules,

and there are a lot of them out there,

if you wanna see them all,

just type in marketplace and then hit search,

and then hit enter.

And you'll see there is a bunch,

there are so many that is actually going off of my screen.

And so if you wanted to actually scroll up here,

you can go through and see that there are a ton of different

modules, and all of these have different functionality.

If you wanna know what each one does,

you can actually look them up inside of Recon NG using the

info command,

and you'll be able to learn more about those particular

tools.

Or again, go online,

go ahead and Google or Bing or Duck Duck Go or whatever your

favorite search engine or choice is and look up those and

you'll figure out which ones they are, what they do,

and which ones may be helpful in your reconnaissance.

Now, if you wanna install all of these, you certainly can,

and to do that, you just type in marketplace install all,

and if I hit enter right now,

it's gonna go and install every single one of those.

Now, the reason I'm not gonna do that is because it's gonna

clutter up our screen.

It's gonna throw a bunch of errors because we haven't set up

all the API keys for all 40 or 50 different recon modules,

and so I'm gonna not do that right now.

But if you wanna do that and you wanna set up your

system fully,

you can install all and then go find API keys for every

single one of those modules.

Now, the next thing we need to do is actually take

that module and load it.

Now again, if I type in modules search,

you're gonna see that I have installed that module,

but it doesn't mean it's loaded and ready for me to go

because you can see where I am inside of the recon

structure.

I'm at Recon NG, Dion, I'm not inside a module,

I'm not inside any of the options.

I'm just inside the workspace.

So again,

we're just gonna type in modules and hit enter.

You're gonna see the syntax.

We have load,

reload and search.

I'm gonna go ahead and type in modules load,

and then I want to use the WHOIS Underscore POCs,

which is the short name for the module I loaded.

There we go.

Notice how my prompt change.

Now I'm inside this module.

Now that we're in the module,

we need to specify the options that we want to use with

the module,

and we're gonna do that by entering information into our

database.

Now, inside of Recon NG,

there's actually a database installed with it and you're

able to put information into the database,

both things you want to search for and information that's

gonna come back when you do those searches.

Now, to do this and work with the database,

we need to use the DB command.

And again,

if you get lost at any time, just type in help.

You'll notice there.

The third line down is db,

which interfaces with the workspaces database.

Each workspace has its own database,

and that's another reason you wanna have a workspace

dedicated to whatever engagement you're working on.

So we're gonna go ahead and use db, and then again,

we can hit enter and it'll tell us what things we can do

with the database.

We can delete a database,

we can insert a line in a database,

we can add notes to a database.

We can query it using SQL commands,

or we can look at the database schema.

Now, if I wanted to see the schema of the database,

I can just type in DB schema and hit enter.

And when I do that, again,

it scrolls off the screen a little bit, but as we scroll up,

you can see the different tables that we're using,

I'm going to go up here to the top.

There we go.

So the first one we have is domains,

and this will hold domains notes.

And the module that found the information under companies,

it has company description, notes and module,

and you're gonna see each of these are gonna interact with

different parts of Recon, NG and different modules.

Domains is the one we're actually gonna be working with here

because I'm using the WHOIS point of contacts module.

So let me go ahead and get back here to the bottom.

I'm just gonna hit internal, drop me right down.

Okay,

and now what we wanna do is go back to our database command,

and what we wanna do is insert information into that

database,

specifically the domain names that we wanna look at.

Now what I'm gonna do is I'm just gonna type in db,

insert and then domains,

which is the field that I want to insert information in.

And at that point I hit enter and it's gonna say,

What do you want to enter in here?

Well, the thing I wanna enter is the domain name,

so if I wanted to search Diontraining.com,

I can enter that in.

If there's any notes I wanna put in,

I could put them here in too.

I'm just gonna hit enter and make that blank,

and it's gonna insert that into one row.

Now, if I want to enter another domain to search,

I can do that here as well.

Let's go ahead and do insert domains,

and in this case I'm gonna use udemy.com.

Another one I want to enter,

let's go ahead and do DB insert and we'll do domains,

and I'm gonna use tesla.com and again, no notes.

All right,

we have now inserted three different domain names,

Diontraining.com, udemy.com,

and tesla.com into our database.

Now, if we wanna see that and verify it took,

we can use the command show and then the name of the table

in this case domains.

So show domains,

and here is our table.

Boom, we have three rows, Diontraining.com, udemy.com,

tesla.com, no notes on any of them,

and the module was user defined because I manually enter

that information.

All right, now that we have loaded our module,

now that we have entered our information into the database,

we are ready to start searching the WHOIS Database for

these points of contact.

Now, what we're gonna do is we are gonna use this WHOIS

POC module.

If you don't know what that module is,

well we're inside of it now,

so we can just type in the command info, by typing info,

it's gonna tell me about the module I'm currently in.

Notice when I typed info for WHOIS POCs,

it tells me this is the WHOIS POC Harvester who wrote it,

the version and a short description of it.

It's gonna use the air and WHOIS database to be able to

harvest POC data from the WHOIS queries for the given

domain.

This is also gonna update our contacts table with the

results inside of our workspace so that data we get back is

gonna fill part of our database in the workspace so we can

go back and look at that information later.

Now we have some options that we have to have.

We have a source, we have the value of default,

it's required field,

and the description is the source of input.

Now, by default,

that means it's gonna go and grab it from the table.

Those three domain names we just put in in row one, two,

and three using the domains table.

If I wanted to do this from a file, I could do that as well.

By changing this default value from default to the file

name, in our case,

we're gonna do it right from the database.

It keeps it nice,

it keeps it clean, and that's the way we're gonna do it.

Now, if I wanted to change that,

I could do that by changing that by saying options set

because I'm changing the options in this case,

I wanna set the value of that from default

to something else.

Now, in our case, I don't wanna do that,

so I'm gonna go ahead and delete that,

but you could do that if you wanted to go ahead and read it

from a file, read it from an SQL query or something else.

As you can see here,

the default is to select distinct domain from domains where

domain is not null,

which just basically means go into the database,

look for the domains table,

and any domain inside of the domains table.

I want to grab each one that's not blank and we're gonna

test it.

So this is gonna allow me test all three of them with one

command.

Now that we know that our options are good to run this

command, we are just gonna simply type run and hit enter.

When we do that,

it's gonna go off and it is pulling that information and

it's grabbing all that information and it dumps it

to the screen.

Now, that's helpful,

but the screen makes it pretty hard to read cause I'd have

to scroll up and look at that because we just found 16 new

records and 12 of those were new contacts that were either

not duplicated or were new things that we wanted to add.

Those all got put into our database.

Now, if you remember back,

I said we can show things from the database using the show

command before we used show with Domains to show the three

domains.

Now I wanna go ahead and use show with contacts to see the

contacts table.

Here it is.

So we have those 12 entries going from row one all the way

down to row 12.

Now my screen is a little bit zoomed in to make it easier

for you to read in the video.

If I was zoomed out,

it would all fit in one nice table here you could see that

three columns,

the phone notes and module row went to the second line.

Now let's go ahead and read the first line.

The first line we have is based on Udemy.

You could see here that we found no first name,

no middle name, the last name was Operations Architect.

The email was netops+aaronudemy.com.

The title was WHOIS Contact because that's the type of

information we got.

The region, San Francisco, California,

the country, United States, and then phone, blank,

notes blank, module, WHOIS POCs,

which is the module we use to find this information.

Now there is lots of modules in Recon NG.

You might be finding contacts by doing Twitter searches,

LinkedIn searches, Google searches, whatever it is,

all that will go into this database to create our table

of people.

But this tells us which module found that information and

which type of information it was.

In this case A WHOIS Contact.

Next, we have lines two through 12,

and these are all based on Tesla.

Now you'll notice Dion training didn't show up.

The reason for that is we actually have our WHOIS records

set up with privacy,

and so those are not gonna be shared in the WHOIS database.

It just says this is a private record.

So there was no way to grab that information and add it to

the table.

That's why we have the one for Udemy,

which is a very common way of doing it.

For a large organization, they actually have a group email,

not a person's email, but if we look at Tesla,

they chose not to do that.

Tesla actually has individual people's names.

Now as I look at them,

what is this information that's gonna be useful?

Well, for one, I have names of people and emails I can use.

That's the obvious one.

But in addition to that,

I might be able to figure out what naming scheme that

company uses.

For example,

maybe you can't find everybody's email when you're doing

your open source research,

but you found their first and last name on LinkedIn.

Well, if you know that the company uses first name

dot last name like Elon.musk@tesla.com,

then you could put that in for everybody you find on

LinkedIn and now you have their email.

Conversely though,

we actually don't see that here with Tesla.

I am seeing multiple different naming schemes.

The first one, Anna actually shows up as Anne,

which is three letters from her first name and then her full

last name, martinez@tesla.com.

The next one is just an abbreviation for the name Cameron,

and they called it cam@tesla.com.

The next one is Sherry,

but it's abbreviated down to SHE,

the first three letters again,

and then her last name Lewis@tesla.com we get down to

Elon Musk, it's first name dot last name.

We get down to Jian Gu.

We're seeing his full name slap together with no period,

so it's another different convention.

We go down to line nine and we see Mah Desai,

and this is the first three letters and the

last name @tesla.com.

We get down to Paul Snicker and we just see Paul@tesla.com.

We get down to Terry Chi and we see tchi@tesla.com.

So we're seeing a little bit of differentiation here,

but several of them I saw at least three that had the first

three letters and then the last name,

so I'm thinking that might be the naming convention at Tesla

for their employees.

Now, if I wanted to validate that,

I could take some people's names who I find on LinkedIn and

I know their first name and last name,

put 'em into that format of the first three letters and

their last name@tesla.com,

and then see is it a valid email by checking something like

email dossier at Central ops or other things like that.

So hopefully you could start seeing how we put all these

different tools together and get information from different

places, consolidate it,

and then we can start doing things with it,

like spearfishing campaigns or whaling campaigns or social

engineering in general.

Lots of different ways to start using this information.

Now, as I said at the beginning,

there are a lot of different modules to Recon NG,

and I just wanted to show you the basic usage because all

the modules work the same way as you go into a workspace.

And then as you go into a module,

you're gonna keep going through the directory structure like

you saw here, Recon NG, Dion, WHOIS POCs.

Now, if I wanted to go back,

I could just type in the word back and it will bring me up a

level as well,

and then I can go ahead and load a different workspace and

then I can go ahead and do another assessment.

Now remember,

you can always use the show command to show anything you

want, just like I did show domains or show contacts.

You can also show companies, you can show credentials,

you can show hosts, leaks, locations, net blocks, ports,

profiles, push pins, repositories, and vulnerabilities,

because all of those have tables in the database,

as you saw when we looked at the database schema.

Now, all of those get things from various modules

that you may or may not install,

so that's the important thing,

is to have the right module for the right thing you're

trying to grab.

In this demonstration,

I only used one module and I only searched for contacts,

so that's why we saw things in this show,

contacts to display my findings as you see here

on the screen.

But the great thing is if I ran other searches from other

modules and they found contacts,

they would also be in this table,

and I could easily see those two.

Now hopefully,

you're starting to see how all of this starts coming

together,

and you're gonna play with it a little bit more on your own.

As I said before,

the thing that's gonna make you a great penetration tester

is hands on the keyboard, practicing with this stuff,

doing different reconnaissance evolutions,

trying to find information that's out there.

Now, the great thing about open source intelligence

and learning

how to do this is that all the information is out there.

You don't need permission from any of these companies to go

look up this information because it's all public information

that's sitting online,

and this is a great way for you to start building up your

skills early on in the reconnaissance process because as

long as you're doing passive reconnaissance,

you're not touching that company servers,

and you are not conducting any kind of hacking.

You're just in the preparation phases,

you're just learning information and using these tools and

getting better at them is gonna make you a better

penetration tester in the long run.