MDS Server


TLDR;

  1. Install the VM. Login with 'maltego/tasx' and sudo. Give it an IP address.
  2. Browse to interface, load the certificate – in browser and in interface. The password for the cert is 'MaltegoServer'.
  3. Click on MDS at the bottom.
  4. In the MDS
    • Set up a data source
    • Set up a seed
    • Create a transform using just a query and a mapping. Reference guide is at the end of this doc.
  5. Discover seed from Maltego GUI
  6. Profit!
  7. Later on when you're feeling stronger:
    • Pre/post filter and global variables
    • Create entities in Maltego, export config, make paired config, attach to seed

What is the MDS?

The Maltego Data Server (MDS) is a server that allows you to easy visualise data kept in SQL databases or indexes (such as Splunk) in Maltego - as a graph.

In the most simplest form you only need to write a query (SQL/Splunk) and a tell the MDS how to map the resultant data back to nodes on the graph. In the most complex form you can write Python code around the query, mapping and nodes as well as use (global) replacement variables anywhere within the items above. With this we mean to say that the MDS can be as easy or complex as you'd like it to become. The system can grow with your abilities and is very flexible. 

Installing the MDS

Introduction, hardware specs and networking

The MDS is supplied as an OVF (Open Virtualization Format) image file and a certificate file (in PFX format). The OVF image can be used with VMWare Server, ESX or anything that can read the OVF format. The file is supplied might be ZIP-ed up or even supplied as an OVA. 

Hardware wise you should give the VM more than 4GB of RAM and a few CPU cores. The more the merrier of course, but more than 8GB of RAM would be an overkill.

On a network level the MDS needs have network access to all of the data sources that you would want to integrate with. For Splunk the default port is 8009 and SQL databases use their own ports (3306 for MySQL, 1433 for MSSQL and so on). The Maltego client(s) needs to access the MDS on port 9001 and 443.

Setting up the server

Step 1: Unzip the ZIP file. You can use any unzipping software. We’re using WinRAR. Extract the directory anywhere on your server. The location of the extracted ZIP file will not be the final destination of the server. Once unzipped the files should look like this:

 

 

Step 2: Import the OVF into VMWare. In this example we’ll show you how to import the OVF into VMWare Workstation. Simply double click on the .ovf file and select where the Virtual Machine should be located:

The import will start and you’ll see the progress:

Step 3: Configure the hardware for the image:

You can see from the screenshot above that we’ve given the server 6GB of RAM, 4 processors. Set the networking interface to be bridged. 

Step 4: Boot up the image. You need to login in with username ‘maltego’ and password ‘tasx’. To get root access simply type ‘sudo -s’ and enter the password again. Give the server an IP address. In our case we’ll use DHCP but if you need a static IP address you need to set it up. Please refer to http://askubuntu.com/questions/470237/assigning-a-static-ip-to-ubuntu-server-14-04-lts if you need help setting up a static IP address.

Activating the server

Your server is now ready to go – but it needs to be activated using the license certificate.

Paterva uses client side certificates for authorisation and authentication on the MDS, CTAS and iTDS. Within the CTAS the certificates are used to activate the seed server (also called a runner). On the MDS and iTDS server the client side certificate is used with a browser to both activate and provide access to the administrative section of the web application. 

For all servers you will initially need to have your PFX certificate on hand. This is usually provided via email but is also available on the server portal. 

Once you know what the IP address of the server, simply browse to the interface on HTTPS:

The first step you’d want to do is to upload the certificate file to the server. Click ‘Choose File’ then choose the certificate file (PFX) from your local filesystem and click on Submit. Certificate details will be shown to you:

Next click on the ‘Activate’ button. This will move the file into the correct area and it will also restart the webserver – using this certificate. This means you might need to reload the page after a few seconds. If the certificate was good and everything went according the plan you’ll see a screen similar to this:

Adding your certificate to your web browser

For the iTDS and MDS servers you will need to install the SAME client side certificate within your browser before browsing to the server module. This certificate is also used to authenticate you to the server. To do this you will need to configure your browser, this periodically does change between the various browsers and here are two simple guides for Firefox as well as Chrome.

The password for the certificate by default is ‘MaltegoServer’ but it can (and should) be changed.

Firefox

The first step in Firefox is to select the menu icon on the right hand side and select Options, then select the "Advanced" tab and click on "View Certificates". From there select "Import..."

      

From here select the file and unlock it with the password MaltegoServer (case sensitive):

      
 

Chrome

The first step in Chrome is to select the menu item on the right hand side and select options, then scroll to the bottom of the page (you may need to select advanced) and select "Manage Certificates" under the HTTPS/SSL heading. Next click on "Import..."

      

From here select the file and unlock it with the password MaltegoServer (case sensitive):

      

Once you have successfully imported your certificate in your browser you can browse to the MDS or iTDS module interface over HTTPs and the browser will either pick the certificate for you or ask you which certificate you would like to use as seen below:

  

After selecting the correct certificate you will be able to use the interface.

You should now see that you are redirected to port 9001 on the server where the MDS administrative interface lives:

Yippee! – your MDS is now ready for use.

First steps / Hello world on the MDS

Setting up a data source

A data source is either a Splunk Index or a database. Supported database types are MSSQL, MySQL, Postgress, DB2 and Oracle as SQL sources. ELK as index support is to follow.

On the front page of the MDS, select ‘Data Sources’. Click on ‘Add Data Source’. Select which type of data source it is, fill out the form and click ‘Add Data source’:

Above example is for a MySQL database. Ensure that the database is accessible from your MDS server, the remote connections to the database are allowed, that the username / password is correct and that the user has access to the database. When successful you’ll see the following response:

In the same way a Splunk index data source can be set up:

Again – ensure that the Splunk server can be reached from your MDS, that the username/password combination is valid. The Splunk App name can also be specified – but this is not mandatory.

Next, we need to add a seed.

Setting up a seed

A seed is a container for transforms – a group of transforms. Click on the ‘Seed’ section at the top and click on ‘Add Seed’. Fill out the form:

The Seed name could be anything that is representative of the transforms in the seed. The Seed URL could also be anything – but anyone with this seed would be able to run your transforms (if they are network wise permitted to reach the MDS) so pick a random name – we picked ‘110011’ as a seed URL. You’ll see where you’ll need that in a bit.

The seed paired config can be left blank for now and also the seed transforms – those we still need to populate. When successful you’ll see the following results:

We’re now ready to start building transforms!

Building a basic first transform

Before you start building a transform you need to decide what the transform should do. Let’s assume we want to build a transform that will take a phrase as input and search our Apache logs, that are indexed by Splunk, for that phrase in the URI, returning the IP addresses as nodes.

Click on the Transforms section at the top of the interface. Click on ‘Add Transform’. At this stage it’s advised that you also open your database query tool (whatever you’re usually using) and where applicable your Splunk web interface. You can also use the Workbench at the top of the MDS interface but we’ll cover that later.

To build a new transform we’re going to need to fill out the following sections:

Transform Name – This is what’s displayed in the Maltego GUI. It’s useful to give this a descriptive name. We’ll use “Search URI in weblogs”.

Input entity – This is the entity that the transform will run on. In our case we’ll use the maltego.Phrase entity. If your transform does not show up in the Maltego GUI then you should make sure that you’ve specified the right input entity type.

Datasource – That’ pretty straightforward – we’ll use the Splunk Server we configured in the previous step.

Transform Data Query - This is an important part of the transform. It’s the query we’ll write to get the data from the data source to us. This query can contain tokens which, at runtime, will be substituted with actual values from the graph.

In our case, we’ll enter here:

host="www.paterva.com" uri="*$$E.Value$$*"| dedup clientip | table clientip 

There a couple of things to note here. Firstly, the token to get the value from the input node is written as $$E.Value$$ and this will be replaced by the node’s value at runtime. Secondly the query ALWAYS has to end in a |table . This is so that we can perform mapping to return entities in the next section. There are many other tokens that we can use – these are described in detail later in this document as well as in the reference guide at the end of this document.

Pre/Post Filters – For now we’ll skip these. We’ll return to this a bit later in the document.

Transform Mappings – It’s here where we map the output table of the query back to nodes inside of Maltego. For now, use the following

{E.Type=”maltego.IPv4Address”; E.Value=”$clientip”}

This tells the MDS that we want IP addresses back from our query and we want their values to be mapped to the column that has a key called ‘clientip’. This corresponds with what we wrote in the Splunk query.

There is a lot more we can do here but we’ll discuss the mapping language in detail later in the document and in the reference guide.

Transform settings – Ignore for now. We’ll cover it later.

Transform Seeds – Pick the seed we’ve set up in the previous section called Paterva Ops.

The complete form should look something like this:

The first thought in your mind should be how the time limits on the Splunk query is specified – don’t worry – we’ll get there. Click on ‘Add’ and voila – your first transform is ready to roll.

Getting the transform in Maltego

The next step would be to see how the transform works in Maltego. To do this you need to add the seed to Maltego and discover from the seed.

The first thing you want to do is to copy the seed URL to your clipboard. Go to the Seeds section at the top and next to your seed select the copy to clipboard button:

Now go to Maltego and in the transform hub click the large [+] sign that’s the first transform hub item. This allows you to add your own transforms:

Give the seed item an appropriate name and copy the URL into the ‘Seed URL’ text field:

Hit OK. You’ll now see the item appear in your transform hub:

Click on ‘Install’ and follow the prompt. Once properly installed you’ll see the following:

The seed is now ready to use. Let’s see how to use it in Maltego.

Running your first transform

Our transform ran on a phrase entity – that’s the input entity type we specified. Open a new graph and drag a phrase entity (in section ‘Personal’) onto the graph and give it a value. Let’s see which IPs addresses requested the file ‘robots.txt’. We thus give the entity a value of robots.txt:

When we right click (and go all the way up to the root of the context menu by right clicking) we see we now have a new item called ‘Paterva Operations’:

In this hub item, we find our transform:

Now – and this is important – we get to set two parameters that will be sent along to the query. The first is the maximum number of entities we want to return. This is specified using the slider which you’ll find in the ‘Investigate’ tab on the ribbon:

The second is the timeframe that we’re passing along to the Splunk query. You find this on the right-hand bottom of the main window. It’s called ‘Hub Transform Inputs’. For ‘Paterva Operations’ you can now specify the date range that’s passed along with the Splunk query:

All transforms in this hub item will be passed these values. It’s handy because once you’ve decided on your timeframes for your investigation you only need to set it once and all transforms will receive this setting.

Let’s assume we only want to see who requested ‘robots.txt’ for today – so we’ll set it as shown in the screenshot above.

When we finally run the transform the results look like this:

From this we can see that 109 unique IP addresses requested the file robots.txt on our website for today.

Of course, now it becomes easy to run the standard Maltego transforms on the result. Assuming we want to see where the requests comes from, the country of origin and so forth and on:

The big blue entity is Google’s netblock.

Your first SQL transform

Let’s quickly see how to make a SQL based transform. We’ve already defined the data source in the previous steps. The table maltego_users looks like this:

Imagine we want to build a transform where we supply the domain and it tells us what users registered from this domain (in their email address).

Transform Name - We’ll use “Maltego users from this domain”.

Input entity – We’re running it from a domain so it will be the maltego.Domain entity.

Datasource – The PatervaDB we’ve set up before

Transform Data Query. This now becomes a SQL query. It looks a like so:

SELECT email from maltego_users where email like ‘%@$$E.Value$$’

We don’t need to end this in a table…because the results will be a table anyhow!

Pre/Post Filters – We will discuss this later…

Transform Mappings – We want to map back to an email address so it becomes:

{E.Type=”maltego.EmailAddress”; E.Value=”$email”}

Note that we can use column names from the returning SQL query here – how handy.

Transform settings – We will discuss this later

Transform Seeds – Let’s add this to the Paterva Ops seed.

The form when filled out thus looks like so:

To update Maltego so it knows about the transform we click on the ‘Refresh Transforms’ button on the ‘Paterva Operations’ item in the Transform Hub. We drag a domain into Maltego, right click and – there’s the transform! Running it with the value of ‘gmail.com’ we get:

It’s as easy as that.

Making quick changes to queries and mappings.

Almost everything on the main screen of the MDS is clickable and directly editable. This is done so that you can easily make changes to transforms or mappings without having to go through the pain of numerous menu systems.

In the SQL transform example we just did, if we want to be able to search for subdomains we might want to change the SQL query from

SELECT email from maltego_users where email like ‘%@$$E.Value$$’

To

SELECT email from maltego_users where email like ‘%@%$$E.Value$$’

Note the extra ‘%’ sign in the like section of the query – marked in red. To do this you can simply click on the statement on the front page and… edit it. When you press Enter your changes will be saved.

The more interesting stuff

That was easy enough – but soon you’ll see that you want to do more complex things. You might need to read and write to entity properties other than the display value, you might need to implement transform settings or perhaps you need to treat or parse the data before it hits the query or before it goes back to the Maltego client.

In the upcoming sections, we’ll show you a little bit more.

Reading and writing to entity properties

Properties of entities can be read by using the $$E.Property(“propertyname”)$$ token. Properties can be written to a mapped entity using {E.Property(“propertyname”)=”value”;}.

Let’s look at an example. Suppose we want to extend the SQL transform in such a way that the license key of the user is added to the email as a dynamic property called ‘lickey’. The SQL query would now look like this:

SELECT email,lic_key from maltego_users where email like “%@$$E.Value$$”

The SQL query would return two columns. We’ll use the extra column (lic_key) in our mapping like so:

{E.Type=”maltego.EmailAddress”; E.Property(“lickey”)=”$lic_key”; E.Value=”$email”}

When we change the query/mapping combination as above and run the transform you’ll now see that the returning entities contains a property (in this case a dynamic property as the original entity does not have this) called ‘lickey’:

The actual license key is blurred out…a bit ;)

Because we have different license keys for each email address merging between nodes does not happen. This means we can have two (or more) nodes with the same value but with different license keys – the same way you’ll have two people called ‘John Smith’ but with different SSNs as separate nodes:

Reading transform settings

A transform can prompt a user for any kind of input in the Maltego GUI prior to it sending the request off to the server. This is called a transform setting. This is useful in situations where you want to prompt the user to add additional information to a query. Let’s see how this works.

Let’s assume we want to change the initial Splunk transform where we supplied part of the URI to the transform to also allow us to specify part of the user-agent.

Go to ‘Settings’ -> ‘Add Setting’. Fill out the fields so that it looks like the screenshot below:

Now let’s see how we use this in the actual transform. Go back to the Transforms section. Edit the transform ‘Search URI in weblogs’.

The first thing we want to do is to add the setting to the transform. This tells the server that when you discover from the seed the transform contains a setting. To add a setting go to Transforms, click on ‘Show Details’ in the Advanced column. Click on the text box at Settings and you’ll see a list of settings that are available. You can also search for your setting in this text box:

You can add as many settings as you want but for now that’s the only one we’re using.

Now that you have the setting added to the transform you can start using it in the query. Change the query from:

host="www.paterva.com" uri="*$$E.Value$$*" | dedup clientip | table clientip

to:

host="www.paterva.com" uri="*$$E.Value$$*" useragent="*$$E.Setting("UserAgent")$$*"| dedup clientip | table clientip

The section of the query that’s edited is highlighted in bold. To change the query, you can either edit it directly on the transform summary screen or you can click on the ‘Edit’ button next to the transform. Assuming you clicked on edit - when you’re done the form should look like this:

Because we’ve added a transform setting we need to discover the transform in Maltego again. In the transform hub click on the ‘Refresh Transforms’ button. Now we’re ready to test the transform. PS - to know when you should reload a transform and when it's not necessary please refer to the reference guide at the end of this document.

In Maltego drag a phrase to the graph, change to ‘robots.txt’, right click and run the ‘Search URI in weblogs’ transform. You’ll see that prior to the transform running a pop-up with appear:

We can now search for something in the URI AND search for a specific user-agent at the same time. For instance – let’s verify that everyone that accessed the web server in the last 24h requesting robots.txt with a user-agent of ‘Googlebot’ was indeed Google.

Looks good. All the 32 IP Addresses that requested robots.txt with a user-agent of ‘Googlebot’ did indeed come from Google’s network.

Writing to other parts of the entity

We covered the main value (display value of the entity) and setting properties but there are other properties that can be set in an entity. These are all write-only meaning that their values cannot be read in subsequent transforms – in other words – the client is not sending these up to the server. These are:

  • Display info (E.DisplayInfo)

  • Bookmarks (E.Bookmark)

  • Notes (E.Notes)

  • IconURL (E.IconURL)

  • Weight (E.Weight)

Properties of the link (or arrow) that goes from the input entity towards the resulting entity can also be set. The following properties are also available for an entity link:

  • Link color (E.LinkColor)

  • Link style (E.LinkStyle)

  • Link label (E.LinkLabel)

  • Link thickness (E.LinkThickness)

For a complete list of what these values can be set to please refer to the reference guide at the end of this document. Let’s look at a simple example of how these could be used.

Taking our previous example a little further – let’s imagine we want to display with user-agent in the Detail View of every node. We change our query from this:

host="www.paterva.com" uri="*$$E.Value$$*" useragent="*$$E.Setting("UserAgent")$$*"| dedup clientip | table clientip

to

host="www.paterva.com" uri="*$$E.Value$$*" useragent="*$$E.Setting("UserAgent")$$*"| dedup clientip | table clientip,useragent

Note how we added another column to the output. In Splunk – when we run the query with E.Value replaced with ‘robots.txt’ and the E.Setting replaced with ‘Googlebot’ we now get two columns:

We want to map the second column to the DisplayInfo of the returning node. As such the mapping becomes:

{E.Type="maltego.IPv4Address"; E.Value="$clientip"; E.DisplayInfo("UserAgent")="$useragent";}

And so the $useragent column will show up in the entity under a display heading called ‘UserAgent’. Best you see it… after we’ve given the transform the same transform setting input (‘Googlebot’) as in the previous example we now see it returning nodes as follows (the selected node was unpinned from the collection just to make it easier to read/understand):

Notice the ‘User-Agent’ heading. It’s important to know that DisplayInfo is HTML aware – meaning you can make it as ugly or pretty as you want to with tables, pictures, links etc.

Filters

Filters are code elements that allow you to add much needed flexibility to manipulate entities, queries, mappings or even returned data with real code. There are two distinct types of filters: pre filters and post filters.

Pre filters

A pre filter runs on the incoming entity and can change two things – the entity itself and the query to the datasource. This means we can change the input entirely or run different queries based on what the input looks like.

To create a pre filter, go the Filters. Add a name for the filter and select pre or post filter. Currently the only language supported is Python. Finally, enter the code in the code block window. For a complete list of functions available as well as examples of code please refer to the Reference Guide at the end of this document.

From this filter, we can see that the query is change via the _query variable – based on if the property (‘IP’) in the input entity exists.

When done click on ‘Add Filter’. To apply the filter to a transform, go to the Transform section on the MDS. Find the transform and click on the Advanced button (grey cog):

A modal dialog will appear. Under ‘Pre Processing’ select the pre filter that you want to apply:

The pre-filter is now applied to the transform. Note that since the pre filter is changing the query it does not matter what is in the query field. In fact – what we’ve found is that it’s useful to enter free text with something like “Override by filter named ‘myfiltername’”. That way it’s clear that the query is set in code.

Post filters

Post filters work in the same way as a pre filter – they just run after the query has been completed but before the mapping is applied. Post filters can change the mapping that’s used via the _mapping variable. It can also change the actual data returned itself. This means that based on the data that’s returned the mapping can be changed in code.

Post filters are applied in the same way as pre filters. For a complete list of methods and members available in post filters please refer to the reference guide at the end of this document.

Global variables

In the process of building transforms you could find that a lot of queries or mappings (or even filters) use the same piece of text/code repeatedly. Imagine having many transforms that map to a URI with DisplayInfo that contains a clickable link to the URI. The mapping might look something like this:

{E.Type=whatever.URI; E.Value=$uri; E.DisplayInfo("Link")="Click on <a href=http://www.paterva.com$uri>link</a>";}

You’ll find that you need to redo this DisplayInfo in every single transform and it seems tedious. Also – if you want to make a change to this you’ll need to change it in every transform. What you really want is a way to define a ‘variable’ – something that renders the link – and in the mapping you just want to reference it. Something like this:

URILinkRender is equal to Click on <a href = http://www.paterva.com$uri>link</a>

And then all the mapping becomes

{E.Type=whatever.URI; E.Value=$uri; E.DisplayInfo("Link")="$$URILinkRender$$";}

Notice how the variable is enclosed in ‘$$’ signs. You can do this with what’s called ‘Global Variables’.

To define Global Variables, go to the ‘Global Variable’ tab and click on ‘Add GlobalVariable’. Enter a suitable name for the variable and enter the text as you’d like to use it. For example:

You might want to even abstract this more – as global variables could be used inside other global variables (not implemented in beta). In other words, you might end up with this:

Debugging transforms – the workbench

The workbench gives you the ability to easily tweak/debug your pre-filters, query, post-filter and mappings. You will probably spend a lot of time in the workbench…

To start the workbench, click on the ‘Workbench’ button at the top of menu. You’ll see the following screen appear:

Although it might look intimidating at first, it’s quite straightforward. The workbench has three distinct sections.

  • Input entity / Pre filter / query

  • Post filter

  • Mapping

Breaking the flow into these three sections allows you to test every step of the way independently. Let’s see how to use every section.

Input entity / Pre-filter / query section

To test these components, we need to supply the interface with what the Maltego GUI supplies to the server component – these are the following:

  • Input entity (with its value, properties and type)

  • The slider value – in others words – how many results the user requested

  • The time span (not implemented in beta, hardcoded for now)

  • Transform settings (called Transform Inputs in the interface)

In terms of the transform itself we need to know which pre-filter is in use, which datasource we should use and of course the query to the data source. To make these a little easier to populate you can select which transforms you want to test and the interface will pre-populate the relevant fields – but feel free the change these on the fly.

Let’s start with something simple – suppose we have a Phrase entity and we want to look it up within a very-plain Splunk query. Start by dragging a Phrase entity into Maltego GUI and give it a value. Then right click on the entity and select the drop down at copy. Copy as GraphML – and the XML will end up on your clipboard:

In the Workbench interface paste the XML into the Input Entity text area:

You can see that it’s indeed a Phrase entity and scrolling down you’ll see that the value is set to ‘maltego’. Now the Workbench interface knows what the input entity looks like.

For now, we’ll keep the number of results to the default of 12 and there will be no transform settings (or inputs as it’s called in the interface). We are going to manually craft the query so we won’t be selecting a transform. There will be no pre filter involved – for now.

We set the datasource to be our Splunk server and finally we can write the Splunk query. Let’s just do something really boring – looking if the phrase exists anywhere as an URI within our Apache logs. Thus, the query becomes:

host=www.paterva.com uri=”*$$E.Value$$*”

Remember the E.Value is replaced by the value of the entity – so that query looks about right. When everything is filled out hit the ‘Run Query and PreFilter’ button.

After a bit of time the results are populated in the next section of the interface:

The first text section gives us a bit of debugging info of what happened in every step. Since we did not have any pre-filter or global variables this output is pretty bland. The next section shows us what happened after the pre-filter ran. Remember that the pre-filter can modify two things – the entity itself and the query. In this case there was no filter and as such the only thing that happened is that the E.Value token was replaced in the query with the value of the input entity. The returned entity (returned from the pre-filter) is also not modified at all – so the returned entity is the internal representation of the input entity.

The output is the data (internal representation) returned from the Splunk server. Side scrolling in the text box we can see that this looks as expected:

This is quite boring so let’s look a more complex example.

Imagine we have a pre-filter in place that first changes the entity to all uppercase and then based on the value changes the query to the data source. Furthermore, there is a transform setting that will ask the user a simple Yes/No question and based on the answer apply the pre-filter. It’s convoluted but it will nicely explain the concepts, so let’s just go with it.

We first create a Transform Setting called ‘YN’ with a default of ‘No’. We then create a pre-filter (we called it ‘PreExample’). The code for the pre-filter looks as follows:

if (E.getTransformSetting('YN')=='Y'):
    E.Value=E.Value.upper()
    _query='host=www.paterva.com URI="*$$E.Value$$*"'
else:
    _query='host=www.paterva.com URI="*admin*"'

From the code it’s clear that, should the setting be set ‘Y’ the input entity will be changed to upper case and the query will use the input value. If it’s anything other than ‘Y’ then…well.. then we just search for the word ‘admin’ in URIs (convoluted... indeed.)

To load this up in the workbench we will in the fields as follows:

Notice how, when you add a transform setting, you need to choose a value for it (YN in green). Since the pre-filter will override the query anyhow it does not make sense to put anything in that value. We put ‘override’ in there just to make us remember that’s what’s happening. When this is executed the output is as follows:

Notice how the entity (and the query) is changed. In our test case, there were no global variables – else those would be substituted too. Also, notice how the Splunk query did provide any results – this why the Output window is empty.

If we change the YN input value to ‘N’ then we get the following:

Now, based on the input, the query is changed again and does not even contain any part of the input entity.

Post filter section

The output of the previous section now becomes the input to the next section. Keep in mind that any of the fields can be changed by hand – except the ‘Returned Query’ field.

Suppose we somehow ended up with the following from the previous step:

The red arrow in the picture above shows the fields that were returned from the previous step.

Next, we get to play a bit with a post-filter and the mapping. Let’s assume we are going to URL encode the user agent returned in the query using a post-filter that looks like this:

import urllib
for k,v in enumerate(rawResults):
    rawResults[k]['ua_escaped'] = urllib.quote_plus(rawResults[k]['useragent'])

We create this post-filter naming it ‘URLencodeUA’ and set it in the workbench. For the mapping, we have:

{E.Type=maltego.URI; E.Value=$uri; E.DisplayInfo=”<a href=www.google.com?q=$ua_escaped>Google userAgent</a>”;}

What’s happening here? Well – as a start we can see that the mapping remains unchanged by the post filter (and it never touches the _mapping variable). However, the post filter is adding a column to the output table called ‘ua_escaped’. The mapping is referring to that column by name and putting it inside a query to Google, then adding that to the (HTML aware) DisplayInfo (without a heading) of the entity (of type URI).

In the workbench, the output is as follows (note that in the screenshot below we’re running the beta version which is still susceptible to XSS!! – bonus points to those that can spot it!):

Notice how the user agent is encoded and added as a column in the output window by the post-filter. For completeness sake, here is the full output for the first 3 lines of the output window:

clientip,uri,useragent,ua_escaped
187.40.65.182,/administrator/index.php,-,-,
193.201.224.205,/wp-admin/tools.php?page=backup_manager&download_backup_file=../wp-config.php,Mozilla/5.0 (Windows NT 6.1; rv:34.0) Gecko/20100101 Firefox/34.0,Mozilla%2F5.0+%28Windows+NT+6.1%3B+rv%3A34.0%29+Gecko%2F20100101+Firefox%2F34.0
193.201.224.205,/wp-admin/blog/admin-ajax.php?action=revslider_show_image&img=../wp-config.php,Mozilla/5.0 (Windows NT 6.1; rv:34.0) Gecko/20100101 Firefox/34.0,Mozilla%2F5.0+%28Windows+NT+6.1%3B+rv%3A34.0%29+Gecko%2F20100101+Firefox%2F34.0

The column names are on the first line. We can thus see that our post-filter is working as expected.

Mapping section

The output of the previous section becomes the input for this section. Again – we can change any field by hand to aid in quick debugging. Continuing with the previous example – once we click the ‘Run Mapping’ button we can see if our mapping is working:

Each line in the Output window is the internal representation of a Maltego entity. Here is the full text for the first two entities:

{"iconURL": "", "displayInformation": [["", "<a href=www.google.com?q=->Google userAgent</a>"]], "weight": "100", "entityType": "maltego.URI", "additionalFields": [], "value": "/administrator/index.php"}
{"iconURL": "", "displayInformation": [["", "<a href=www.google.com?q=Mozilla%2F5.0+%28Windows+NT+6.1%3B+rv%3A34.0%29+Gecko%2F20100101+Firefox%2F34.0>Google userAgent</a>"]], "weight": "100", "entityType": "maltego.URI", "additionalFields": [], "value": "/wp-admin/tools.php?page=backup_manager&download_backup_file=../wp-config.php"}

From this you can see that it looks good to go.

Paired configs

Imagine that you have created entities in Maltego and on the MDS and perhaps even some machines to automate some processes. When you take your transforms to production you want everyone that discovers from the seed (transform hub item) to get your nicely crafted icons, the entity definitions, the machines and so on. The way to do this is with paired configs.

Once you have created all the entities (and any other configuration items) in the Maltego GUI you can export it to a config file (MTZ). To do this in Maltego click on the Application Button (top left), click on Export -> Export configuration -> Custom selection. You should see the following dialog appear:

Typically, the things you want to export would be the Entities you’ve made, Icons and Machines you’ve created. It’s pretty easy to export (e.g. select none in the parent object and then only select the ones you want to export):

Once done, select the filename and save the file to your local disk. Now we’re ready to move this file to the MDS interface.

Select ‘Paired Config’ from the MDS top menu. Click on ‘Add PairedConfig’. Give the config a suitable name, click on ‘Choose file’ and navigate to the file you just saved from Maltego.

Finally click on ‘Add Paired Config’ and if everything went according to plan you’ll see:

Your config is now loaded in the MDS. You can load as many configurations as you like.

The last step would be to pair the configuration with the seed. This would have the effect that the Maltego GUI will import the configuration when it installs from the seed. To do this, go to the Seed section of the MDS. Find the relevant seed and under ‘Paired Config’ select the paired config that you just created. In the interface, it looks like this:

Once you’ve selected the paired config it is immediately in use. Now everyone that discovers from this seed will also get the config (with all the entities etc.) that you’ve exported from the GUI.

Backup and restore

It's useful to make backup of your configuration and to be able to restore it on another server. The backup and restore method allows you to backup a single seed and restore it - this is useful when migrating to a new server image or when moving transforms from one server to the other. As the information contained in the backup is likely sensitive we use AES 256 bit encryption on the backup files.

To make a backup of your configuration click on the 'Backup' button at the top.

backup1.jpg

Enter a password and select which seeds you wish to backup. In most cases you'd want to also back-up all your custom entities as well as global variables.

To restore a seed on another system select the file, enter the password again and hit restore. The seed will be restored on the new system. Please note - when there are conflicts between any of the components the restore function will not restore any duplicated items. To override such items you need to first delete them from your MDS.

 

Reference guide

The reference guide for the MDS can be found on the following page.

Continue to the CTAS Server page.



© Copyright 2017, Paterva PTY Limited