Integration Details

ThreatQuotient provides the following details for this integration:

Introduction

ThreatQ v6 Users – while the connector’s overall operation remains the same, installation and usage commands for ThreatQ v6 users will differ from what is listed in this guide. See the Installing Custom Connectors in ThreatQ v6 and Installing Custom Connectors on Another Instance topics for additional information.

The IBM X-Force Exchange connector imports data from the X-Force Exchange API into ThreatQ. It does this by setting up a connector, then, once enabled by the UI, connecting to the cloud instance of IBM X-Force Exchange and pulling down data. This is a unidirectional connector as it only consumes the data that XFE provides.

Prerequisites

Review the following requirements before attempting to install the connector.  

Time Zone

The time zone steps are for ThreatQ v5 only.  ThreatQ v6 users should skip these steps.  

You should ensure all ThreatQ devices are set to the correct time, time zone, and date (UTC is recommended), and using a clock source available to all.

To identify which time zone is closest to your present location, use the timedatectl command with the list-timezones command line option.

For example, enter the following command to list all available time zones in Europe:

Enter the following command, as root, to change the time zone to UTC:

timedatectl set-timezone UTC

Integration Dependencies

The integration must be installed in a python 3.6 environment.

The following is a list of required dependencies for the integration.  These dependencies are downloaded and installed during the installation process.  If you are an Air Gapped Data Sync (AGDS) user, or run an instance that cannot connect to network services outside of your infrastructure, you will need to download and install these dependencies separately as the integration will not be able to download them during the install process.

Items listed in bold are pinned to a specific version.  In these cases, you should download the version specified to ensure proper function of the integration.

Installation

The following provides you with steps on installing a Python 3 Virtual Environment and installing the connector.

Creating a Python 3.6 Virtual Environment  

Run the following commands to create the virtual environment:

Proceed to Installing the Connector.

Installing the Connector

Upgrading Users - Review the Change Log for updates to configuration parameters before updating.  If there are changes to the configuration file (new/removed parameters), you must first delete the previous version's configuration file before proceeding with the install steps listed below.  Failure to delete the previous configuration file will result in the connector failing.

1. Navigate to the ThreatQ Marketplace and download the .whl file for the integration.
2. Activate the virtual environment if you haven't already:
   source /opt/tqvenv/<environment_name>/bin/activate
3. Transfer the whl file to the /tmp directory on your ThreatQ instance.  
4. Install the connector on your ThreatQ instance:
   pip install /tmp/tq_conn_xfe-<version>-py3-none-any.whl 

   A driver called tq-conn-xfe will be installed.   After installing, a script stub will appear in /opt/tqvenv/<environment_name>/bin/tq-conn-xfe.

5. Once the application has been installed, a directory structure must be created for all configuration, logs and files, using the mkdir -p command. Use the commands below to create the required directories:
   mkdir -p /etc/tq_labs/
   mkdir -p /var/log/tq_labs/
6. Perform an initial run using the following command:
   /opt/tqvenv/<environment_name>/bin/tq-conn-xfe -ll /var/log/tq_labs/ -c /etc/tq_labs/ -v3
7. Enter the following parameters when prompted:

   Parameter	Description
   ThreatQ Host	This is the host of the ThreatQ instance, either the IP Address or Hostname as resolvable by ThreatQ.
   ThreatQ Client ID	This is the OAuth id that can be found at Settings Gear → User Management → API details within the user’s details.
   ThreatQ Username	This is the Email Address of the user in the ThreatQ System for integrations.
   ThreatQ Password	The password for the above ThreatQ account.
   Status	This is the default status for objects that are created by this Integration.

   Example Output

   /opt/tqvenv/<environment_name>/bin/tq-conn-xfe -ll /var/log/tq_labs/ -c /etc/tq_labs/ -v3
   ThreatQ Host: <ThreatQ Host IP or Hostname> 
   ThreatQ Client ID: <ClientID> 
   ThreatQ Username: <EMAIL ADDRESS> 
   ThreatQ Password: <PASSWORD>
   Status: Review 
   Connector configured. Set information in UI

You will still need to configure and then enable the connector.

Configuration

ThreatQuotient does not issue API keys for third-party vendors. Contact the specific vendor to obtain API keys and other integration-related credentials.

To configure the integration:

1. Navigate to your integrations management page in ThreatQ.
2. Select the Labs option from the Category dropdown (optional).
3. Click on the integration entry to open its details page.
4. Enter the following parameters under the Configuration tab:

   Parameter	Description
   Feed Name	The feed's name.  This field should only be updated if deploying multiple versions of the connector.
   X-Force Exchange API Key	The API key generated by the X-Force Exchange.
   X-Force Exchange API Password	The password that was generated by the X-Force Exchange.
   Initial Collection Days	The number of days to use during the initial pull request. The default setting is 5.
   Initial Attachment Days	The number of days to use during the initial pull of attachments, as these can be large this number can be tuned. The default setting is 10.
   X-Force Exchange URL	This field should not be changed unless the IBM X-Force team makes an upstream change.

5. Review any additional settings, make any changes if needed, and click on Save.
6. Click on the toggle switch, located above the Additional Information section, to enable it.

Usage

Use the following command to execute the driver:

Command Line Arguments

This connector supports the following custom command line arguments:

CRON

Automatic CRON configuration has been removed from this script. To run this script on a recurring basis, use CRON or some other jobs scheduler. The argument in the CRON script must specify the config and log locations.

Add an entry to your Linux crontab to execute the connector at a recurring interval. Depending on how quickly you need updates, this can be run multiple times a day (no more than once an hour) or a few times a week.

In the example below, the command will execute the connector every two hours.

1. Log into your ThreatQ host via a CLI terminal session.
2. Enter the following command:
   crontab -e

   This will enable the editing of the crontab, using vi. Depending on how often you wish the cronjob to run, you will need to adjust the time to suit the environment.

3. Enter the commands below:

   Every 2 Hours Example

   0 */2 * * * /opt/tqvenv/<environment_name>/bin/tq-conn-xfe -c /etc/tq_labs/ -ll /var/log/tq_labs/ -v3  
4. Save and exit CRON.

ThreatQ Mapping

Each main indicator type (Malware, URL, IP), has an associated report. Within this report are a few more attributes that we can flatten out as we ingest it into the TQ database. 

Below is the mapping of the XFE Data.

All indicators are related to a specific collection. An indictor will not be processed or picked up if it is not part of a collection. Each indicator can discover another family of indicators, although, at this time, they are ignored.

Attachment Types

Attachment Types are based off of report.type within an attachment.

The file is also included as an attachment.

Change Log

• Version 1.4.3
  • Fixed a bug that occurred when trying to ingest events with private IPs.   
• Version 1.4.2
  • Fixed a bug that occurred when XFE downloaded a file and added a password to it.
  • Fixed an error that occurred when attempting to check if an IP address was private.
• Version 1.4.1
  • Added arguments to execute the connector with historical start and end dates.
  • Added the ability to filter out private IPs.
  • Resolved an issue where an incorrect reference to a dictionary key from fdict['file']['size'] to fdict['file']['length'].  
• Version 1.4.0
  • Added Python 3 support.
• Version 1.3.2
  • Resolved an issue where the connector did not fetch information about indicators with .tmp file extensions in the URL. 
• Version 1.3.1
  • Zero-width space is now stripped form report titles.
• Version 1.3.0
  • Added support for external proxy.
• Version 1.1.2
  • Resolved HTTPError Import error. 
  • Resolved OS import error.
• Version 1.1.1
  • Resolved issue with Exchange type during event creation
  • Updated integration to use new ThreatQ SDK
• Version 1.0.0
  • Initial Release