IMAP Connector

The web format of this guide reflects the most current release.  Guides for older iterations are available in PDF format.  

Time Zone

The time zone steps are for ThreatQ v5 only.  ThreatQ v6 users should skip these steps.  

You should ensure all ThreatQ devices are set to the correct time, time zone, and date (UTC is recommended), and using a clock source available to all.

To identify which time zone is closest to your present location, use the timedatectl command with the list-timezones command line option.

For example, enter the following command to list all available time zones in Europe:

Enter the following command, as root, to change the time zone to UTC:

1. Download the connector integration file from the ThreatQ Marketplace.  
2. Transfer the connector whl file to the /tmp/ directory on your instance.
3. SSH into your instance.
4. Move the connector whl file from its /tmp/ location to the following directory: /opt/tqvenv
5. Navigate to the custom connector container:
   kubectl exec -n threatq -it deployments/custom-connectors -- /bin/bash

   Copy Code

6. Create your python 3 virtual environment:
   python3.6 -m venv /opt/tqvenv/<environment_name>

   Copy Code

7. Active the new environment:
   source /opt/tqvenv/<environment_name>/bin/activate

   Copy Code

8. Run the pip upgrade command:
   pip install --upgrade pip

   Copy Code

9. Install the required dependencies:
   pip install setuptools==59.6.0 threatqsdk threatqcc python-dateutil

   Copy Code

10. Install the connector:
    pip install /opt/tqvenv/tq_conn_imap-<version>-py3-none-any.whl

    Copy Code

11. Perform an initial run of the connector:
    /opt/tqvenv/<environment_name>/bin/tq-conn-imap --cron="0 */2 * * *"

    Copy Code

    The --cron argument above is used to generate a cron job for the connector.  After running the command above, the cronjob will be created under the /etc/cron.d/ directory.  This entry will initially be commented out upon creation - see the CRON chapter for more details.  

12. Enter the following parameters when prompted:

    Parameter	Description
    ThreatQ Host	Leave this field blank as it will be set dynamically.
    ThreatQ Client ID	This is the OAuth id that can be found at Settings Gear → User Management → API details within the user’s details.
    ThreatQ Username	This is the Email Address of the user in the ThreatQ System for integrations.
    ThreatQ Password	The password for the above ThreatQ account.
    Status	This is the default status for objects that are created by this Integration.

    Example Output

    /opt/tqvenv/<environment_name>/bin/tq-conn-imap --cron="0 */2 * * *"
    ThreatQ Host:  
    ThreatQ Client ID: <ClientID> 
    ThreatQ Username: <EMAIL ADDRESS> 
    ThreatQ Password: <PASSWORD>
    Status: Review 
    Connector configured. Set information in UI

You will still need to configure and then enable the connector. 

1. Navigate to the ThreatQ Marketplace and download the .whl file for the integration.
2. Create the following directory:
   mkdir /opt/tqvenv/

   Copy Code

3. Install python 3.6:
   sudo yum install -y python36 python36-libs python36-devel python36-pip

   Copy Code

4. Create a virtual environment:
   python3.6 -m venv /opt/tqvenv/<environment_name>

   Copy Code

5. Activate the virtual environment:
   source /opt/tqvenv/<environment_name>/bin/activate

   Copy Code

6. Run the pip upgrade command:
   pip install --upgrade pip

   Copy Code

7. Install the required dependencies: 
   pip install threatqsdk threatqcc setuptools==59.6.0 python-dateutil

   Copy Code

8. Transfer the whl file to the /tmp directory on your ThreatQ instance.  
9. Install the connector on your ThreatQ instance:
   pip install /tmp/tq_conn_imap-<version>-py3-none-any.whl 

   Copy Code

   A driver called tq-conn-imap will be installed.   After installing, a script stub will appear in /opt/tqvenv/<environment_name>/bin/tq-conn-imap.

10. Once the application has been installed, a directory structure must be created for all configuration, logs and files, using the mkdir -p command. Use the commands below to create the required directories:
    mkdir -p /etc/tq_labs/ mkdir -p /var/log/tq_labs/

    Copy Code

11. Perform an initial run using the following command:
    /opt/tqvenv/<environment_name>/bin/tq-conn-imap -ll /var/log/tq_labs/ -c /etc/tq_labs/ -v3

    Copy Code

12. Enter the following parameters when prompted:

    Parameter	Description
    ThreatQ Host	This is the host of the ThreatQ instance, either the IP Address or Hostname as resolvable by ThreatQ.
    ThreatQ Client ID	This is the OAuth id that can be found at Settings Gear → User Management → API details within the user’s details.
    ThreatQ Username	This is the Email Address of the user in the ThreatQ System for integrations.
    ThreatQ Password	The password for the above ThreatQ account.
    Status	This is the default status for objects that are created by this Integration.

    Example Output

    /opt/tqvenv/<environment_name>/bin/tq-conn-imap -ll /var/log/tq_labs/ -c /etc/tq_labs/ -v3
    ThreatQ Host: <ThreatQ Host IP or Hostname> 
    ThreatQ Client ID: <ClientID> 
    ThreatQ Username: <EMAIL ADDRESS> 
    ThreatQ Password: <PASSWORD> 
    Status: Review
    Connector configured. Set information in UI

You will still need to configure and then enable the connector.

Copy Code

Copy Code

ThreatQ version 6 aggregates the logs for all custom connectors to its output container.  You can access the container's log using the following command:

Copy Code

The connector log directory was created in 10 of the installation process and is identified using the -ll argument flag when executing the driver.

The custom connector configuration file can be found in the following directory: /etc/tq_labs/.  

The custom connector configuration file was created in step 10 of the install process and identified using the -c argument flag when executing the driver. 

The addition of the --cron argument in the initial run of connector, performed during the install process, resulted in the creation of a cron job file for the connector in the following directory: /etc/cron.d/.  The contents of the file will resemble the following structure:

The {schedule} will be replaced with the cron settings you entered with the --cron flag and the {executable} will be replaced for with the connector's driver command.

You will also see a # at the beginning of the file.  This comments out the job.  This allows you to configure the custom connector in the ThreatQ UI first.  After you have configured the connector in ThreatQ, you can remove the # from the file content's in order to activate the cron job.

To summarize this process:

1. Install the connector and perform an initial run using the --cron argument to create the cron job.
2. Complete the connector's configuration settings in the ThreatQ UI.
3. Access the connector's cron file in the /etc/cron.d/ directory and remove the # from the beginning of the file.  

Automatic CRON configuration has been removed from this script. To run this script on a recurring basis, use CRON or some other jobs scheduler. The argument in the CRON script must specify the config and log locations.

Add an entry to your Linux crontab to execute the connector at a recurring interval. Depending on how quickly you need updates, this can be run multiple times a day (no more than once an hour) or a few times a week.

In the example below, the command will execute the connector every two hours.

1. Log into your ThreatQ host via a CLI terminal session.
2. Enter the following command:
   crontab -e

   This will enable the editing of the crontab, using vi. Depending on how often you wish the cronjob to run, you will need to adjust the time to suit the environment.

3. Enter the commands below:

   Every 2 Hours Example

   0 */2 * * * /opt/tqvenv/<environment_name>/bin/tq-conn-imap -c /etc/tq_labs/ -ll /var/log/tq_labs/ -v3  

   Copy Code

4. Save and exit CRON.

PDF Guides