Current ThreatQ Version Filter

Scheduled Exports

You can utilize CRON to run recurring, incremental Air Gapped Data Sync exports from a network-connected source system to support ongoing updates of an air-gapped ThreatQ deployment. Instead of requiring administrators to manually manage dates or track changes, cron for AGDS relies on ThreatQ’s internal synchronization records to determine what data should be exported on each run.

When an export is executed via cron, ThreatQ calculates a configuration hash based on a fixed set of command-line flags (such as --target, --include-deleted, and --include-investigations). If a previous successful export exists with the same hash, ThreatQ automatically uses the start time of the last completed run as the starting point for the next export. This allows cron-based exports to behave incrementally, pulling only new or updated data since the last run. If the hash changes, ThreatQ treats the export as a new sequence and resets the incremental cursor.

CRON jobs must be run by a non-root user that installed RKE2.

Determine which, if any, CRON configuration flags to use. The CRON configuration flags listed below must be the same for every run, but they only need to be specified if different from the defaults.

Option	Details	Default	Flag	Example
Target Directory	The directory where the export tarball file will be placed.	/tmp	`--target`	`--target=export/runs`
Include ThreatQ Investigations and Tasks	Include ThreatQ Investigations and Tasks in the export tarball file.	N	`--include-investigations`	`--include-investigations=Y`
Include Deleted Objects	Include soft-deleted objects in the export tarball file.	N	`--include-deleted`	`--include-deleted=Y`

The example command below includes all three flags as they differ from the default settings

kubectl exec --namespace threatq --stdin --tty deployment/api-schedule-run -- ./artisan threatq:sync-export --target=export/runs --include-deleted=Y --include-investigations=Y

Retrieve the DB password:
kubectl get secret mariadb-root --output go-template='{{range $k,$v := .data}}{{printf " %s: " $k}}{{if not $v}}{{$v}} {{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}' -n threatq
Run the initial export command along with any Export Command Flags needed:
kubectl exec --namespace threatq --stdin --tty deployment/api-schedule-run -- ./artisan threatq:sync-export --target=export/runs --include-deleted=Y --include-investigations=Y
Enter the DB password when prompted.
Set up CRON for the non-root user:
sudo crontab -u <non-root_user_that_installed_RKE2> -e
Enter the following command while replacing target placeholder as well as adding any other export flags you need. Remove the --include-deleted and --include-investigations flags if you do not want to include soft-deleted files and investigations.
/var/lib/rancher/rke2/bin/kubectl exec --namespace threatq --stdin --tty deployment/api-schedule-run -- ./artisan threatq:sync-export --target=<relative_path> --include-deleted=Y --include-investigations=Y

You can add the following to the end of the command to add logging for a chronological execution history: >> /var/log/threatq/agds-export.log 2>&1. The >> appends output (does not overwrite), the 2>&1 captures errors (stderr) and normal output (stdout).

Example:

/var/lib/rancher/rke2/bin/kubectl exec --namespace threatq --stdin --tty deployment/api-schedule-run -- ./artisan threatq:sync-export --target=export/runs --include-deleted=Y --include-investigations=Y >> /var/log/threatq/agds-export.log 2>&1

Steps for Larger Data Sets

Attempting to perform a full run on larger data sets can result in decrease in system performance.

ThreatQuotient recommends that you use the --end-date flag to specify an upper limit on the date range pulled. Multiple runs will be necessary to process all data up to the current date. Once the current date has been reached, the --end-date flag is no longer necessary.

Steps for Larger Data Sets Starting at the Beginning of Time

The following example does not use the start-date flag so the command will use the start date default: 1970-01-01 00:00:00.

kubectl exec --namespace threatq --stdin --tty deployment/api-schedule-run -- ./artisan threatq:sync-export --target=export/runs --include-investigations=Y --end-date="2017-01-01 00:00:00"

Steps for Larger Data Sets Starting at a Specific Start Date

The --initial-start-date flag be used for the first run if only a subset of data needs to be processed up to the current date.

kubectl exec --namespace threatq --stdin --tty deployment/api-schedule-run -- ./artisan threatq:sync-export --initial-start-date="2016-01-01 00:00:00" --target=export/runs --include-investigations=Y --end-date="2017-01-01 00:00:00"

The --end-date option flag should be used for each additional run until current date has been reached.