Current ThreatQ Version Filter

Reports, Logs, Synchronization

You can review AGDS import and export process details from the following:

Console Output
Sync Export Report
Sync Import Report
Cron Logs
Synchronization Database
Application Logs

Console Output

You can view live statistics of the AGDS process when performing a on-demand export or import. The information will be appear in the same terminal window that you used to enter the command.

The output includes details such as:

Progress and Phase Transitions
Object Counts
File Write Status
Fatal Errors regarding memory, permissions, and disk.

Sync Export Report

The Sync Export report is generated at the end of the export process and is bundled within the export tarball file. The file name is sync-export.txt and is located within the sync directory of the tarball.

The report provides information on:

Flags used
Object counts
Files created
Execution time

Sync Import Report

The Sync Import Report is generated at the end of the import process and uses the following naming format: <tqSync_filename>-sync.import.txt.

The report provides information on:

Flags used
Object counts
Files created
Execution time

AGDS Cron Logs (Export)

Cron-based AGDS logs are only available if cron logging has been enabled. You can enable cron logging by adding the following to the cron command:

>> /var/log/threatq/agds-export.log 2>&1

Example

/var/lib/rancher/rke2/bin/kubectl exec --namespace threatq --stdin --tty deployment/api-schedule-run -- ./artisan threatq:sync-export --target=export/runs --include-deleted=Y --include-investigations=Y >> /var/log/threatq/agds-export.log 2>&1

Using the command example above, the logs will be located at: /var/log/threatq/agds-export.log.

The AGDS logs allows you to check for the following:

Command Start and End timestamps
Run Failures (you should focus on repeated failures)
Permission or kubectl errors
Sudden terminations

Recording Handling and Synchronization Tables

The record handling for Air Gapped Data Sync (AGDS) is managed through a dedicated database table called synchronizations, which serves as the authoritative system of record for all AGDS export and import operations. Every time an AGDS export or import is initiated, ThreatQ creates a synchronization record that tracks the lifecycle, configuration, and outcome of that run.

A Synchronization record is finalized when the command run has completed. At this time, the finished_at column is filled with the completion date/time, and the report_json column is updated to include information about the run (object counts, files created, etc).

The synchronization table columns are as follows:

Column	Details
`id`	The auto-incremented id for the Synchronization record.
`type`	The Synchronization direction - either export or import.
`started_at`	The date and time the command run was started.
`finished_at`	The date and time the command run completed. This column will read as `NULL` until the sync action (export or import) has been completed.
`config_json`	A JSON representation of the command run configuration.
`report_json`	A JSON representation of the command run parameters (command line options, object counts, files created, etc).
`pid`	The process id of the command run.
`hash`	A unique identifier for a command run. The Synchronization record hash column is automatically calculated as an MD5 of the `config_json` column on record creation.
`created_at`	The date and time the Synchronization record was created.
`updated_at`	The date and time the Synchronization record was updated.

Application Logs

You can look for issues not present in the console view or AGDS reports in the pod's logs such as:

Database errors
PHP exceptions
Serialization failures
Sync table errors

To check application logs for AGDS:

Access the API pod:
kubectl exec -n threatq -it deployment/api-schedule-run -- bash
Navigate to the logs directory:
cd /var/lib/threatq/logs
Run the following command to show the most recent events (last 200 lines of the log):
tail -n 200 laravel.log