Aegis System

NOTE: We're still building the Readme files for Aegis.

Aegis is a security automation tool that is designed to coordinate between your vulnerability management solution and your vulnerability remediator while providing reliable and up to date information. No longer is the tedium of emailing spreadsheets and trusting that users are remediating vulnerabilities necessary as Aegis generates tickets that are tracked and confirmed remediated by the automation.

Users can trigger rescans of a vulnerability by marking a ticket as fixed, or letting Aegis passively rescan tickets as they become due. Aegis can automatically create dynamic scan templates and target one or many IPs with one or many scan engines to ensure the patch or configuration change was successfully implemented. If it fails, the ticket is reopened, and the proof can be added as a comment to the user. Watch your tickets close-out automatically and let users track open vulnerabilities in real time. Aegis can handle exception and false positive tracking, and can notice when a device is taken offline. Aegis tracks all scans that cover a ticket during its remediation, which is great to have when auditors come!

Installing Aegis:

• Go 1.11 or higher installed
• $GOPATH set
• Have ~/.aws/credentials and ~/.aws/config setup (or some other form of AWS authentication)

Have ready

• Credentials for all API accounts
• An enumeration of the asset groups you plan on rescanning (comma separated)
• AWS KMS encryption key
  • Must be symmetric

Nexpose

• Templates for your vulnerability/discovery scans existing in your Nexpose instance and their IDs
• Nexpose site ID (integer) that you will be using for vulnerability rescans

Qualys

• SearchList and OptionProfile for your vulnerability scans in your Qualys instance and their integer IDs
• OptionProfile for your discovery scans in your Qualys instance and it's integer ID
• A list of all the group IDs that require external scanners (comma separated)

JIRA

• The project key for the project you plan on storing tickets
• If using your own ticket transition workflow/ticket schema, mapping must be done in the JIRA source config
  • The default ticket schema utilized by Aegis is outlines in setup/jira_ticket_schema.csv if you do not want to do the mapping
• Need your JIRA workflow XML
  • JIRA Administration (cog) -> Projects -> Workflows (on left) -> Select "actions" button next to desired workflow -> Export as XML
• (optional) a separate CERF project for tickets that remediators create for tracking exceptions and false-positives
• (optional) JIRA supports Oauth, if you have Oauth credentials, they can be set in the JIRA source config in the database after installation

Database

• Must be MySQL
• Create a schema in your database for Aegis to utilize

---

Installation (Mac/Linux)

git clone https://github.com/nortonlifelock/aegis

?.
cd ./aegis/cmd || exit
go install aegis.go

cd ../api/listener || exit
go install aegis-api.go

cd ../..

aegis -init-config -scaffold -init-org -cpath $PWD -sproc $PWD/init/procedures -migrate $PWD/init/migrations -tpath $PWD/init

Aegis commands

# Run Aegis
# -config defaults to app.json if not specified
aegis -config app.json -cpath "path to configuration file"

# Create/update DB schema
aegis scaffold

# Create a new application config (same process that's done "aegis init")
aegis init-config

# Create a new organization (same process that's done during "aegis init")
aegis init-org

Base Json Configuration File:

{
  "db_path":"127.0.0.1",
  "db_port":"3306",
  "db_username":"",
  "db_password":"",
  "db_schema": "Aegis",

  "logpath":"",

  "key_id":"",
  "sns_id":"",
  "kms_region": "",
  "kms_profile": "",

  "log_to_console":true,
  "log_to_db":true,
  "log_to_file":true,
  "log_no_delete": true,
  "debug":true,

  "api_port":4040,
  "websocket_protocol":"wss",
  "transport_protocol":"https",
  "ui_location":"localhost:4200",

  "path_to_aegis": ""
}

The install-config script will generate this application config with your input and name the file app.json. The purpose of each field are as follows:

# Configurations used for the Aegis jobs
db_path
    The URL or IP address where your database resides
db_port
    The port on which your database listens
db_username
    The username that is used to authenticate against the database
db_password
    The (encrypted) password that is used to authenticate against the database
    The password is encrypted with the KMS key you provide by the install-config scirpt
db_schema
    The name of the schema that Aegis will utilize

logpath
    The path at which the logs are stored

key_id
    The ID of the (symmetric) KMS key that is used for the encryption
kms_region
    The region (e.g. us-east-1/us-west-1) that the KMS key lies in
kms_profile (optional)
    If using file authentication for AWS (~/.aws/credentials) that contains multiple profiles, you can specify which one here
sns_id (optional)
    The ID of the SNS topic that is used to publish critical/fatal logs

log_to_console
    Boolean controlling whether logs are printed to the console
log_to_db
    Boolean controlling whether logs are stored in the database 
log_to_file
    Boolean controlling whether logs are written to log files
log_no_delete
    Boolean controlling whether logs stored in files are deleted after a day
debug
    Boolean controlling whether debug logs are processed
 
# Configurations used by the Aegis API
api_port
    The port on which the Aegis API listens
websocket_protocol
    The protocol that is used for websocket connections [ws|wss]    
transport_protocol
    The protocol that is used for http connections [http|https]
ui_location
    The URL that the API lies at (used to check the origin header of HTTP requests)
path_to_aegis
    The path where the Aegis system is stored

• Step 4: Execute Aegis

aegis -config app.json -cpath "path to configuration file"

---

Major processes

Rescan Process

Jobs involved:

1. RescanQueueJob

   Starts automatically and runs continuously. This job monitors JIRA and kicks off a RescanJob for tickets in particular statuses. There are four types of RSQ (the type being controlled by the Payload in the JobHistory entry for the RSQ):

   1. Normal - looks tickets in a Resolved-Remediated status. These are for standard vulnerability rescans. Tickets are moved to Closed-Remediated once remediation has been confirmed by a scanner, or reopened if the scanner still detects the vulnerability.
   2. Decommission - looks for tickets in a Resolved-Decommission status. These are for confirming a device has been moved offline. These tickets are moved to Closed-Decommissioned once a scanner has confirmed they are a dead host, or reopened if the host is still alive.
   3. Passive - looks for tickets in an open, reopened, in-progress, resolved-exception status created within 20 days and due within 15 days
   4. Exception - looks for tickets in Closed-Exception status with an exception that expires within 30 days and verifies that the ticket has the vulnerability fixed before the exception expires. If the vulnerability is fixed the ticket is moved to Closed-Remediated. If the vulnerability is not fixed, the ticket is left in Closed-Exception.

2. RescanJob

   Created by the RescanQueueJob. This job is responsible for creating the scan in Qualys/Nexpose for the devices/vulnerabilities reported in the tickets.

3. ScanSyncJob

   Monitors scans created by the RescanJob. When it notices a scan has finished, the ScanSyncJob queues up a ScanCloseJob to close out the tickets based on the results of the scan

4. ScanCloseJob

   Pulls the detection information for the scanned hosts from Qualys/Nexpose and uses that detection information to close out the tickets. This job updates the LastFound/LastUpdated/LastChecked date of the ticket. If no detections were reported for the device and the device was not reported as dead, the ticket is moved to Scan-Error (likely host data purge)

   1. Normal
      1. If the device is reported as dead, it’s tickets are moved to resolved-decommission
      2. If the last updated date is before the scan date, the ticket is moved to scan-error (likely authentication issues)
      3. If the detection was marked as fixed, or no detection was returned for that device/vulnerability combo, the ticket is moved to closed-remediated
      4. If the device/vulnerability combo still exists and the ticket is in resolved-remediated, the ticket is reopened
   2. Decommission
      1. If the device is reported as dead, it’s tickets are moved to closed-decommission and the device is marked as decommissioned in the Ignore table
      2. If the device is not reported as dead and the ticket is in resolved-decommission, the ticket is reopened
   3. Passive
      1. If the device is reported as dead, it’s tickets are moved to resolved-decommission
      2. If the detection was marked as fixed, or no detection was returned for that device/vulnerability combo, the ticket is moved to closed-remediated
      3. If the device/vulnerability combo still exists and the ticket is in resolved-remediated, nothing happens
   4. Exception: same as passive

Synchronization Processes

Jobs involved:

1. ExceptionJob

   Creates entries in the Ignore table by pulling recently updated Closed-FalsePositive tickets and Closed-Exception tickets with non-empty CERFs

2. CloudSyncJob

   Sync cloud tag information into the Tag table of the database. We correlate could assets to scanner assets by correlating a scanner asset group to a cloud environment in the AssetGroup table (using ScannerSourceID and CloudSourceID fields)

3. VulnSyncJob

   Creates and updates entries in the VulnerabilityInfo table

4. TicketSyncJob

   Pulls recently updated tickets from JIRA and keeps the DB information on the ticket fresh

5. AssetSyncJob

   Pulls detection information from Qualys/Nexpose API and stores it in db. Uses the vulnerability information pulled and stored during the VulnSyncJob. Also stores device information in DB. Looks at Ignore table and attaches it to detection if appropriate. Global exceptions are also checked here. Detections created/updated here are what are processed during a ticketing run

Ticketing Process

Jobs involved:

1. TicketingJob

   1. Unfixed detections with no ignore ID attached that were recently updated/created are pulled from the DB
   2. If they have a CVSS lower than what is specified in the CVSS min field in the organization payload, they are skipped.
   3. We first check the DB, and then JIRA for an existing ticket in the following statuses: Open, In-Progress, Reopened, Resolved-Remediated, Resolved-FalsePositive, Resolved-Decommissioned, Resolved-Exception
   4. We do mapping from cloud tags to ticket fields (if a tag mapping is specified)
   5. The ticket is assigned according to the org’s AssignmentRules table
   6. The ticket is created in JIRA and has an entry made in the Ticket table of the DB

---

Other Processes

---

Notes for developers

Creating a new job

1. The go file for the job's code is placed in aegis/internal/implementations/
2. The name of the file should reflect the name of the job
3. The file should contain a struct for your job as well as all methods that this struct implements
4. The job struct should implement a function with the following method signature. This method will act as the entrypoint for your job

Process(ctx context.Context, id string, appconfig domain.Config, db domain.DatabaseConnection, lstream log.Logger, payload string, jobConfig domain.JobConfig, inSource []domain.SourceConfig, outSource []domain.SourceConfig) (err error)

5. The struct for the job needs to be registered with the job runner so the job runner knows which struct is tied with which job. This is done in aegis/internal/implementations/registration.go
6. An entry for the job must be made into the database. Create a SQL migration file in aegis/init/migrations. Scaffolding should be ran after the file is created for the SQL to be executed against the database. Example SQL below

SELECT S.Id
INTO @scanner_id
FROM SourceType S
WHERE S.Type = 'ImageScanner' LIMIT 1;

SELECT S.Id
INTO @ticket_id
FROM SourceType S
WHERE S.Type = 'TicketEngine' LIMIT 1;

INSERT INTO Job(Struct, SourceTypeIn, SourceTypeOut, Priority, CreatedBy) VALUE ('ImageRescanJob', @scanner_Id, @ticket_id, 4, 'ryan.everhart@nortonlifelock.com');

7. By convention, the API integration where the job pulls data from is called the insource and the API integration that the job writes to is called the outsource. The type of this API integration (e.g. TicketEngine, ImageScanner) are referenced in the SourceType table.

How the job runner works

1. The job runner checks the JobHistory table every 60 seconds (change with -run_wait flag) for entries with a StatusID 1
2. All entries found with status 1 are loaded into memory, and have their StatusID values in the DB set to 2 (processing)
3. The job runner looks at the JobID column for these entries. This JobID corresponds to the ID column in the Job table. The corresponding Job name is found from this table
4. The job name is looked up in the job registry (aegis/internal/implementations/registration.go), and the equivalent job struct is found
5. The job runner loads configurations from the DB for the job struct, and passes them to the Process method
   1. The job runner takes the ConfigID from the JobHistory entry, and uses it to pull from the JobConfig table using the ID column
   2. The job runner takes the DataInSourceConfigID from the JobHistory entry, and uses it to pull from the SourceConfig table using the ID column
   3. The job runner takes the DataOutSourceConfigID from the JobHistory entry, and uses it to pull from the SourceConfig table using the ID column

How scaffolding works

How to make schema changes

Important database tables and fields

API funnel & concurrency settings

---

Other

How duplicates are identified

Encrypting passwords