Automating Power-Ups to Aid Data Ingest, Enrichment, and Analysis

by savage | 2023-03-24

Synapse Power-Ups are Storm services and packages designed to facilitate activities such as data ingestion and enrichment to analysis and assigning additional tasking. Power-Ups are an integral part of the workflow for us analysts at The Vertex Project, and we often choose to automate many of them. This gives us several advantages:

  • Improved efficiency - automation runs in the background and ensures that tasks are completed in a timely manner;

  • Greater consistency in workflow - automated processes execute the same way each time, whereas relying on an analyst to manually do something can result in variations, especially if the analyst makes a mistake or forgets entirely;

  • Helps prevent analyst burn-out - leveraging automation in place of manual effort helps reduce the risk of analyst burn-out by alleviating the need for analysts to spend time on tedious tasks and allowing them to make better use of their time and effort.

In this piece, we’ll highlight different examples of ways analysts might automate Synapse Power-Ups to assist with their workflow. Our goal is for this to serve as a starting point for others looking for ways to automate Power-Ups in their environment.

Scheduling Power-Ups with Cron Jobs

We can create cron jobs that execute Power-Ups at specific times (either once or on an ongoing basis) to support data ingestion, enrichment, and analysis. The cron job below, for example, will use the Synapse-TOR Power-Up and run the tor.sync command on the 37th minute of every hour to import an updated list of TOR exit nodes into the Power-up testing View:

_images/cron.webp

Some other examples of cron jobs that analysts may use for research and analysis tasks include:

Ingesting Data:

Use the Synapse-MISP Power-Up to regularly import published MISP events into Synapse. The cron job can run the following Storm code:

misp.sync

Use the Synapse-Twitter Power-Up to regularly ingest tweets from a Twitter feed and scrape them for indicators of compromise:

twitter.feed.pull

Enriching Data:

Pipe all hash nodes with a #rep tag and no accompanying file:bytes node(s) to vxintel.download to check for and download associated bytes through the Synapse-vxIntel Power-Up. The cron job can run the following Storm query:

hash:sha256#rep hash:sha1#rep hash:md5#rep -{-> file:bytes} | vxintel.download

Use the Synapse-Alienvault Power-Up to enrich all inet:fqdn and inet:ipv4 nodes created within the last week and tagged with #rep, #cno.threat, or #cno.mal to collect passive DNS A records:

inet:fqdn inet:ipv4 +.created@=(now, -7 days) +(#rep or #cno.mal or #cno.threat) | alienvault.otx.pdns

Analyzing Data:

Use the Synapse-YARA Power-Up to run all file:bytes nodes tagged with #rep, #cno.threat, or #cno.mal (but excluding those tagged #cno.mal.* and any file:bytes nodes with associated it:app:yara:match nodes) against YARA rules tagged with #cno.detect:

file:bytes +(#rep or #cno.threat or #cno.mal) -#cno.mal.* -{-> it:app:yara:match} | yara.match --rules ${it:app:yara:rule#cno.detect}

Executing Power-Ups with Event-Based Triggers

In some cases, creating an event-based trigger can be a more appropriate solution for some tasks rather than relying on a scheduled cron job. Triggers are better suited for on-demand tasks in particular, as they are designed to execute as soon as a predetermined condition is met. The trigger below, for example, is designed to enrich inet:ipv4 nodes tagged with a #rep tag:

_images/trigger.webp

When an analyst working in the Power-up testing View adds a #rep tag to an inet:ipv4 node, the trigger will pipe that node to the maxmind command to pull in geolocation data through the Synapse-Maxmind Power-Up, and the alienvault.otx.pdns command to pull in passive DNS data through the Synapse-Alienvault Power-Up.

Some other examples of triggers that analysts may use to support research and analysis tasks include:

Enriching Data:

When a hash:sha256, hash:md5, or hash:sha1 node is tagged with #rep, use the Synapse-vxIntel Power-Up and Synapse-VirusTotal Power-Up to download the file and provide additional file information:

+(hash:sha256 or hash:sha1 or hash:md5) | vxintel.download | virustotal.file.report | virustotal.file.behavior

When an inet:fqdn node is tagged with #cno.mal or #cno.threat, use the Synapse-Alienvault Power-Up to pull in passive DNS data, the Synapse-Nettools Power-Up to import WhoIs and DNS A records, and the Synapse-URLScan Power-Up to populate additional URLs:

+(#cno.mal or #cno.threat) | alienvault.otx.pdns | nettools.dns | urlscan.enrich

If an inet:email node is tagged with #cno.threat, use the Synapse-WhoXY Power-Up to perform a reverse search for related WhoIs records:

| whoxy.whois.reverse

Analyzing Data:

When a file:bytes node is tagged with #cno.threat or #cno.mal and is not linked to an it:app:yara:match node, use the Synapse-YARA Power-Up to run the file against YARA rules tagged with #cno.detect:

+(#cno.mal or #cno.threat) -#cno.mal.* -{-> it:app:yara:match} | yara.match --rules ${it:app:yara:rule#cno.detect}

If a file:bytes node matches a YARA rule, creating an it:app:yara:match node, push the #cno.mal.* tag from the it:app:yara:rule node to it:app:yara:match node and the matching file:bytes node while also tagging both with #int.review.yara to flag them for an analyst to review

+it:app:yara:match

{ -> it:app:yara:rule $mals = $node.globtags(cno.mal.*)}
-> file:bytes
for $m in $mals {[ +#cno.mal.$m +#int.review.mal.$m ]}

Creating Follow-on Tasking:

When the tag #int.review is applied to a node, use the Synapse-Jira Power-Up to create a ticket tasking an analyst to review annotations on the node. An example of how this Storm code might look for a tagged inet:fqdn node would be:

+#int.review  -{-(refs)> proj:ticket}

switch $node.form() {

"inet:fqdn": {
    $fqdn=$node.value()

    //defang fqdn
    $fqdn=$fqdn.replace(".", "[.]")

    //create ticket description
    $text=$lib.str.format("FQDN: {fqdn}", fqdn=$fqdn)
    $desc=$lib.import(jira).text2ADF($text)
    $fields=$lib.dict(labels=("int.review",),description=$desc)
    jira.issue.create MDR support "Research request for tagged node review" --fields $fields
    }
 }

Automating Power-Ups to Improve Workflow Efficiency

We can improve our consistency and efficiency by automating Power-Ups to assist with data ingestion, enrichment, and analysis, among other tasks. Using cron jobs and triggers to automate tasks ensures that those actions are done consistently, freeing us from needing to remember when and how to do a task, and allowing us to use that time for other obligations. Event-based automation such as triggers, for example, can help us to operate more efficiently by ensuring that tasks are completed at machine speed so that data is enriched and tags are pushed to provide us with the data and context that we need to continue with our analysis. While the specifics of how we might decide to craft automation will depend on our operations and use cases, hopefully this blog can help serve as a starting point with ideas for how automating Synapse Power-Ups can facilitate analyst workflows.

For more information about Synapse, including tips, tricks, and use cases, join our Slack community, check out our videos on YouTube, and follow us on Twitter.