Limited Series Podcast

TLDR: Key Takeaways

• Accurate intelligence isn't enough - without sufficient precision, it's difficult to make informed decisions. Reporting needs enough detail to justify action while remaining grounded in evidence.
• Trust is earned over time through transparent methodology, not confident conclusions. Showing how an assessment was reached allows consumers to evaluate and reuse the analysis, even if they disagree with the final conclusion.
• Every intelligence product is a balance between speed, completeness, and confidence. Sometimes timely reporting is more valuable than fully developed reporting - as long as assumptions and limitations are clearly communicated.
• Threat actor names are far less important than understanding the underlying evidence and methodology. Labels help analysts communicate efficiently, but they should never replace the analysis itself.
• Good intelligence is written for its audience. Reports should communicate uncertainty, explain technical concepts appropriately, and enable the reader to take meaningful action instead of simply consuming information.

What makes intelligence trustworthy?

It's tempting to assume the answer is simply "being right," but in practice the answer is much more nuanced. Intelligence consumers don't just need correct conclusions, they need enough context, evidence, and transparency to understand why those conclusions were reached and whether they're worth acting on.

In this episode, Kali is joined by Vertex co-founder, visi stark, along with Vertex analysts, Jen Kolde (thesilence), Mary Beth Lee (savage), and Ryann Hallback (reign) to explore the relationship between accuracy, precision, and trust in cyber threat intelligence. Along the way, the conversation expands into topics like confidence language, analytical transparency, public reporting, threat actor naming conventions, marketing, and the growing influence of AI-generated reporting.

Accuracy Without Precision Isn't Actionable

The discussion begins with an important distinction:

• Accuracy describes whether an assessment is correct.
• Precision describes how specific and detailed that assessment is.

An accurate statement like "it will rain this month" may be technically correct, but it provides almost no value for decision making. By contrast, a forecast that predicts a 60% chance of rain after 6 p.m. gives consumers enough detail to decide whether carrying an umbrella - or postponing an outdoor event - is worth the cost.

Throughout the conversation, the panel argues that intelligence exists to enable decisions. If reporting isn't actionable, even accurate conclusions lose much of their value.

Trust Comes From Showing Your Work

One of the recurring themes throughout the episode is that trust isn't created through confident writing: it's created by transparency.

Rather than asking consumers to simply believe an assessment, trustworthy intelligence should make it possible to understand how the conclusion was reached. That doesn't necessarily mean exposing sensitive sources or proprietary collection methods, but it does mean clearly explaining the evidence, assumptions, analytical process, and confidence behind an assessment.

Even when analysts disagree with a report's conclusions, transparent reporting allows them to reuse portions of the evidence while arriving at different conclusions of their own.

Confidence Should Be Communicated Not Hidden

Intelligence is rarely certain. Many of the most valuable assessments involve predicting future behavior, which means uncertainty is unavoidable. Instead of presenting every conclusion as fact, analysts should communicate their confidence levels honestly and consistently.

The panel discusses how confidence language only becomes meaningful when organizations evaluate their own assessments over time. Trust isn't binary. it develops through a history of producing reliable analysis, acknowledging uncertainty, and revisiting previous conclusions when new evidence emerges.

The Naming Debate Misses the Bigger Problem

One of the liveliest portions of the discussion centers around threat actor naming conventions.

The group argues that the industry often spends disproportionate time debating whether two organizations use the same threat actor names while paying far less attention to the quality of the underlying analysis.

Names serve an important purpose. They provide shorthand that helps analysts communicate efficiently, but they don't inherently improve intelligence.

What matters far more is:

• How the activity was attributed
• What evidence supports the assessment
• How confident the analyst is
• Whether the reporting enables defenders to act

The panel suggests that methodology and transparency deserve significantly more attention than industry-wide naming debates.

Timeliness vs. Completeness

Another recurring tension is the tradeoff between publishing quickly and publishing comprehensively.

In fast-moving incidents, organizations may need to release reporting before every detail has been fully validated. The panel agrees this can be appropriate as long as reports clearly communicate:

• what is known,
• what remains uncertain,
• what assumptions are being made,
• and what information may change as the investigation continues.

Being explicit about limitations helps preserve trust even when reporting must prioritize speed.

Writing for the Right Audience

Toward the end of the episode, the discussion shifts from intelligence analysis to communication itself.

Analysts often write for other analysts by default, but public reporting frequently serves multiple audiences simultaneously:

• security practitioners,
• executives,
• customers,
• marketing teams,
• and the broader security community.

That creates difficult tradeoffs between technical precision, readability, brevity, and business communication.

The group emphasizes that effective reporting isn't simply about writing well, it's about understanding who needs the information, what decisions they're trying to make, and how much detail they actually require.

AI Raises the Stakes

The conversation also touches on the increasing use of large language models in threat intelligence. While AI can accelerate writing and repackage information for different audiences, the panel warns that current models often produce overly confident prose while obscuring uncertainty or overstating conclusions.

As AI-generated reporting becomes more common, clear methodology, confidence language, and transparent analytical reasoning become even more important for distinguishing trustworthy intelligence from polished but unsupported assertions.

Final Thoughts

The episode closes with a reminder that trustworthy intelligence isn't achieved by chasing perfect terminology or unanimous agreement across the industry.

Instead, it comes from consistently producing reporting that is:

• Accurate
• Precise enough to support decisions
• Transparent about evidence and limitations
• Appropriate for its audience
• Willing to acknowledge uncertainty and improve over time

Ultimately, intelligence earns trust not because it sounds confident, but because it consistently helps people make better decisions.