Blue Team Con 2026

Name: LLMs - How to Trust When We Can’t Verify
Start: 2026-09-13T14:00:00-0500
End: 2026-09-13T14:50:00-0500

LLMs - How to Trust When We Can’t Verify

Sunday September 13, 2026 2:00pm - 2:50pm CDT

Swissôtel Chicago

LLM internals remain black boxes. We can’t fully trace concepts through LLM layers, and chains of thought offer some degree of visibility but can be unreliable and may become less straightforward as models grow more advanced.

I’ll review the current state (at conference-time) of LLM interpretability and challenges to trustworthiness. Then I’ll discuss how to establish and set boundaries for trust in LLM systems that may be inherently hallucinatory, deceptive, or otherwise untrustworthy.

We’ll need to blend old and new techniques, none of which is sufficient on its own. This may include:

- Independent monitoring tools for LLM chains of thought
- Approaches to evaluate and measure AI output (e.g., like the approach NIST is developing)
- Service-level agreements (SLAs) establishing accuracy requirements for green-lighting various AI use cases within an organization
- Requiring review by multiple humans at critical points in process flows, tailored to the use case
- Early warning indicators, near-miss reviews, and incentives for early-warning and near-miss reporting
- Cross-organizational information-sharing on lessons learned (e.g., via ISACs) to prevent individual firms from duplicating the same mistakes in isolation

The goal is for participants to:
1. Understand the current state of LLM interpretability and challenges to trustworthiness
2. Identify strategies for establishing appropriate boundaries for and degrees of trust in LLM systems
3. Develop a list of techniques to tailor for their organizations' needs

Speakers

Stephanie Losi

Independent consultant

Stephanie is an independent technology risk consultant and writes the Risk Musings newsletter. Previously, she worked as a bank examiner and senior bank examiner at the Federal Reserve Bank of New York, focusing on IT and operational risk assessments of large financial firms. Her interests include operational resilience, risk management of AI and other emerging... Read More →

Sunday September 13, 2026 2:00pm - 2:50pm CDT
Swissôtel Chicago 323 E Wacker Dr, Chicago, IL 60601, USA

Talk Track 3

Audience Intermediate
Subject Governance / Risk / Compliance
global Y

Tickets

Success you have shared X tickets.

Blue Team Con 2026

Stephanie Losi

Get help with the event

Choose your checkout method

Success you have shared X tickets.

Blue Team Con 2026

Stephanie Losi

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Get help with the event