LLM-generated explanations may contain mistaken numbers or subjects, or unsupported additions. For operational use, what evidence can verify the content matters alongside readability.

This article introduces the principles BASTION emphasizes when verifying output. It does not provide specific internal processing procedures or guarantee that every error can be prevented.

1. Checking the evidence behind an explanation #

Counts, subjects and time periods in monitoring reports affect operators’ decisions. It is necessary to distinguish what an AI-generated text says from what observation records can confirm.

The same number can mean different things when it refers to a different subject or aggregation period. Fluent wording or the appearance of numbers alone does not establish that an explanation is correct.

2. Recognizing the limits of the underlying observations #

Logs and aggregated results are also affected by missing observations, delays, duplication and differences in collection scope. The existence of a record does not necessarily mean it represents everything that actually happened.

We consider which subjects and periods the information covers and what has not been observed. Agreement between an LLM’s output and the records confirms consistency within that scope. It does not guarantee a correct account of the entire situation.

3. Distinguishing content that cannot be verified #

When evidence is insufficient, records conflict with the explanation or processing is incomplete, it is important not to treat the resulting account as verified fact.

Checking numerical consistency is also different from checking whether an explanation of causes or intentions is reasonable. We distinguish what can be compared mechanically from what requires further investigation.

4. Separating output verification from actual actions #

BASTION’s automated defense is governed by defined decision criteria and protective conditions. It is not based on the assumption that an LLM-generated explanation should lead directly to blocking legitimate users or changing settings.

When an improvement proposal is actually applied, its authorized scope and application conditions are also checked. Reviewing an explanation, inspecting a proposal and observing the results after application serve different verification purposes.

5. Distinguishing verification coverage from effectiveness #

Examples of internal processing in earlier public explanations cannot be treated as a current specification that fully audits every output. We describe the scope of implemented checks, records of their operation and evaluation of their effects separately.

Adding checks does not necessarily eliminate missed errors or incorrect findings. If the model, inputs or observation conditions change, verification under those conditions is also necessary.

6. What to check before deployment #

For each environment, we define which outputs will be used, for what purpose and how far their supporting evidence can be verified. If external services are used, this also includes checking the information sent, its destinations, and the handling of customer information and credentials.

Deployment design, integration and evaluation are quoted individually after confirming the scope and work involved. This public article introduces the approach without disclosing internal verification procedures or actual customer logs.