Updated 2026-11-01
Assessing implementation, operation and effectiveness separately
Operating automated defense and bandwidth control requires checking not only the intended behavior but also the effects on legitimate users and business operations. BASTION records verified facts separately from areas that remain unverified.
This article explains the basic approach to verification and the operational scope publicly described as of 2026-11.
1. Separating implementation, operation and effectiveness #
Implementing a function, deploying it to an environment, observing it run and obtaining a useful effect are different things to verify. A record of a passed test alone cannot guarantee effectiveness in every environment.
We record what was checked, under which conditions and to what extent, making clear where the results apply. If further verification changes a conclusion, the explanation is updated too.
2. Checking impacts on legitimate business operations #
The ability to react to signs of an attack alone does not establish suitability for operational use. We also check whether normal actions are disrupted and whether impacts could spread when multiple users share the same connection source.
When evaluating a new approach, we use an isolated environment or observation-only comparisons according to the potential impact of the change. Passing a limited test is not treated as a guarantee that every operation will be free of problems.
3. Not treating insufficient evidence as success #
Missing observation periods, unavailable results or a lack of opportunities for comparison leave insufficient information for a decision. The absence of recorded anomalies alone cannot be treated as evidence of normal operation or a successful improvement.
BASTION’s improvement evaluation includes a mechanism that takes the sufficiency of observations into account when deciding whether to adopt a change. Incomplete or inconclusive results are retained separately from verified effects.
4. Stating comparison conditions and the limits of conclusions #
When comparing approaches, we align the subjects, conditions and outcomes being evaluated, and check both missed detections and incorrect responses. Checks using the material used for tuning are distinguished from evaluation using different material.
More operational records or a change in detection counts alone does not explain why an improvement occurred. We also consider alternative explanations and record it plainly when no difference can be established.
5. Current operation and the scope of changes #
As of 2026-11, we operate automated security defense and bandwidth control in production in our own environment. Approaches still under research and results from isolated-environment testing are treated separately from functions adopted in production.
Self-improvement is operated through a combination of predefined authorization boundaries, checks, limited application, observation and rollback. Some processes can proceed automatically within an approved scope; changes outside it are considered separately. The benefits of referring to past cases remain under evaluation.
6. Verifying conditions for each deployment environment #
Experience in our own environment does not necessarily produce the same results elsewhere. Deployment conditions are defined after reviewing the business operations involved, available observations, permitted actions and recovery methods.
Deployment design, integration and evaluation are quoted individually after confirming the scope and work involved. This public article outlines the verification policy and does not disclose internal evaluation formulas, control conditions or customer information.
