Revisiting Past Assessments To Refine Future Ones

Some agencies task their intelligence analysts with making predictions about near-term as well as “over-the-horizon” threats. These can be specific, such as the potential targeting of an upcoming local event, or they can be broader, like the types of international or domestic threats that are most anticipated in an upcoming year or beyond. But unlike the analysis of a past event, which tends to offer a stronger and more defensible result, foretelling the actions of a person or group, or discerning intent, is a tricky business, even for experts because it involves anticipating human behavior.

At the same time, there are the analyst’s own human tendencies to overcome, specifically, unintended biases. Recency bias can lead one to believe a current trend will continue. “Magnitude bias,” when an incident results in a significant loss of life or property, or receives a great deal of public attention, can lead to the conclusion an event is pivotal, rather than an outlier. Political bias may be personal, embedded in an agency, or exist at levels beyond an agency’s control. It can push intelligence production that aligns with legislative priorities.

Despite these challenges, when agencies request forecasts, analysts generally provide them. In these cases, follow your agency’s guidance as to its preferred analytical methodology. Given a choice, a structured thinking technique is preferred because of its intrinsic capacity to reduce bias, as well as its ability to document the path you took to reach your conclusion.

Because forecasts are so challenging, it is critical to revisit past assessments when “prediction dates” expire. If your outlook was inaccurate, you might be able to improve future projects by identifying weaknesses in the process, and spotting where you might have taken a bad turn.

The most important step in “checking the work” is maintaining a comprehensive and consistent database of incidents of the type cited in the forecast. For example, on 1 March 2021, the Office of the Director of National Intelligence issued “(U) Domestic Violent Extremism Poses Heightened Threat in 2021.” Since it predicted an upturn in domestic violent extremist (DVE) activity during 2021, a database of all DVE incidents for that year should be maintained. This prediction should be revisited once all of the DVE cases from the time period cited in the forecast are adjudicated and closed.

Checking your work by revisiting past assessments is good practice. For assessments that involve forecasting and predictions, it is important to note whether your outlook proved correct, and/or if refinements to the process can be identified and implemented. For more traditional assessments, which involve adding context and perspective to a past event or set of circumstances, a follow-up may reveal whether the analysis offered value to the consumer or how it might be improved, and also to find if the analysis held up over time.