For Analysts: How to Properly Prep, Analyze, and Conclude Investigations

The “Mindset” Challenge 🔗︎

Why and How to Establish Effective Analyst Mindset 🔗︎

OSINT (Open-Source Intelligence) is becoming more complex in terms of both sources and methods. Information is being made available in ways that never existed before, including online expressions of personal sentiment, photographs of locales and events, and publicized social and professional networks. Compounding computer power and data-science techniques allow for the retention and processing of mass quantities of publicly available data. Machine learning, computer algorithms, and automated reasoning further expand the capacity to process information and discern the specific data that are of intelligence value.

Modern OSINT requires extensive supplementary acquisition, processing, and exploitation to produce an open-source product – this product must then be disseminated and integrated into an all-source product. As this process becomes both broader and more intricate, an analyst must seek to organize their work in such a way as to deal with the problems inherent in large, complex data sources. To this end, here we will discuss three core considerations of intelligence analysis to settle you into “The Analyst Mindset”.

This content will establish itself upon what Jack Davis identified as Richards J. Heuer’s central ideas when he wrote the foreword to Psychology of Intelligence Analysis, paraphrased as follows:

• Human beings have trouble dealing with uncertainty. This could be either inherent uncertainty that arises through natural means i.e., the ‘natural fog’ which is normal for complex issues, or it could be induced uncertainty. I.e., ‘man-made fog’ resultant from denial and deception operations.
• Whilst increased awareness of the principles of cognition and the several types of biases is valuable, these do little to help an analyst deal with uncertainty.
• The key to improving intelligence analysis capabilities is the implementation of critical thinking, structuring of information, challenging assumptions, and the exploration of alternative interpretations.

The biases noted above in the second point are mental constructs based on experience and are a range of assumptions and expectations about the world and specific topics. Naturally, these constructs weigh upon how analysts view and interpret information – either consciously or subconsciously. The irony of these constructs is that they both help to deal with vast amounts of data which would otherwise be incomprehensible, but also cause analysts to overlook, reject or forget information that may not fit within their existing assumptions and expectations, thus reducing objectivity in information analysis.

The key points regarding constructs that should be remembered going forward are that analysts perceive what they expect to perceive; opinions, once formed, are resistant to change; and that conflicts that arise between information gathered may be ignored, with little or no justification.

To address these points, we at Maltego have put together a shortlist of helpful tips to help guide your investigations, from the initial objective hypothesis, through the investigative process, and finally, during retrospectives and the critical analysis phase. This list is not exhaustive, as to list every technique and method would require writing an entire volume on the topic. Remember—a toolbox is not just for tools, but to help you think in diverse ways.

Preparing for An Investigation 🔗︎

How to prepare or set the initial objective for your investigations:

1. Understand how you or your team conduct investigations
2. Create a checklist of questions to be answered
3. Challenge inherited assumptions
4. Reflect and introspect your preparation, both mentally and materially

1. Understand How You or Your Team Conduct Investigations 🔗︎

The first step is to understand how you yourself, or your team are undertaking investigations. Is there a specific workflow and/or playbook which is to be followed? Or are investigators responsible for determining their own goals and processes? Investigators must fully understand what the goals of an investigation are, as this will help direct their activities and determine what data is deemed valuable and what can be de-prioritized.

2. Create a Checklist of Questions to Be Answered 🔗︎

Analysts must fully understand the investigation line and the questions to be answered. There may be more context, explanations, or information to be gathered. These lines of investigation and the questions to be answered direct the investigation and provide more context, which is essential to the framing of the investigation/operation. Alternatively, it may be helpful to identify key stakeholders who may be consulted to offer opinions or add value to an investigation.

3. Always Challenge Inherited Assumptions 🔗︎

An investigation may start without some form of previous work having been done. However, if an investigation is transferred from one analyst, team or department to another, risks may arise. Analysts may find themselves “inheriting” a case or line of investigation. It is imperative that an analyst critically examine any key assumptions that have already been made, in order to remove existent bias and unsubstantiated claims from their starting point.

4. Reflect and introspect your preparation, both mentally and materially 🔗︎

Few investigations are without time pressure or constraints upon an analyst’s capacity. However, “diving right in” should be avoided before the right moment. Take a step back from the investigation and ensure that you are prepared, both mentally and materially, which will increase both efficiency and effectiveness in the long term.

This step boils down to considering the following aspects of an investigation: What do I know, what are my tools, and what are the questions I want to answer? This will help analysts reduce “stray” during the investigation process.

A couple more things to keep in mind:

• What Tools Am I Working With?

Have a toolkit to support your work or to understand which methodologies are attached to the tools to incorporate yourself.

• Understand How “Getting Information” Works

How can analysts preserve data that is identified, and they want to access, even if access is not currently, or no longer, available? How does this process work? To minimize the risk to an investigation, analysts may take precautions such as creating snapshots of social media pages, cataloging, and downloading links found on a forum or identifying access rights requirements.

During the Investigation 🔗︎

There are four principles to keep in mind to guide your investigations:

1. Analysis is driven by evidence
2. Perform a key assumption check
3. Avoid a rush to judgement
4. Use devil’s advocacy

1. Analysis Is Driven by Evidence 🔗︎

Unsupported lines of investigation always need to be critically analyzed. A weak link in reasoning or gaps in an evidentiary chain can inject instability and vulnerabilities into an investigation. The saying “a chain is only as strong as its weakest link” holds true for investigations—especially if the evidence gathered is to be passed along to prosecute a legal case.

2. Performing a Key Assumption Check 🔗︎

Performing such a check will help identify unsupported assumptions and explore how, if these assumptions were to change, the investigation’s conclusion would also change. In doing so, an analyst can explore how they can further strengthen key junctions in their investigation and increase confidence in their ultimate evaluation.

3. Avoid A Rush to Judgment 🔗︎

Much like reflection and introspection, time should be taken to ensure that due weight is given to any judgements. By rushing to make judgements, an analyst may run afoul of the anchoring bias. Future attempts to revise this judgment on the basis of new information or further analysis may result in the analyst not changing their judgement enough—mindsets are quick to form but resistant to change.

4. Use Devil’s Advocacy 🔗︎

A quick and effective method to test the validity of critical assumptions/pivot points is to be the devil’s advocate to find holes in your logic or judgments that are not well supported by the facts.

After the Investigation 🔗︎

After conducting an investigation, we encourage an analyst to reflect and take the following actions:

1. Be aware of the trap of “satisficing”
2. Conduct post-mortem analysis and structured self-critique
3. Encourage investigators to broaden their skillset to adjacent domains

1. Be Aware of The Trap Of “Satisficing” 🔗︎

Concluding an investigation by saying “that’s good enough” is an example of satisficing. Some analysts may be content to choose the first hypothesis which appears good enough, rather than pursuing an investigation through all possibilities and determining a conclusion based upon the greatest consistency with the evidence obtained. To counter this problem, it is essential that analysts have sufficient time to establish a wide range of hypotheses, and work through these hypotheses to conclusion. These conclusions can then be compared and contrasted in a structured manner.

2. Post-mortem Analysis and Structured Self-Critique 🔗︎

This technique allows the analyst to conduct a post-investigation critical analysis of assumptions made, biases encountered, and to identify any inconsistencies that undermined the objectivity of the investigation. This also helps analysts ask questions relating to the process by which information was gathered and processed. Critical analysis seeks to move deeper into the cognitive process—looking not just at an answer, but the ‘math’ behind the answer.

3. Encourage Investigators to Broaden Their Skillset to Adjacent Domains 🔗︎

Analysts do not need to become an expert on everything, but they need to understand overlaps and the environment within which they are working. A detective is much better positioned to examine cell phone data if they understand how SIM cards, cell towers and messaging apps work. Similarly, an analyst investigating the case of a people-smuggling operation may find value in having a basic understanding of blockchain and cryptocurrency.

• Points to remember at this stage:

Expertise is an admirable goal, but if an investigator focuses too much on a particular information domain, they will only ever view information through that lens. The quote “when all you have is a hammer, all problems look like nails” comes to mind. Developing a wide range of knowledge, irrespective of its superficiality or lack of direct applicability, can generate huge value for an analyst. To quote Daniel Kahneman:

The reason this topic is so fundamental to the work of analysts can be seen in the following two examples regarding the results of a faulty intelligence investigation:

Example 1: Financial Damage 🔗︎

• Law Suit Against Government Agency – 2001 Anthrax Attacks

Steven J. Hatfill, an American physician, pathologist, and biological weapons expert was named as a suspect in the 2001 anthrax attacks, although at the time, no criminal charges had been brought against him. His home was raided numerous times by the FBI, he was extensively surveilled, his employment was terminated, and he was hounded by the media.

Ultimately, evidence from witnesses, laboratory access records, and other information proved that Hatfill did not have access to the anthrax strain used in the attack. Hatfill subsequently filed a lawsuit against the US government, which was settled for $4.6 million.

Here we see the risks of rushing to a conclusion, anchoring on a particular suspect, and not engaging in the practice of trying to disprove a hypothesis, as opposed to continuing to seek supporting evidence.

Example 2: Effort Wasted 🔗︎

• Reputational Damage – Curran-Gardner water plant  - 08 November 2012

A water-pump failure in Illinois was initially mistaken as cyberattack on a public utility in the United States because a login to the plant’s computer system took place from a Russian IP address.

After considerable effort and an escalation of the investigation due to the foreign element, it was found that the remote access had been undertaken by Jim Mimlitz. Mimlitz was on vacation at the time and received a request to examine the SCADA computer system. Mimlitz later stated “I could have straightened it up with just one phone call.”

Later examination found that an electromechanical problem was the source of the pump failure, rather than a SCADA system problem.

Here we see the risks of not examining competing hypotheses. If a structured approach had been taken to determine as many plausible reasons for the remote access as possible, and other potential causes for the failure, the investigation would have proceeded more objectively.

Mistakes such as these may lead to reputational damage for the investigating organization and undermine confidence in their investigative competency.

Conclusion 🔗︎

Intelligence investigations rely upon the computational power of software tools to deal with enormous amounts of information—this point is not in contention. However, tools, in and of themselves, do not determine whether the conclusion derived from an investigation is either correct or justifiable. That determination is dependent upon the analyst ensuring that they are adhering to the fundamental principles of intelligence analysis—that they are operating with objectivity and awareness, and that they are employing reasoning techniques and practical mechanisms.

Why is this important? 🔗︎

By considering the points listed above, analysts reap the benefits of an objective, aware and process-oriented investigation. The result is to provide recourse for the issues mentioned above - reduced uncertainty, an increase in the analyst’s ability to identify cognitive pitfalls, and greater capacity to deal with the torrent of information to which they are exposed. As analysts become more practiced with this approach, both the efficiency and efficacy of their work will increase – this leads to better time and effort investment, more value derivation from processed information, and a greater objective, data-driven basis for the conclusions reached.

Don’t forget to follow us on Twitter and LinkedIn and sign up to our email newsletter, so you don’t miss out on updates and news!

Happy investigating!

About the Author 🔗︎

Aaron Dixon

Aaron Dixon is a former member of the New Zealand Military who has spent the last 6 years working as a consultant in the areas of IT Security and Compliance, Data Privacy, Digital Forensics and Cyber Threat Intelligence. He holds a bachelor’s degree with a double major in History and Defense Studies, as well as a Postgraduate Certificate in International Security. His primary areas of interest are terrorism and geopolitical conflict, as well as focusing on the foundational processes and principles of the Intelligence Cycle.