The value of open-source intelligence (OSINT) as a legitimate intelligence discipline has been demonstrated several times over the past two years in a wide range of operational scenarios, including the Russian invasion of Ukraine, the civil war in South Sudan, and most recently in the Middle East crisis. Once perceived by many as the preserve of gap filling for the Intelligence Community (IC), to cover those areas of the world that were too low in priority to receive the attention of discrete, highly classified, and resource-intensive national collection assets, intelligence from publicly and commercially available sources has now demonstrated real utility in the full spectrum of intelligence outputs. From providing foundational data to near-realtime reporting, indicators and warnings, and even predictive intelligence, this transformation has been enabled not just through the explosion of unclassified data but also through investment and the application of innovative technology. Much of this innovation has originated in the commercial sector, encouraged by increased dialogue with governmental organisations. However, despite the tangible progress, there is a long way to go before OSINT truly reaches its full potential. Why is this the case, and what needs to be done next so that OSINT becomes the āintelligence of first resortā?
In March of this year the Office of the Director of National Intelligence (ODNI) published its much-anticipated OSINT strategy, a six-page document outlining what needs to be done to āstrengthen OSINT as a core intelligence discipline and position the IC to capitalize on the full potential of open-source dataā. That the strategy has been pored over in detail by industry, academia, and think tanks speaks volumes for the level of interest in this emerging discipline and illustrates a dearth of practical mechanisms for meaningful stakeholder engagement. The strategy provides some helpful guidelines on how the IC intends to organize and position itself in the OSINT arena in the short term (it only looks out to 2026), but detail is lacking and implementation will be key. The IC functional manager for OSINT is now working with other IC stakeholders to develop an implementation plan. In addition, a Commercially Available Information (CAI) Policy Framework was published in May, which is the first IC-wide policy document governing the ICās access, collection, and processing of CAI.
For the commercial OSINT sector, there remain several unanswered questions, and a fervent hope that industry will be fully considered and engaged as an important stakeholder and partner in the process. But why should this be the case? The IC is a huge enterprise, well used to embracing and incorporating new technology, adapting its output to reflect current and emerging security trends. Industry therefore needs to make a clear and compelling case as to why the IC should not just āgo it aloneā. And it can. The first and arguably the most important reason is the combination of factors that can be conveniently wrapped into the term ācultureā. Like any large government organisation, each IC element has its own institutional culture, honed and reinforced over many yearsā experience, within a highly specialist and regulated environment. Employees are intelligent, motivated, professional, patriotic, and generally hold very high-level security clearances. These strengths do, however, tend to inculcate a common worldview and similar mindsets. And while all of this is entirely normal and even essential for the effective functioning of such organisations, it can stifle genuine creative thinking and impact institutional agility in adapting to new situations. The very existence of commercial OSINT providers depends on organizational and technical agility, and they can recruit a diversified, professional workforce. Many commercial analysts have highly relevant regional and cultural backgrounds and provide invaluable alternative perspectives on global security challenges. This argument is relevant to finished intelligence, but it does not address the more fundamental question of why internal IC elements cannot just conduct their own data collection from unclassified sources to support their analytical processes. OSINT, by its very nature, derives its output from a very broad range of data sources that can be brought together in a genuinely multisource way, constrained only by the data collection and filtering processes applied. Unlike the single-source IC elements whose collection and structures focus exclusively on their own discipline (HUMINT, SIGINT, and GEOINT, among others) and whose expertise is in collection using their own exquisite (and expensive) assets, industry has the latitude to collect, validate. and integrate all their data, no matter the source or format. This has been recognized by ODNI, and a request for information (RFI) was published in August 2024 seeking commercial solutions to improve the ICās acquisition of and access to CAI and publically available information (PAI). Inherent to this, but not explicitly stated, is the need to apply AI and ML techniques to cohere disparate data, thereby making it much easier for analysts to turn source material into finished intelligence. Significant effort and investment have gone into the development of open-source data collection, wrangling, and management by industry, over an extended period, efforts that do not therefore need to be duplicated by the IC. When set against IC competing priorities, partnering with industry represents real value for money.
However, to what extent is the IC open to and ready for analysed and assessed output? So far, most focus has been on the data, specifically the reference material and foundational layers from which finished intelligence is derived. There has been a reluctance by government security organisations to rely on commercially provided unclassified reporting to inform their outputs. To an extent this is understandable. Government users do not necessarily want their potentially sensitive intelligence requirements and gaps to be exposed in the public domain. This is a particular challenge for the IC, but national security and defence priorities are hardly a surprise and are even exposed at the unclassified level through publications such as the Annual Threat Assessment of the US Intelligence Community and National Intelligence Council Global Trends reports. Specific intelligence gaps can be obscured by more generic RFIs, and commercial providers can contribute to the collective knowledge by reporting on areas that the IC is simply not positioned to cover in any detail or has not prioritised. It could further be argued that industry can provide additional value even in areas prioritised by the IC. It can do so by providing an alternative perspective and using trusted human sources who would not want to engage with the IC. An important associated benefit of commercially derived OSINT is to facilitate the sharing of assessments with non-traditional partners without exposing sensitive sources and methods. For those steps to be taken, however, a deep level of trust is essential, underpinned by a formalised process that demonstrates coherence with IC best practice. Neither of these two critical, linked elements are currently evident.
Industry therefore has an obligation to clearly demonstrate the same attention to analytical tradecraft and standards as those practiced within the IC. This includes alignment with official IC guidance such as Intelligence Community Directive 203 and 206, the adoption of similar terminology, and a comprehensive citation of sources. The planned development of a new IC standard for OSINT should go some way to help address these issues. Importantly, though, there should not be one-way traffic, and the IC could learn a great deal from reputable, trusted, and well-established commercial OSINT providers that have developed and honed their tradecraft over many years and who are at the cutting edge of the specialisation. It is an area in which true partnership would be of great benefit to the IC and, while culturally challenging, efforts should be made to bring industry more fully into the process. The production of and adherence to a coherent standard that encompasses OSINT best practice across the board would go a long way to inculcate the trust that would lead to genuine partnership.
Another fundamental requirement is a robust and accredited security and counter-intelligence posture that not only protects commercial OSINT providers from potential compromise, but which incorporates a robust and co-ordinated tasking mechanism, and optimises the ability to ingest, manage, and integrate data and products into the classified realm. The requirement to do this at multiple classification levels in a timely and efficient manner is an enduring challenge and outside the scope of this paper but must be recognised and guidance provided.
The IC could assist in all this by providing unambiguous advice and guidelines on its processes and by much closer dialogue with industry that is more than a transactional, contractual relationship to meet a stated requirement. To an extent, this has been recognised in the strategy paper, which states ārobust partnerships with industry, academia, and foreign counterparts will be essential for success in the OSINT missionā and the āIC must be postured to capitalize on pioneering efforts in the private sector in partnering in new ways with industry and academia to develop, test, and deploy OSINT tools and tradecraftā. What is missing so far is the āhowā and this will likely be within the remit of the OSINT Strategy Action Plan, the implementation of which will be led by the OSINT functional manager, in partnership with the IC OSINT executive and the defence intelligence enterprise manager. To capture genuine best practice and inculcate a genuinely trusted partnership between the IC and industry, this process should be conducted in a frank, open, and collaborative manner both with commercial OSINT providers and proponents in academia.
Now is a pivotal time for the incorporation and ānormalisationā of OSINT as a sub-discipline within the IC. The commercial OSINT sector is forging ahead and will continue to innovate and proliferate regardless of the degree to which the IC engages with it and there is both an inevitability and recognition that OSINT, produced outside of the IC, will be used on an ever-increasing basis. There is a real opportunity for the IC to influence this effort through genuine partnership and collaboration to the benefit of both. Co-opting trusted external OSINT subject matter experts into the implementation process at an early stage would go a long way in ensuring that OSINT fully takes its rightful place as the āintelligence of first resortā.
