As a data-driven tool, the Index aims to standardize the complex concepts of criminality and resilience across 193 countries. While standardization promotes comparative analysis and easy interpretation, it does not come without challenges. The collection of data may vary across countries in terms of availability, reliability, uniformity and compatibility. These issues are even more pronounced given the inherently clandestine nature of organized crime. While the Index endeavours to overcome such challenges by expert cross-checks and triangulating sources, another challenge arises when there is abundant information on a particular country or component. Research and information promote greater understanding of the organized crime situation in a given country and its resilience, informing better policymaking and responses. Nevertheless, for comparative tools such as the Index, an information bias – where more information is published on some areas rather than others – may risk skewing results and interpretation of findings. In other words, an organized crime problem may appear to be more acute in countries where more information, research and reporting have occurred. One of the main goals of the Index is to highlight areas where information is lacking in an effort to promote further research. It is important for stakeholders to accept indices for what they are and the information they provide as supplementary to other means of gathering information. Similarly, instances arise where published information and evidence are lacking but experts with in-depth knowledge of a specific context acknowledge that problems exist. In such cases, the Index as an expert-led assessment heavily relies on expert knowledge in evaluating country contexts.

Diversity in the Index components

Countries that differ in their criminality and resilience may nevertheless be assigned the same scores, while those that host a particularly acute organized crime problem may appear to score lower than other countries. These results can be explained by the structure of the Index. Because overall criminality and resilience scores are assigned based on a simple average of their respective composite indicators, countries that have a diverse range of criminal markets and criminal actors will score higher than those that have fewer, albeit more pervasive, criminal characteristics. The same can be said to describe resilience indicators. Overall resilience scores will be dependent on a country’s ability to tackle the organized crime situation based on a holistic and multifaceted approach, rather than a criminal justice- or economically-driven approach.

Limitations

Recognizing the difficulties in creating a tool that studies a subject matter that is highly varied and inherently clandestine, the Index comes with limitations and potential biases. Nevertheless, it can be described as a worthwhile exercise, although with certain limitations, that we hope will become a catalyst for further debate.

On a methodological note, there are considerations that must be taken into account when interpreting the scores. First and foremost, the Index relies heavily on individual expert knowledge and experience, which introduces the possibility for an implicit bias, where experts’ personal convictions might affect their judgement. The ramifications in this case are diametrically opposed. On the one hand, experts might have been too critical, given their knowledge of a specific country’s deficiencies, and on the other, they might have been tempted to be too lenient. Although the latter was presumed more probable prior to the evaluation rounds, it was observed that experts tended to be more critical and often held countries, especially developed ones, to a higher standard. Throughout the development of the tool, we have attempted to control that bias by providing preliminary country profiles as a basis on which experts were able to make their assessments and by specifying scoring thresholds to guide the scoring process, as outlined in the methodology section. In addition, all countries underwent a number of anonymous verification rounds, comparing the scores across indicators and regions in an attempt to account for implicit bias.

Experts in the initial scoring round provided scores that were presented to experts in the following geographic and thematic scoring rounds. This opened the possibility for a confirmation bias, where experts would confirm the scores assigned in previous rounds. To address this, an additional score verification round was carried out, where groups of representatives from numerous areas of expertise came together in moderated regional discussions to debate and scrutinize the scores and justifications for each country.

In addition, as one aspect of the Index tool is to help policymakers improve their approach to organized crime, it is fundamental to understand where harms for different markets are coming from. It is undeniable that many of the harms associated with specific markets stem from existing policies. One example of that is the cannabis trade. Policies related to the policing and use of cannabis differ from country to country, and even within countries. Thus, evaluating the impact associated with that market has been ambiguous. While an increasing number of states are moving to decriminalize or legalize cannabis, there is some room for illegality, like trafficking cannabis to countries with stricter policies, for instance. Thus, to be as consistent as possible, the importance of capturing that aspect of illegality when evaluating the market was emphasized to expert scorers. Nevertheless, consistency in that case has been difficult to achieve.

Another critical issue during the scoring process was the debate on the harm and impact of markets, namely whether harms are comparable across markets. Here lies another limitation of the tool – the weighing of different components of the Index. Currently, as has been already specified, indicators are weighted evenly. Nevertheless, four of the 15 markets are drug-related, which puts implicit weight on the impact of drugs, which, depending on the context, may pose issues. Environmental crime markets in Europe, for example, are almost non-existent. Yet they are weighted equally with more pervasive markets, such as human trafficking and the synthetic drug trade. That has, on a number of occasions, raised the obvious question among experts on European organized crime whether this approach was justified. However, environmental crime has had a significant impact in Africa and Oceania, for instance, where it has endangered entire ecosystems and even threatened the existence of coastal communities. Arguably, the impact of environmental crime markets has been more severe there than human smuggling, for example, which is perceived as problematic in a European context.

Thus, the current weighting of indicators might lead to some curious results. What often occurs is that two countries that perhaps would not be expected to rank on the same level criminality-wise, turn out to be very similar to one another. This is where we advise caution and recommend readers to look at the disaggregated scores, as we have stressed at the beginning of the report, because countries might have similar or the same overall ranking but for different reasons.

Broader scope: additional Index indicators

To address recognized limitations in the scope of the criminal markets covered by the Index, the current edition has been expanded. An additional five criminal markets (financial crimes, cyber-dependent crimes, illicit trade in excisable goods, trade in counterfeit goods, and extortion and protection racketeering), and a criminal actor type, private sector actors, have been added. Incorporating these new indicators provides for a more thorough global assessment of the organized crime landscape.

We recognize the difficulties in measuring organized crime in general, but an accurate assessment of cross-cutting markets, such as financial crimes and cyber-dependent crimes, comes with its own set of challenges. As a result, expert technical groups were set up to assess whether these markets would fit into the existing Index methodology. Expectedly, given the sheer volume of criminal activities that fall under these two crime types, financial crimes and cybercrime were the most difficult to define and measure. Therefore, the definitions of these two criminal markets include a set of specific criminal activities. Narrowing down the types of offences that would fall under the respective markets allows us still to adequately measure a specific market while avoiding double-counting (i.e. counting the same criminal activity under two different markets). Needless to say, double-counting would artificially increase the average criminality score of a country, skewing the overall results of the Index.

To illustrate, take the example of cybercrime. Our approach has been to distinguish between cyber-enabled and cyber-dependent crime. What are often described as cyber-enabled offences (i.e. activities carried out online to enable traditional ‘offline’ forms of crime) have already been captured under the first iteration of the Index. So using the internet to target and recruit victims of human trafficking would be captured under the human trafficking market, for instance. In that sense, it is necessary for cyber-enabled and cyber-dependent crimes to be separated to avoid such significant overlaps between markets. To avoid issues stemming from such overlaps, a decision was made for the Index to measure cyber-dependent crime as a standalone market.

As a cross-cutting phenomenon, defining and measuring financial crime also came with some challenges. Again, to avoid overlap when assessing financial crimes and to allow comparability with the previous iteration of the Index, any activities that can be attributed to another criminal market under the Index fall under that respective market indicator. Procurement fraud for logging contracts is one such example, which is considered under the flora crimes market, as opposed to financial crimes. Notably, the laundering of illicit proceeds and bribery are not included in the financial crimes category either. Excluding money laundering from this category is justified by the fact that it does not exist in itself but rather occurs as a secondary crime linked to illicit proceeds of a predicate offence. As the predicate crimes are already assessed under the various markets (e.g. drug trafficking), the secondary offence – in this instance laundering of the proceeds of drug trafficking – would not fall under the financial crimes market. It is instead classified under the primary market: drug trafficking. The one exception is when money laundering occurs as a result of fraud or another offence that is classified as a financial crime.