Introduction

Welcome to the online website of the Security Policy Forecasting Tournament. On this site you find background information which should help you to get a better grasp of the tournament. Eventually, this page will contain the forecasts, the scoreboard, the research results and the methodology. If anything remains unclear, don’t hesitate to write an email.

Purpose and aim

Forecasting tournaments are a useful tool to better understand and improve forecasting. They have been extensively used in many future-oriented fields like weather predictions and financial markets. However, only recently they have been introduced to forecasting in international politics. (e.g. The Good Judgement Project). The Security Policy Forecasting Tournament is continuing this effort. Its particular goals are to increase awareness of crowd-based forecasting in Europe and to gain further understanding in the relevance of successful forecasting for policy makers.

Forecasting questions and answers

The forecasting tournament took place between February 6th and 12th. During this time the participants were asked about the probability of 24 geopolitical events happening before April 24th, 2017. The results can be found below. The graphs illustrate how the different estimates of the 223 participants were distributed. The dashed lines indicates the average estimate.

Answer 1: No

There has been some technical adjustment of the sanctions against Russia during the time, but they are seen as a technical fix and not as a lifting or loosening (See e.g. here or here).

Answer 2: No

A list of all Nuclear Weapon tests is available on Wikipedia. In the time frame considered here, there was no test.

Answer 3: No

During the time of the forecasting tournament the Vietnam government did not file a case against Bejing. Some general discussions on the issue can be found here and [here]](http://www.iris-france.org/wp-content/uploads/2017/02/Asia-Focus-16-South-China-Sea-Elie-Khoury-feb-2017.pdf).

Answer 4: No

There are no reports of a meeting between the U.S. President and Taiwan representatives for the time period 12.02.2017 to 24.04.2017. For some general discussions of the US-Taiwan relations since the begin of Trump’s presidency see here.

Answer 5: No

On February 13th there were fightings in Kashmir between militants and Indian security forces, but they where not operating out of Pakistan. (See here). The Council of Foreign Relations is tracking the conflict did not report any attack as described in the forecasting question.

Answer 6: No

There have been some minor changes to the NATO deployment, but the general commitment was not changed. The Trump administration apparently decided to stick to the commitments made by the Obama administration.

Answer 7: No (disputable)

The U.S. government stayed commited to the JCPOA agreement with Iran and did not reintroduce any sanctions lifted by the agreeement. This can be seen on the site of U.S. treasury. However, the new U.S. administration announced new sanctions after a missile test on the 3rd of February 2017 (shortly before the forecasting tournament started). They are not regarded as applicable for the question because (1) the questions is explicitly related to the Nuclear deal sanctions, and (2) the event happened before the forecasting tournament.

Answer 8: Yes

On April 24th 2017 the Islamic State was still under full control of Ar-Raqqa according to most recent public available conflict maps from different sources (e.g. Syria Live Map, BBC or Southfront).

Answer 9: No

As of April 24th, 2017 IS still holds control of several districts in Mosul (for recent reports look here).

Answer 10: Yes (Preliminary)

In April 2015 the Afghan government controlled 72%, in August 2016 63,4% and as of November 2016 the Afghan government controlled 57% of the county according to SIGAR. Since, it has been losing several regions to the Taliban (Sangin and Shorabak) but gained some from IS.(see here). Since SIGAR has not reported any new numbers in 2017 yet the available information is used and which indicates that the Afghan government has lost more territory during the considered time period.

Answer 11: No

As of April 24, 2017 there have been no reports of deadly attacks by former FARC units. However, paramilitary and criminal groups have been conducting attacks and social leaders have been killed. (See here or here)

Answer 12: No (Preliminary)

According to data from the International Crisis Group 89 individuals died between February 10 and April 23, 2017.

Answer 13: No

International media reported of several hundert protesters, but not thousands (here, here)

Answer 14: Yes

On March 22, 2017, a SUV attacked pedestrians on Westerminster Bridge. The Islamic State claimed responsibility for this attack.

Answer 15: No

Boko Haram was responsible for several deadly attacks, in particular in Nigeria. However, all events had less than 20 people killed (see List of terror incidences).

Answer 16: Yes

At least two bombing strikes killed more than 30 individuals. On March 15th, 2017, a market in Yemen was hit (see here) causing the dead of at least 119 people and on March 16th 42 Somali refugees were supposely killed by a Saudi helicooper strike (see here).

Answer 17: No

On 7th of April 2017 the U.S. government used missiles to strike Syria, but it did not annouce a no-flight zone at any time.

Answer 18: No

The parties never signed the agreement and it sign of ultimate failure was when the opposition decided no to attend further negotiations initiated by the Russians (see here.

Answer 19: No

No document has been signed yet.

Answer 20: No

The U.S. increased its deployment in Syria by 400 troops, but the troop limit of about 5000-5200 troops remained unchanged for Iraq. However, the U.S. used temporal deployments during the time (see here) as of end of March stopped disclosing troop deployments in the Iraq and Syria (see here). As the question asked for an annoucement, the question can be answered with no.

Answer 21: No

On March 15th, 2017, the OHCHR published its report stating that there were 130 civilian casualties in the respective time period.

Answer 22: Yes

Several events happened: On March 12th, 2017, Damascus was hit by a twin bombings causing the death of atleast 74 individuals as reported by international media. On April 15th 2017, in Aleppo a car bomb killed at least 126 people.

Answer 23: No

The Islamic State has declared solidarity with the Muslim Rohingyas in Myanmar, but there have been not jihadism motivated attacks as of April 24, 2017. For some background see here.

Answer 24: Yes

On 23. of February a suicide bombing happened [see here]. On March 20th, two would-be suicide bombers died. (see here). On April 8th a suicide attack in Kolofata happened. See here. As there is no good information source for this, we can expect that also more attacks happened which have not been reported.

Score Board

The score board will be published after the 24th of April 2017, when it is clear which events materialized and which did not.

Methodology

Brier Score

The Brier score was originally proposed to quantify the accuracy of weather forecasts, but can be used to describe the accuracy of any probabilistic forecast. Roughly, the Brier score indicates how far away from the truth your forecast was.

The Brier score is the squared error of a probabilistic forecast. To calculate it, we divide your forecast by 100 so that your probabilities range between 0 (0%) and 1 (100%). Then, we code reality as either 0 (if the event did not happen) or 1 (if the event did happen). For each answer option, we take the difference between your forecast and the correct answer, square the differences, and add them all together. For a yes/no question where you forecasted 70% and the event happened, your score would be (1 – 0.7)2 + (0 – 0.3)2 = 0.18. The best (lowest) possible Brier score is 0, and the worst (highest) possible Brier score is 2.

For further information see Wikipedia or The Good Judgement Open.

Score Board

The score board is computed by averaging the Brier scores of each participants and ranking them from the lowest to the highes Brier score. As all participants are required to give judgements to all question, no gambling by not answering questions is possible.

Terminology

  1. Question: “Sanctions against Russia” refer to executive orders (13660, 13661, 13662, 13685) of the U.S. president implemented after the 2014 annexation of Crimea by Russia and the hostilities in Eastern Ukraine. “lift or loosen” refers to changes or terminations of these orders which effectly reduce the scope of the sanctions.

  2. Question: “nuclear weapons test” refers to an event which is mostlikely due to a intented explosion using radioactive explosives. The resulting quake is usually recorded by non-North Korean geological services (e.g. Federal Institute for Geosciences and Natural Resources, Germany).

  3. Question: “International court or arbitration” is any judicial procedure which is not under national jurisdiction (e.g. International Tribunal for the Law of the Sea (ITLOS)) and usually based on an international treaty (e.g. United Nations Convention on the Law of the Sea (UNCLOS)).

  4. Question: “High-level official representative” refers to the prime-minister, president and minister. For the event to occur an actual face-to-face meeting has to take place in the given time frame. However, it is not required to be a bilateral meeting.

  5. Question: “Operating out of Pakistan” refers to the operational headquarters of the attack. The involvement of individuals from Pakistan has to be reported by international media or outside governments. “Attack” refers to the use of physical violence which has the potential to kill people.

  6. Question: “announce” referes to a statement in an official document (e.g. press release) or public speech. The NATO deployments plans in Eastern Europe are part of the NATO operation Atlantic Resolve.

  7. Question: In the Iran Nuclear Deal (Officially Joint Comprehensive Plan of Action (JCPOA)) the U.S. government agreed to lift sanctions against Iran. “Reintroducing” refers to making these sanctions legal binding again or implementing new sanctions aiming at Iran’s Nuclear Program.

  8. Question: “administrative control” referes to the capacity to enforce rules and legislation

  9. Question: “administrative control” referes to the capacity to enforce rules and legislation

  10. Question: The assessment of the Special Inspector General for Afghanistan Reconstruction (SIGAR) will serve as reference.

  11. Question: “FARC splinter group” refers to any group which has or had a relationship with FARC and which split away from FARC in the light of the peace deal.

  12. Question: As reference for the number of deaths serve the statistics published by the International Crisis Group.

  13. Question: All protest before and after the selection of the Hong Kong Chief Executive which takes reference to it is considered. If different counting estimations about the number of participants exist, they are averaged.

  14. Question: “Claims responsibility” refers to any statement from the organization or a high ranking member attributing the attack them. “Attack” refers to the use of physical violence which has the potential to kill people.

  15. Question: Number of deaths according to official statistics or international media reports.

  16. Question: Bombing attack refers to any air-to-ground use of force by the Government of Saudi-Arabia.

  17. Question: “announce” referes to a statement in an official document (e.g. press release) or public speech. “No-flight zone” refers to a distinct area were flight are only allowed with permitance of the U.S. government and any violation entails the risk to be shot done.

  18. Question: “Hold” means that the involved parties publically endorse the cease-fire.

  19. Question: “Peace deal”" refers to an agreement which goes beyond a cease-fire and regulates the long-term relation between the involved party and the Syrian government. “Some armed group” refers to any group which was actively fighting in the Syrian Civil War.

  20. Question: “deployed U.S. troops” refers to the U.S. non-combar mission with currently 5200 military members advising and assisting local forces.

  21. Question: “Casualty” refers to a person killed or injured in a war or accident (Source: Oxford Dictionary). The OHCHR Ukraine report is published here.

  22. Question: “Single event” refers to everything which is connect through the same point of time or the source of death.

  23. Question: “Claims responsibility” refers to any statement from the organization or a high ranking member attributing the attack them. “Attack” refers to the use of physical violence which has the potential to kill people.

  24. Question: “Suicide attacks” include all attacks where the attacker purposely killed him/herself, attempted to do so or acted as if she/he had the intention to die.

Disclaimer

The Security Policy Forecasting Tournament is part of an independent research project at Hertie School of Governance Berlin. The research project is conducted as part of a master thesis and is not funded by a third-party.