Rwanda Comments on UN Genocide Mapping Report: III. The Methodology is flawed
III. THE Draft Mapping Report’s Methodology is Demonstrably Flawed
33. The methodology employed in The Draft Mapping Report, a creation of the authors themselves, is deeply flawed and one-sided. The following analysis into the scope of the mapping exercise, the quality of its sources, the standard of proof and the lack of transparency presents serious questions about the credibility and reliability of its conclusions. Some of the misrepresentations can hardly be unintentional, such as the constant assertion that the Terms of Reference for the mapping exercise require that they make legal qualifications – which they do not.
Scope: Seven Months to Cover Ten Years
34. The Mapping Team had just under seven months, from 17 October 2008 to 15 May 2009 to (1) create a chronology of events through the analysis of 1500 documents which covered alleged atrocities committed throughout the DRC (a country ninety times the size of Rwanda)17, and (2) once this timeline was created, five teams of three people – fifteen people in total -verified as many incidents as possible throughout the DRC. Each incident recorded in the timeline had to be verified by only “one independent source in addition to the primary source”.18 As will be discussed below, the “primary source” – the 1500 documents used to create the timeline – was collected from NGOs, the Congolese government and the media – none of whose credibility or bias has been assessed.19
35. Given the constraints of such a compressed amount of time combined with a broad geographical (the DRC) and temporal (ten years) mandate, it is not surprising that the authors of The Draft Mapping Report wrote the following:
“The choice of priority areas for investigation and the main incidents for verification was also influenced by the short timeframe – six months – allocated to the implementation of the Mapping Exercise itself. Investigations that would take too long to achieve the anticipated findings that would feature in the final report were not included.”20
Given that the authors conclude this report by making an initial hypothesis of genocide, the GoR believes that no investigation should have been spared to get the facts correct before making such a serious allegation. According to the authors of The Draft Mapping Report, investigations were abandoned or simply not included if they could not be resolved in six months, what seems to be an impossible timeframe given the mandate.
Crimes Committed Many Years Prior: The Unreliability of Witness Statements
36. The scope of this investigation is complicated further by the fact that these rapid fire investigations were conducted between five to fifteen years after the alleged commission of the atrocities. The GoR, in prosecuting crimes from the 1994 genocide, is acutely aware of the difficulties in investigating crimes committed over a decade prior. Identifying and procuring reliable witnesses is complicated by death (especially in war ravaged areas), relocation, and the dilution of accurate memory.
The Draft Mapping Report itself acknowledges the unreliable nature of their sources
“It is very hard to determine how many people died in the first few months of the conflict. Every community has its own version of the facts and its own estimate of the number of victims. Furthermore, killing sprees often occurred at very heavily dispersed sites that are hard to access even now. Where it is possible to visit these sites, it is rare to find first hand witnesses to the events, because the successive wars that ravaged the province often entailed the displacement of the people in the villages that came under attack.”21
37. The authors note two huge, separate problems that speak to the absolute necessity of verifying allegations by more than only one independent source. First, “every community has its own version of the facts”. Given this admission, only one possible version of events can be the “truth”. More likely, none of the versions of the event is the absolute truth, but instead contains elements of truth. Thus, many “version[s] of the facts” are simply not true, and knowing that, the authors of The Draft Mapping Report never should have allowed incidents to be included that were validated by only one independent source.
38. Second, “it is rare to find first hand witnesses to the events…” The authors of The Draft Mapping Report admit that the one outside source that was consulted very likely was not a first hand witness to the events. Do the witness accounts come from things they have heard from others, years prior? Is all of the “evidence” hearsay? Even initial hypotheses of such serious allegations should be confirmed by evidence more credible than that.
THE TERMS OF REFERENCE
39. The Terms of Reference (TOR) that govern the objectives of The Mapping Exercise state three main objectives: (1) to conduct a mapping exercise; (2) to assess the justice system of the DRC; and (3) to suggest appropriate transitional justice mechanisms.22 However, on multiple occasions, the authors assert that the TOR requires them to make legal classifications of crimes saying, “[n]onetheless, as described previously, the Terms of Reference of the Mapping Exercise required it to carry out a general legal classification of the crimes committed, including genocide.”23
40. These classifications, allegedly “required” by the TOR, formulate the entirety of Chapter V of The Draft Mapping Report. However, in reviewing the entire TOR, it at no time requires the mapping team to make such a classification. In fact, the words “legal classification” are nowhere to be found in the TOR. Furthermore, the TOR explicitly recognizes the limits of this mapping exercise, stating “[i]t should gather basic information…and not replace in-depth investigations into the incidents uncovered.”24
41. Given this mandate, legal classifications of crimes were in no way required by the TOR as written. Further, given the language of the TOR, a legal classification is beyond the scope of the objectives of a mapping exercise, in so much as that objective is, as stated, to “gather basic information”. The authors of The Draft Mapping Report went beyond the terms of their mandate, indeed beyond the logical application of the quality of evidence gathered in making such legal classifications, especially as they relate to genocide.
Defining a Mapping Exercise
42. Again, the first of the three defined objectives for The Draft Mapping Report is to “[c]onduct a mapping exercise of the most serious violations of human rights and international humanitarian law…” The methodology behind the mapping exercise is a tailor-made creation of the authors themselves, based in no precedent or legal standard whatsoever. The report first unsuccessfully seeks to shed light on the nature of a mapping exercise, describing it as “a generic expression implying no predefined methodology or format”25. Despite this recognition of the terms utterly amorphous and malleable nature (or perhaps with this recognition), the authors go on to define a mapping exercise as requiring “various activities to be carried out, including the collection, analysis and assessment of information contained in multiple reports and documents from different sources, meetings and witness interviews, as well as consultation with field experts and consultants.”26This is the first indication that the authors did not feel that consultations with the implicated governments or militaries was a required activity to be carried out during a successful mapping exercise.
Delegating the Investigation to politicized NGOs
43. Faced with a large task and a limited timeframe, but also wanting to make their point, the authors of the report delegated the initial investigation to politicized Congolese NGOs who provided many of the 1500 primary source documents which laid the foundation for future investigation.27 Illustrating the circular nature of information gathering, these same NGOs provided access to witnesses who later served as independent confirmation of their own primary source documents.28
44. The investigation relied disproportionally on the information provided by Congolese NGOs whose bias and credibility is never assessed by The Draft Mapping Report. Anyone familiar with the dynamics of Congolese civil society knows that NGOs did not remain politically neutral during the conflict. Many of these NGOs took vocal political sides, some even actively diffused messages of intolerance against Congolese Tutsi in North and South Kivu, and Rwandans. Not only does The Draft Mapping Report fail to disclose the identity of the NGOs associated with the investigation, but considering the particular socio-political situation in the region, the authors put no distance whatsoever between their investigations and these strongly politicized organizations.
Assessing the Purported Evidence
45. The Draft Mapping Report states that “[a]ssessing the reliability of the information obtained was a two-stage process involving evaluation of the reliability and credibility of the source, and then the validity and truth of the information itself.”29The credibility of each source was determined using the “admiralty scale”.30 Once again, the authors fail to explain the “admiralty scale” or cite to any source whatsoever. Apparently, in employing this scale, the authors assess the “nature, objectivity and professional standing of the source”, the “methodology used”, and the “quality of prior information obtained from that source”.31 Despite this explanation, it is difficult to assess how credibility was measured without further explanations of these terms or reference to any precedent or rules of evidence. First, objectivity is crucial and, in a criminal proceeding, the defense would bring to light any bias on the part of the witness. In assessing methodology, this is an admission that work was delegated out to other NGOs, and no where do the authors say how they assessed the methodological standards of over 200 NGOs and 1500 documents. Finally, the reference to prior information given suggests that at least some individuals or sources provided “evidence” in reference to multiple alleged incidents. Thus, if one source were later found to be unreliable, it would taint the account of every alleged incident he/she reported.
46. The validity of the information was “evaluated through comparison with other data on the same incidents to ensure cohesion with other verified elements and circumstances.”32 The authors state that each incident is backed by at least two independent sources. This statement is misleading in two ways. First, collection of the primary source documents and access to witnesses were both facilitated by the same NGOs, thus raising legitimate concerns about the interconnectedness of the primary and confirming source. Second, an incident was reported if it was alleged in documentation obtained from an NGO, Congolese government or the media plus “one independent source in addition to the primary source”.33 Given that the authors met with 1280 witnesses relating to over 600 reported incidents, it either implies that – with very few exceptions – at most two independent sources validated each incident or that the authors relied on the same sources repeatedly in relation to multiple incidents who continually validated one another. Either way, this raises serious questions regarding the validity of information.
The Unrecognizable Standard of Proof
47. The authors assert that a mapping exercise does not require the collection of evidence meeting the criminal law standard of proof “beyond reasonable doubt”. The purpose of the mapping exercise, it seems, is to formulate “initial hypotheses”, and the standard of proof required is that of “reasonable suspicion”. Reasonable suspicion is defined as “a reliable body of material consistent with other verified circumstances tending to show that an incident or event did happen.”34 The citation at the end of this definition is to the authors themselves, and states an alternative definition: “reliable and consistent indications tending to show that the incident did happen.” The authors cite no outside support of any kind for this formulation of the standard of proof. “Reasonable suspicion” has, however, been defined by the U.S. Supreme Court and is the absolute lowest evidentiary standard articulated in U.S. law.35 Put into context, in order of reliability, it is a lesser standard of proof than “probable cause”36, “preponderance of the evidence”37, “clear and convincing evidence”38, and “beyond reasonable doubt”39. It is fitting that the authors chose the term “reasonable suspicion”, as the quality of the evidence merits, at most, cursory attention.
48. These methodological weaknesses highlight the inherent conflict of interest
created where both the standard of proof and the reliability indicators are defined and assessed by the same individuals who conduct the investigations and write the conclusions. Anathema to transparency, this system lacks any sort of checks on the authors and investigators themselves, or on their sources. This becomes especially egregious considering that neither the alleged perpetrators nor their governments were ever consulted for input.
17 The Draft Mapping Report, paragraph 109.