On 7 August 2019, the Incorporated Council of Law Reporting’s newly formed research lab, ICLR&D, launched the prototype version of its legal natural language processing system, Blackstone.
For a more technical run down of what Blackstone is and how to use it, go over to the project’s GitHub repository here. The purpose of this article is to provide a more general account of how Blackstone came into being and the thinking driving its development.
What is Blackstone?
Zoom out
Before diving into Blackstone, it’s worth first zooming out and going back to the what led to ICLR&D’s formation in the first place.
ICLR&D is the research division within the Incorporated Council of Law Reporting for England and Wales (“ICLR”). ICLR was itself founded in 1865 and is the creature of some genuinely innovative and disruptive Victorian thinking.
Prior to ICLR’s establishment, the dissemination of case law in England and Wales was a chaotic and haphazard affair. Law reports and word-of-mouth were the only mechanisms available through which the existence and parameters of new precedents could be transmitted throughout the legal community. Word-of-mouth had obvious deficiencies and the law reports that were available were expensive, incomplete, slow to arrive and often inaccurate.
For a common law jurisdiction founded on the doctrine of precedent, the barriers to accurately disseminating the content of the law as decided by judges were becoming a real problem. To deal with what was a crisis waiting to happen for the administration of justice, senior members of the legal profession joined forces to devise a way forward so that the courts and practitioners had affordable and timely access to accurate reports of cases that changed the law.
The specification for the way forward was to establish a single body that would be responsible for monitoring the courts and producing reports of cases that needed reporting. There would be a clear and rigorous set of benchmarks that would be used to determine whether a case ought or ought not to be reported. There would be a single, well-structured style governing the presentation of the reports. And the authors of the reports would be skilled drafters who had been called to the English bar.
That specification led to the founding of the Incorporated Council of Law Reporting in 1865. 154 years later, here we are.
Zoom back in
Clearly, things have moved on a bit since the 1860s where law reporting in the UK is concerned, but not as much as you’d probably think. The most notable changes over the last 150 years relate to the quantity of case law made available over online subscription platforms and the fact that search has taken over as the method for case law retrieval (cumulative indexes and the library shelves have given way). But, in all over respects, the content itself has remained much the same as it was during the reign of Queen Victoria.
Uncontrolled environments
Another thing that has not changed over the last century and a half is the fact that the judgments themselves continue to be produced in uncontrolled environments. The evolution of the tooling of judgment drafting has gone from pen and paper, to the typewriter and then on to Microsoft Word.
In contrast, recent years have seen a surge in the development of contract drafting and review technologies designed to cast the business of contract generation and review into a controlled environment.
The virtue of controlling the environment in which legal content is generated lies in the fact that it enables the re-use of that content for other useful purposes. In the case of contracts and other legal documents of a transactional nature (such as lease agreements) the systemisation and standardisation of their construction makes it possible build knowledge bases and analytical tools that can be used by law firms to more effectively identify surfaces of risk, error and inefficiency.
Judgments, purely by virtue of what they are and what they contain, are rich with latent features that can be mined for all sorts of socially beneficial applications, such as:
The development of better search tools to enable the general public to locate and understand case law
Deeper analysis of the development of laws with respect to broader social trends and concerns
Aiding drafters of legislation, particularly when common law concepts are being transposed into primary legislation
Aiding law reporters to identify important, law-changing cases
It is unrealistic to expect that the business of judgments-drafting will migrate from Microsoft Word (i.e. an uncontrolled environment) to a controlled environment in the foreseeable future and it goes without saying that in the meantime we want the judges to continue focussing their efforts on ensuring that their judgments are correct in law, fully reasoned and given as quickly as possible.
Back to the original questions
It’s easier, against that background, to articulate why ICLR&D came into being: ICLR, as a charitable organisation charged with providing systematic coverage of case law for a century and a half, recognised that technological innovation in the primary law space was significantly lagging behind the pace of technological innovation and change taking place in the commercial settings of large law firms. ICLR&D was therefore formed to provide ICLR with the resources it would need to begin carrying out more advanced research in the field of legal informatics to help us catch up.
So, what is Blackstone?
Blackstone is an experimental project to investigate the ways in natural language processing can be used to impose control and structure on legal content generated in uncontrolled environments. The project’s deliverable is an open source piece of software, the Blackstone library, that allows researchers and engineers to automatically extract information from long, unstructured legal texts (such as judgments, skeleton arguments, scholarly articles, Law Commission reports, pleadings etc.)
Deep dive into Blackstone and the problem space
Blackstone is founded on the assumption that long and unstructured legal texts contain elements and characteristics that can be harnessed in a systematic way to improve our understanding of the meaning of the text and how it might fit into a larger corpus of text. Consider the following extract taken from the UK Supreme Court’s decision in R v Horncastle [2009] UKSC 14; [2010] 2 AC 373:
6 The appellants submit that an affirmative answer must be given to this principal issue. In each case it is submitted that the trial judge should have refused to admit the statement on the ground that it was a decisive element in the case against the appellants. This the judge could have done, either by “reading down” the relevant provisions of the 2003 Act so as to preclude the admission of hearsay evidence in such circumstances or by excluding it under section 78 of the Police and Criminal Evidence Act 1984 (“PACE”).
7 In so submitting the appellants rely on a line of Strasbourg cases, culminating in the decision of the Fourth Section of the European Court of Human Rights (“the Chamber”), delivered on 20 January 2009, in the cases of Al-Khawaja and Tahery v United Kingdom (2009) 49 EHRR 1. In each of those applications statements had been admitted in evidence at a criminal trial of a witness who was not called to give evidence. The Strasbourg court held that, in each case, the statement was “the sole or, at least, the decisive basis” for the applicant’s conviction. The court reviewed its own jurisprudence and concluded that this established that the rights of each applicant under articles 6(1) and 6(3)(d) had not been respected. The court took as its starting point the following statement in Lucà v Italy (2001) 36 EHRR 807, para 40:
“where a conviction is based solely or to a decisive degree on depositions that have been made by a person whom the accused has had no opportunity to examine or to have examined, whether during the investigation or at the trial, the rights of the defence are restricted to an extent that is incompatible with the guarantees provided by article 6.”
I shall call the test of fairness that this statement appears to require “the sole or decisive rule”.
In this short extract, we can identify a range of elements and characteristics that may be potentially useful to capture. For example,
The first sentence of paragraph 6 contains references to a submission and an issue in the case.
The second sentence of paragraph 6 contains another submission and identifies the ground for the submission
The last sentence of paragraph 6 contains two references to legislation, a reference to a provision and an abbreviated form of a statute name.
In the first sentence of paragraph 7, the verb “rely” appears with reference to a line of case law, a court is referred to and a decision of that court (including the name and citation for that decision) is cited
In the third sentence there is a summary of what a court decided
In the fourth sentence reference is made to provisions in the European Convention on Human Rights.
Reference is made to another case and that case’s name and citation is given
There is a block quote of a specific paragraph in a related case.
The very last sentence in paragraph 7 refers to a legal test
Blackstone’s objective is to develop a free and open source system for the extraction of this sort of information from unstructured legal texts. To frame the problem a different way: we know that there is lots of useful information buried in legal texts, so how can be go about extracting it automatically?
How Blackstone is designed to solve the extraction problem
There are two available strategies that can be used to solve this extraction problem.
Rules
The first strategy involves devising a set of explicit rules ahead of time that define the sorts of information we want to extract. There are several benefits to a rules-based approach. First, it eliminates the “blackbox” problem, because we are setting, and are in a position to inspect, the rules governing the extraction process. Second, computationally rules are relatively inexpensive.
The problem is that rules only work well when you can be sure that you can legislate for every possible permutation of the extraction targets. Rules work incredibly well where the raw data is predictable. Long, unstructured legal writing is not predictable. For example, take case citations. We could set out to identify all of the various shapes and sizes a citation may take and build a canonical list of rules that deal with those shapes and sizes. But what do we do if the author of the text being processed introduces a citation we haven’t seen before? What if the author has made a mistake that our rules cannot mitigate because we could not foresee the mistake ahead of time?
Predictions
The second strategy, involves developing a system capable of predicting whether or not a feature in the text is of interest and then categorising that element. The objective is to train a statistical model to form a sufficiently accurate yet generalised view of the things we are interested in extracting. As with the rules-based approach outlined above, there are trade-offs here, too.
The overwhelming benefit of the prediction-based approach to the extraction problem is that an adequately trained model removes the need to legislate for every possible permutation of the information we are seeking to extract. There are disadvantages, however. The first is that we lose the ability to peer inside and see precisely why the model does what it does — the “blackbox” problem. The second problem is that we are never likely to train the model to achieve 100% accuracy — we need to make peace with the fact that our model will fail to extract something it ought to have extracted and our model will mislabel things. From time to time it might do things that a human will regard as completely stupid!
Compromise
In developing Blackstone, we have opted to pursue a blended strategy: the extraction task is fundamentally prediction-driven, with rules buttressing the model at various points along the way.
How Blackstone was built
The development of Blackstone hinges on a single Python library called spaCy, which we selected because it’s extremely well documented, robust, feature-rich and fast. What follows is a general overview of the process we followed the build Blackstone.
