Read the full post here

The abstract:

The English conquest and colonisation of Ireland, which began in the years around 1170 was accompanied by the introduction of an administrative system based on English models. From the point of view of the crown, perhaps the most important of the new offices of government that it established was the exchequer, which coordinated the financial exploitation of its Irish lordship. The exchequer generated a vast quantity of written documents recording its operations. This paper subjects one such document, detailing the sums received at the exchequer for the year 1301–2, to data science techniques in order to gain added insight into the routine functioning of the financial arm of English government in its oldest colony. It thereby also reveals previously unrecognised patterns in the nature of English power in Ireland. The purpose of the paper is not to assess the state of Irish finances in the early fourteenth century, but rather to argue that a deep reading of a single document produced by an elaborate bureaucratic system, combined with data science visualizations, can help to generate new research questions in relation to a substantial body of financial records which are soon to become more widely available to both scholars and the general public.

Read the full article here

I’ve been lucky enough to have the opportunity to collaborate with the Beyond 2022 project. I’ve been helping historians and archivists on the project develop a TEI/XML schema that will be used to encode a modern English translation of original Latin documents. These documents include receipt rolls from the Irish Exchequer, something I’m familiar with after working with Prof. Brendan Smith at the University of Bristol. Part of the encoding includes marking up dates in the documents in a format that can be read by computers, and this blog post describes how I wrote a plugin for the Oxygen XML editor in collaboration with Dr Elizabeth Biggs, a researcher at Trinity College, Dublin, and The National Archives (UK), to make this process easier and less error-prone. It is a nice example of how a Research Software Engineer can make a positive contribution to a Digital Humanities project.

Read the full post here

In 1301–2 there were eleven shires in Ireland: Connacht, Cork, Dublin, Kerry, Kildare, Limerick, Louth (Uriel), Meath (Trim), Roscommon, Tipperary, and Waterford. Some of them were recent creations, with Kildare and Meath being created in 1297. Each county had one sheriff who, by a decree of 1293, was appointed by the treasurer and barons of the Irish Exchequer. It was an important royal office with numerous financial, administrative and judicial responsibilities, including collecting Crown revenues, delivering writs, repairing gaols and castles, empanelling juries for the county courts and presiding over their tourns (courts). Even though the office was a financial burden for the individual, it remained popular. According to Jocelyn Otway-Ruthven:

‘The attractiveness of the office was perhaps as much a matter of prestige as anything else, for its legitimate rewards do not seem to have been large.’[1]

If you look at the compiled list of sheriffs for Louth given by Brendan Smith in Crisis and Survival in Late Medieval Louth it is clear that reconstructing who held the office in the thirteenth and fourteenth centuries involves a lot of detective work, consulting a wide and diverse set of manuscript and printed primary sources.[2]

I naïvely assumed that it would be obvious from the receipt roll who was the sheriff that year by looking at who was proffering accounts to the Exchequer. However, at best, you can make an educated guess based on patterns, but any results would need to be checked against other sources.

Table I gives a list of forty sheriffs proffering accounts for the eleven shires and the terms in which they appeared. This table was manually created from data derived from a pandas query, because I needed to resolve the inconsistent spelling of the names of the sheriffs. Hopefully, future work will use a fuzzy string-matching algorithm to reduce this overhead. Table II filters out payments that don’t mention arrears.

Since each county only had a single sheriff for a financial year, we can see that sheriffs from previous years continued to answer for arrears on their accounts. We can take Cork as an example. In Table I, William de Cauntetone seems the most likely candidate to be sheriff for that year, since he is the most active name who proffers accounts in all terms, while the others only answer for accounts in one term. Filtering out the arrears (Table II), seems to confirm Cauntetone although three other individual payments that aren’t declared as arrears.

Filtering out arrears has other interesting side-effects. Meath and Roscommon disappear! Why? Was this due to the difficulty on collecting income in these counties? Were payments often made in arrears?

Although we have only pulled out the name of the sheriffs, we can see that analysing these documents at scale will provide interesting insights into who were the sheriffs, and for how long after holding the office they were still encumbered with financial responsibilities. Do patterns of proffering and the amounts proffered reflect the ebbs and flows of English colonial rule in the various shires?

Table I. The sheriffs mentioned in the receipt roll.

The variations in spellings are given in brackets, but no effort has been given to modernise the names. For example, in the table Richard of Oxford is Richard de Oxon’ and Nicholas Devenish is Nicholas de Deveneis.

Table II. The sheriffs where the payment doesn’t mention arrears.

[1] J. Otway-Ruthven, ‘Anglo-Irish Shire Government in the Thirteenth Century’, Irish Historical Studies, Vol. 5, No. 17 (1946), 1–28 (http://www.jstor.org/stable/30007236)

[2] B. Smith, Crisis and Survival in Late Medieval Ireland: The English of Louth and Their Neighbours, 1330-1450 (Oxford, 2013), p. 15 (DOI:10.1093/acprof:oso/9780199594757.001.0001).

Interesting analysis. Was income concentrated in the first week or two of the Michaelmas and Easter terms, from the sheriffs’ adventus, as in the English receipt rolls from the 1250s and 60s?

— Richard Cassidy (@rjcassidy) March 20, 2020

The adventus refers to the Adventus Vicecomitum or the ‘appearance of the sheriffs’, where sheriffs would appear at the English Exchequer twice a year to pay in money they had collected for the king. [1]

In this post, we’ll first break-down the returns on a monthly, then weekly basis, which will help answer Dr Cassidy’s query.

In the following two plots, we can see the monthly distribution of income coming in and how many individual transactions. The peaks and troughs seem to follow a similar pattern between the amount of business before the exchequer and the amount of revenue, i.e. the totals aren’t dominated by a small number of high-value receipts.

This can be seen more clearly in the following plot where the data is normalised by showing the total value of receipts and the total number of transactions as a percentage of the totals for the whole year.

How does the picture look if we break-down the Exchequer business onto a weekly basis? The following two plots show the total value of receipts and the number of transactions for each term. The line plots are a little jarring since the financial terms are not of equal length. It should also be noted that I take a week as business occurring from Monday to Saturday. Thus, not all weeks will have the same number of days, depending on when they start and end. For example, the first week of Michaelmas only represents one day, since the term began on a Saturday. Also, some weeks exchequer didn’t sit due to feast days.

The trends are more evident if we look at each financial term individually. Here the data is normalised by showing the total value of receipts and the total number of transactions as a percentage of the totals for each financial term.

Once again, the weekly break-down generally shows a similar pattern between transactions and values with some exceptions on certain weeks. For example, in the tenth week of Michaelmas, the spike in payments against a lower number of transactions is accounted for by Roger Bagot, sheriff of Limerick, returning £76.6s.8d. for the ‘debts of divers persons’; and £100 being returned by William de Cauntone, sheriff of Cork, in forfeited property of felons and fugitives. In Hilary, the higher spike of values against a low number of transactions occurs due to a small amount of high-value payments from Cork on 5–6 February 1302, including £99.13s.7d. for the farm of the city, and a further £173.6s.8d. in aid promised to the king. The spike in payments received in the third week of Trinity is caused by a substantial sum of £310 proffered by Thorosanus Donati, attorney of the merchants of the society of Frescobaldi. Whereas, the high number of transactions compared to the lower total values in the second week is caused by that week having a high number of receipts (80), of which around 45% were valued at a pound or less.

So, returning to Dr. Cassidy’s question: ‘was income concentrated in the first week or two of the Michaelmas and Easter terms, from the sheriffs’ adventus, as in the English receipt rolls from the 1250s and 60s?’

We can see that the bulk of the income for 1301/2 came in October (Michaelmas) and May (Easter). However, in both terms, most of this was accounted for in the fourth week, and not the first two. Did the sheriffs account for the bulk of these payments? By querying the dataset, we can pull out amounts that refer to sheriffs in the Michaelmas and Easter terms, and plot those values against the total weekly sums.

Warning: in a couple of places, the value returned by the sheriffs will be under-represented slightly. Generally, if a sheriff is entered in two or more consecutive rows, the first row gives the name, e.g. ‘John Wodelok, sheriff’. The following and subsequent entries state ‘the same sheriff’. However, on at least two occasions, the next entry just declares ‘the same’, and these are not recognised as a sheriff in our current query. This should be rectified with future improvements in our entity recognition, and the occurrences are so few for sheriffs to have a minor effect on accuracy.

For Michaelmas, sheriffs accounted for just under half of the values returned in the fourth week, but most of the total on the tenth week. Again, in the Easter term, sheriffs accounted just under half of the total amount returned.

So, the initial analysis would indicate both similar and different practices in Ireland during the reign of Edward I, compared to England under his father, Henry III. The spikes in Michaelmas and Easter suggest that the Crown expected a proffering at the Irish Exchequer, but the timings of the payment differed. The later payments in the Easter financial term might be explained by the timing of the feast of Easter, which was 22 April in 1302. However, a note of caution is needed. This is only a snapshot of a single financial year. Does this pattern reflect other years? Does practice shift over time? Is there a difference in recording, i.e. is the sheriff bringing money to the Irish Exchequer, but another person or institution is recorded in the receipt roll as answering for the rent, debt or fine?

[1] R. Cassidy, ‘Adventus Vicecomitum and the Financial Crisis of Henry III’s Reign, 1250-1272’, The English Historical Review, Vol. 126, No. 520 (June 2011), 614-627 (https://www.jstor.org/stable/41238716.

I’ve created a collection of scripts to process and clean the data. One script goes through each line of the transcript, looking for details of interest, namely payments and a record of the daily sums. It ignores weekly, monthly, termly and other periodic totals.

• For each payment made, it records the details of the payment, membrane number, term, date, day, the geographic location or other entity they are categorised by, date, day, value in £.s.d. or marks, and the computed value in pence.
• A second CSV file tracks the daily sums entered by the exchequer clerk, recording the value in in £.s.d. or marks, and the computed value in pence.

A second script creates a CSV file that compares the daily sum value entered by the clerk, with a computed daily value using the pandas library, highlighting where they diverge. This was a useful exercise for finding errors:

• Lots of initial bugs in my script in parsing values and extracting data. The code now has several unit tests for checking my regex/parsing logic.
• Errors in the transcript. There were a small number which I corrected in the source text file.
• One error by the scribe (1 penny out!)

A third script parsing the details of the payment and attempts to extract people, places and keywords:

• I use the NLTK default Parts of Speech (POS) tagger, to tag the details of a payment, into personal nouns, nouns, prepositions, adjectives etc. I then fix obvious errors, such as marking ‘de’, ‘le’ and ‘fitz’ as foreign words.
• I use the NLTK RegexParser to look for patterns that match people and places. The fact the transcript is English and provides names and places in common patterns, we get reasonable results. Places are generally identified as pronouns that are after certain nouns and prepositions, e.g. ‘city of Cork’
• For keywords I just extract nouns with a few stop words removed, namely ‘divers’, ‘others’ and ‘persons’

Anyway, the results mean that the data in the transcript is now in a tabular form, which means it can be queried using the pandas data library. For example, we can create the following bar plot which shows the total payments received by the Irish Exchequer for each financial term.

The code and data for the project is hosted here: https://github.com/ilrt/ReceiptRollE101

Using Python and pandas, it was possible to easily explore the data and visualise it with plots via the Matpotlib and Seaborn visualisation libraries.

The total value of the receipts for the financial year was £6159.18s.5d. This total is one penny more than that recorded by the exchequer clerk, who missed some fractions of a penny recorded in the 24 May 1302.

Of the payments:

• the smallest payment was 8 pence
• the largest payment was £310
• the median payment was £1.10s.2d.
• the mean payment is £7.1s.1d.
• the mode payment was 6s.8d.

The financial year was broken into four terms: Michaelmas (September–December), Hilary (January–March), Easter (April–June) and Trinity (June–August). Using these terms, we can break down the value of receipts by these terms.

The terms themselves were not of an equal length of time. Also, within each term, there are various days the exchequer did not sit, or no payments were received.

Within each term, the amount of business, i.e. the number of receipts received in the exchequer, also varies.

We can plot all three variables – total revenue, length of the term and amount of business – in a radar plot with each variable represented as a percentage of the total value for the year.

What is immediately striking in these plots is that the Hilary term is relatively long but has the least business and income, whereas the Easter term is quite short but provides the most income.

Clearly this is only a window on a specific year, and it would be interesting to see trends over time. How does the 1301/2 financial year compare to others with Edward I’s reign? What trends can be seen over the years, decades and centuries? How was the income from Ireland affected by war, rebellion, famine and plague? Are there trends to be gleamed from the different administrations under varying chancellors? Also, does income reflect the changeable effectiveness of English royal authority in Ireland?

In future posts, I’ll be digging further into the data.

The purpose of the seed corn funding is to undertake some initial exploration of applying ‘Digital Humanities methodologies’ in constructing digital calendars – detailed descriptive summaries of original documents – as well as using data analysis tools and techniques to extract entities (people, places, communities etc.) from these calendars and provide example visualisations to interpret and understand the data.

It is anticipated that TEI/XML would be used to encode a calendar digitally and we’ll then extract data so we can make queries with the pandas data analysis library and create visualisations that can be published via a Jupyter Notebook. The construction of these calendars still involves considerable endeavour from trained historians and archivists, since they will be English-language summaries of original Latin documents. However, we should explore how computer science techniques can improve the construction and quality of the calendars.

The focus of this project is a single Irish Exchequer receipt roll from the latter years of King Edward I reign (1301–2). A receipt roll holds information on the day-to-day financial dealings of the Crown and provides a rich source of material not only on the machinery of government but also the communities and people that, for various reasons, owed money to the king. An English-language calendar exists and was edited by Brendan and Paul and published in the Handbook of Select Calendar of Sources for Medieval Ireland in the National Archives of the United Kingdom (Dublin, Four Courts Press, 2005). This calendar builds upon an incomplete earlier calendar by H. S. Sweetman that was published in the Calendar of Documents Relating to Ireland, 1293–1301 (London, 1881). The original document is in The National Archives (TNA), London, with the call number E 101/233/16.

As a starting point, I’ve been writing some Python code to convert the text of the calendar into a CSV/Excel file. In many ways, the roll is a medieval equivalent to an accounting spreadsheet (apologies if this makes any historians cringe). I’d be interested to see if there is any mileage in doing the original data creation for other calendars in a spreadsheet (or database) rather than creating XML documents as a starting point.

It is hoped that future funding will be obtained to create digital calendars of other receipt rolls (they exist for most years from 1280 to 1440), possibly as part of the Beyond 2022 project.

I’ll write other posts about processing the text and data in the future. The code and data for the project will be hosted here: https://github.com/ilrt/ReceiptRollE101

Since the publication of the article in 2016, Jenny has created a dataset of metadata for c. 10,000 published poems in 22 newspapers or magazines. The data includes publication details (publication, publication format, date), details about the author (if known) and first line, second line, penultimate and last line of the poem. This is an impressive endeavour, but much more data collection and investigation are needed. In this regard, Jenny was awarded a pump-priming project funded by the British Academy / Leverhulme Small Research Grants scheme to analyse the data collected to date and create visualisations that help to formulate further research questions for future projects that will allow further data collection.

As the Research Software Engineer (RSE) on the project, it has been a privilege to work with Jenny, her dataset and to collaborate with two other colleagues at the University of Bristol. Namely, Natalie Thurlby, a data scientist in the Jean Golding Institute ; and Chris Edsall, a RSE in the Advanced Computing Research Centre.

We opted to use Jupyter Notebooks to provide a working environment where Jenny could run and rerun visualisations. We would use pandas for querying and analysing Jenny’s data, and the Matplotlib and Seaborn libraries for creating plots. Initially, I thought we could use a cloud-based hosted service, such as Microsoft Azure Notebooks. However, this wasn’t practical for several reasons. In the case of Azure the default pandas library version is quite old and the suggested solution of using conda to update it is very slow, especially when an Azure instance will shut down after a relatively short period of inactivity. We therefore opted to use Anaconda – ‘The World’s Most Popular Python/R Data Science Platform’ – to allow Jenny to run Jupyter Notebooks on her PC.

I created several different visualisations that ranged from broad overviews of the data, to specific ones around authorship and gender, with the ability to drill-down into the data. This post includes a few examples of the plots created for the project to give a taster of the work being used to construct additional research questions.

The following graph shows the total number of poems published each year in our dataset (blue) against the number identified as copies (orange) i.e. published more than once.

The following heat map shows the coverage of our dataset, which highlights that we are dealing with an incomplete landscape. Red indicates that we have data for a publication for a certain year, while the cream highlights we have gaps. That said, we do need a third colour since some publications had a very short print run and therefore our data might be complete for them.

The following heat map shows where poems are shared between publications. We can quickly identify that a large number of poems were shared between London Magazine and both Newcastle General Magazine and Scots.

With regards to ‘copies’, we can see that the vast majority were only published twice, leading to some outliers that were published eight or nine times. This pattern might change when we get more data.

With regard to attributed authors (lots were published anonymously) we can see in our dataset that a large number of authors only appear once with a long tail of a small number of authors who were published many times.

In our dataset Colley Cibber (actor, playwright and Poet Laureate) is the most published. Mainly, because as Poet Laureate, he was commissioned to write two poems a year – one for the New Year, and one for the King’s Birthday – and these were widely printed in the newspapers.

The discovery of copies was originally provided by some earlier input from the Jean Golding Institute, who provided a ‘matching’ algorithm. This process took several hours, but by replacing it with the ‘tone-deaf’ named FuzzyWuzzy library we reduced the processing time to around ten minutes. Chris thinks the processing time can be reduced much further if we replace fuzzy string matching with something like cosine similarity. In a future project, I’d also like us to explore creating a workflow with parallelized code that runs on the High Performance Computing facilities that we have at Bristol.

Jenny is working with the visualisations to explore her existing dataset and to formulate more research questions, with the hope of obtaining further funding to expand the dataset to give a more comprehensive picture of this neglected poetry.