Master's and doctoral theses are some of the most detailed research a country produces. In Zambia, these electronic theses and dissertations (ETDs) are scattered across institutional repositories, described in different ways, and often missing basic details such as the supervisor or degree level. Today we describe how we rebuilt the Zambia National ETD Portal, which brings 5,923 postgraduate ETDs from four institutions into one searchable collection, and how we use machine learning to fill gaps in their descriptions.

Research hidden in silos

The four universities whose ETDs we currently harvest each keep them in their own institutional repository, all running DSpace. Each repository serves its own users well, but a student looking for prior work on, say, maternal health in Lusaka has to know which institutions to search and how each one describes its theses. Earlier DataLab studies documented the scale of the problem: Zambian research has low online visibility, repository records rarely comply with the ETD-MS metadata standard, and key details such as supervisor, department and degree level are frequently missing, as Adrian Chisale's MLIS dissertation showed in detail.

The portal therefore has two jobs: gather ETDs from many places into one, and make their descriptions consistent and complete enough to search and browse.

Harvesting into one portal

The portal is a harvester and a search interface on a single modest server, hosted by ZAMREN. A Python harvester collects records every night using OAI-PMH, the standard protocol repositories use to share metadata, falling back to a repository's REST interface or to a CSV upload for institutions without a repository. Records are mapped to ETD-MS, cleaned, and kept in a staging store that preserves the original harvested XML untouched. The public interface is built on Project Blacklight, the open-source discovery framework used by many university libraries, with Apache Solr for search; its record pages are modelled on Columbia University's Academic Commons.

The portal lists all 34 institutions with postgraduate programmes accredited by the Higher Education Authority (HEA), including the 30 whose ETDs are not yet available to harvest, so the gaps are visible. It holds postgraduate work only; records identified as undergraduate are excluded, with the reason logged. The portal also gives data back: its own OAI-PMH provider (in Dublin Core and ETD-MS, ready for aggregators such as NDLTD), a documented REST API, and an admin panel for corrections and data uploads.

Filling the gaps from the theses themselves

Harvested records were sparse: no source supplied a supervisor or department, and only 40% of records had a degree level. The missing details are usually printed in the thesis itself, on the title page ("submitted in partial fulfilment of the requirements for the degree of Master of…"), the certificate of approval, and the acknowledgements. Work by Frazer Nyambe and colleagues, in an article now in revision, found the acknowledgements page the most promising source of supervisor names.

So the portal reads the first pages of each ETD. Some repositories split a thesis into several files, so cover and preliminary pages are read before the main document, and scanned theses, about a quarter of UNZA's collection, go through optical character recognition (OCR). A transparent rules model then extracts degree, level, discipline, supervisors, school and department, keeping the exact text it relied on. We designed the machinery around four principles:

• Confidence before display: a predicted value only fills an empty field when its confidence is at least 0.8, and it is shown with a "derived" label naming the model.
• Repositories first: a value the repository supplies is never overwritten by a model.
• People have the last word: corrections in the admin panel override everything and survive re-harvests; predictions that would exclude a record go to a review list unless the model is very confident.
• Only redo what changed: a prediction cache keyed by each model's input means models rerun only for new or changed records.

We evaluated against a ground truth of 3,267 UNZA theses that eight DataLab students annotated in 2020, holding back half for reporting only. On that held-out half, values the portal displays were correct 99% of the time for degree level, 96% for department, 86% for degree, 84% for discipline and 80% for supervisor. Coverage is lower: about half of these theses had no readable text yet when we measured, and the rules miss many supervisors. Across all sources, supervisor completeness rose from 0% to 37.8%, degree level from 40% to 71.2% and degree from 17% to 62.3%.

One subject vocabulary for all

Subject headings were the hardest problem. Each repository uses its own, our earlier work on controlled vocabularies found them inconsistent, and 16.6% of records had none. We therefore give every record standard fields of study from OpenAlex Topics, an open (CC0) scheme of 4 domains, 26 fields and 252 subfields, while keeping each repository's headings as keywords.

The model turns a thesis's title and abstract into a sentence embedding, a list of 384 numbers that captures meaning, using the compact all-MiniLM-L6-v2 model on the server's processor; no GPU or paid service is involved. A linear classifier trained on 23,384 works that OpenAlex had already labelled predicts the subfield. When the model is unsure, it backs off to the broader field, then to the domain, so every record receives a value.

On held-out OpenAlex works, the model picks the right subfield 53% of the time (75% within its top three), the right field 71% and the right domain 85%. Readers can now browse by Research domain, Field of study and Subfield across all institutions.

Limitations

We see several limits in the current system:

• Thesis style differs from papers: the field-of-study model learned from journal articles, and many education theses receive only the broad field "Social Sciences" or an unlikely subfield. A statistical correction for subfield frequencies made this worse, so we withdrew it.
• Evaluation covers UNZA only: our ground truth comes from one university, and we have not yet annotated fields of study for portal records.
• Coverage is uneven: four of 34 institutions currently contribute records, and some repository links return errors.

What's next

Our next steps are to bring more institutions on board through OAI-PMH or CSV uploads, to label theses from the portal itself so the models learn how Zambian ETDs are written, and to evaluate them on more institutions. Initial talks are under way for the HEA to formally take over the platform, which we hope will encourage more institutions to set up repositories. More details are on the project page.

Acknowledgements

The Zambia National ETD Portal is a project of the DataLab Research Group, led by Lighton Phiri (Principal Investigator and Lead Developer) with Elijah Chileshe (Assistant Programmer/Analyst), who presented the portal's design in an ETD 2024 poster. We thank the students whose work underpins it: Adrian Chisale and Cecilia Kasonde (MA Library and Information Science), whose studies of metadata completeness, quality and ETD-MS compliance shaped the design; Chanda Chilufya, Kamfwa Chileshe and Mainda Chuulu (B.ICT with Education), who built the ETD pre-processing pipeline; Frazer Nyambe, Malaizyo Muzumala, Nkole Mulenga, Richard Mufuzi and Geoffrey Ngoma, authors with Lighton Phiri of the article on supervisor extraction now in revision; and Blessed Makungu, Linda Mwanza, Kevin Wamunyima, Emmanuel Kuyela, Hockings Mambwe, Wezi Munthali, Jackson Mwanaumo and Mathews Lungu, who annotated 3,267 UNZA theses in 2020. We thank ZAMREN for hosting the portal. Much of the current implementation, including code, documentation and models, was developed with the help of an AI assistant (Claude, by Anthropic), working under the direction and review of the project team.