Methods.txt

Collation of Corpus - Action Plan documents
Technician Commitment action plans and other documents produced by Universities are held on their websites (Table S1), in compliance with the TC. In addition, they are held on the Technician Commitment website (ITSS, 2025b). Google Drive was used to collate the corpus. All available documents were downloaded and renamed in a consistent manner (by city and institution, and indicating whether it was an action plan document, RAG analysis, both or other type of document). Where there was more than one document of a type from an institution, these were also named by date of authorship to identify the most recent. Only the most recent Action Plan documents were taken forward into the analysis (listed in Table S1), and were copied to a separate folder. Python was used to script and carry out many of the functions of data extraction and cleaning, action categorisation and performance evaluation, as described below.
Extraction of action items from each document (document_extract2.py)
Authentication and Data Access
Initial access to the Google Drive repository was established using the OAuth 2.0 protocol. The script used the google-auth-oauthlib and google-api-python-client libraries to handle the authentication flow. 
File Retrieval and Content Extraction
The script looped through all files within a specified parent folder in Google Drive. For each identified file, the MIME type was inspected to determine the appropriate text extraction procedure. For native Google Documents (application/vnd.google-apps.document), the content was extracted by exporting the file as plain text (text/plain) using the Google Drive API's export_media method. For Portable Document Format (PDF) files (application/pdf), the files were downloaded in-memory using the get_media method. The pdfplumber library was then used to parse the binary PDF data and extract the text content from each page. Files of other formats were converted to Google Doc or PDF prior to analysis.
Text Pre-processing
The raw text extracted from the documents underwent a reflowing process to correct for artificial line breaks introduced during optical character recognition (OCR) or text wrapping. 
Action Item Extraction via Large Language Model
The pre-processed text from each document was submitted to a generative large language model for analysis (gemini-2.5-pro) via the Vertex AI platform. A specific prompt was engineered to extract all specific action items, tasks, or responsibilities, presenting them as a bulleted list. The model was configured with a temperature of 0.0 to ensure deterministic and consistent output. The extracted action items were parsed from the model's response and written to a comma-separated values (CSV) file with a link to the originating file.
Categorisation of Action Items (predefined_categories6.py)
Following the extraction of action items, a second script was used to classify each item into predefined categories. This involved natural language preprocessing, the application of a weighted keyword lexicon, and a dynamic scoring algorithm to assign the most relevant categories.
Development of the keyword lexicon (keyword_categories_4.json)
Categories and Keywords were defined through iterative extraction and refinement of keywords using a human-in-the-loop implementation of Google gemini-2.5-pro and development versions of the predefined_categories6.py script. Briefly, starting with a manually curated list of 9 categories and 84 keywords likely to feature in the reports (Table S2), the script was run and evaluated, and the keyword list iterated based on the outcomes. 
The resulting lexicon, shown in Table S3 was structured as a dictionary, mapping high-level categories to a set of specific keyword phrases, with each phrase assigned a numerical weight to signify its importance or relevance to the category.
Natural Language Preprocessing (predefined_categories6.py)
The input for this stage was the CSV file generated by the preceding extraction script, containing a list of discrete action items, and the keyword lexicon, which was loaded from a JSON file. To standardise both the action items and the keyword lexicon for comparison, a multi-step text preprocessing pipeline was employed using the Natural Language Toolkit (NLTK) library. For any given text string, the procedure was as follows:
Tokenization: The text was converted to lowercase and segmented into individual tokens.
Filtering: Non-alphanumeric tokens were removed. A standard English stopword list was applied to remove common, low-information words, with a set of custom exceptions to retain domain-specific context.
Part-of-Speech (POS) Tagging: Each remaining token was assigned a POS tag (e.g., noun, verb, adjective) using NLTK's averaged_perceptron_tagger.
Lemmatization: Tokens were reduced to their dictionary base form (lemma) using the WordNetLemmatizer. The POS tag from the previous step was used to track word characteristics for the lemmatization process, ensuring, for example, that verbs and nouns were treated according to their grammatical function.
Weighted Scoring and Categorisation Algorithm
An algorithm was developed to score and categorise each action item. For a given action, its set of unique lemmatized tokens was generated. The algorithm then iterated through each category in the keyword lexicon comparing with the token set. A multi-word keyword phrase was considered a match if all of its constituent tokens formed a subset of the action item's token set. A category's score was calculated by summing the weights of all its keyword phrases that were found within the action item. 
Output Generation
The final categorised data was written to a new CSV file and the assigned categories were appended using a ‘one-hot’ encoding scheme. This resulted in a binary column for each possible category, where a value of '1' indicated the assignment of that category to the action item in that row. An additional 'Uncategorised' column was added to flag items that did not meet the scoring threshold for any category.

Performance Evaluation (evaluate_multi_col.py)
To quantitatively assess the performance of the action item categorisation algorithm, the algorithm's output was compared against a manually curated ground truth dataset, and sensitivity and specificity were calculated for each category.
Ground-truth classification
The ground truth file was prepared by manually assigning a single, correct category to each action item to 10% of the total actions (n=341). Categories containing less than 10 actions were then supplemented with additional actions by manually searching the whole actions list for strong keywords and categorising. 
Data Ingestion and Alignment (evaluate_multi_col.py)
The evaluation process used two data sources: the one-hot encoded output file from the categorisation script, serving as the predictions; and a separate CSV file containing the ground truth classifications. The text in both the prediction and ground truth files was normalised by converting to lowercase and standardising whitespace, and the two datasets were then merged into a single dataframe.
Performance Metric Calculation
For each category, a confusion matrix was generated using the scikit-learn library; true labels were derived from the ground truth column, while predicted labels were taken from the corresponding one-hot encoded column in the merged dataset. From this matrix, the counts of True Positives (TP), False Positives (FP), True Negatives (TN), and False Negatives (FN) were determined.
Evaluation Metrics
Two primary metrics were calculated to evaluate the model's performance for each category:
Sensitivity (Recall): The proportion of actual positives that were correctly identified, calculated as: Sensitivity=TP/(TP+FN)​
Specificity: The proportion of actual negatives that were correctly identified, calculated as: Specificity=TN/(TN+FP)
Reporting
The results of the analysis were compiled into a final summary csv file providing the sensitivity, specificity, and the raw confusion matrix counts (TP, FP, TN, FN) for each category.
Development of a sub-category (“sub-themes”) classification lexicon
Prior to rules-based sub-classification of action items, the most salient keywords within each high-level category were identified. This data-driven approach was designed to provide an empirical basis for the manual definition of sub-categories.
Keyword Salience Calculation using TF-IDF (subclus_calc_weight.py)
The data was split into category-specific corpi. For each, a Term Frequency-Inverse Document Frequency (TF-IDF) analysis was conducted using the TfidfVectorizer from the scikit-learn library. 
This technique was chosen to identify terms that were not only frequent within a category but also distinctive. The vectorizer was configured with sublinear term frequency scaling (sublinear_tf=True) to reduce the impact of very high-frequency words and used a standard English stopword list to filter out common, non-descriptive terms.
The TF-IDF scores for each term were summed across all documents in the corpus, producing a single, cumulative importance score for every term within the context of its category.
Output for Rules-Based Sub-categorisation
The terms were ranked based on their cumulative TF-IDF scores. For each high-level category, the top 15 keywords with the highest scores were identified. These ranked lists were compiled into a text file. 
A hierarchical lexicon was automatically curated by Google Gemini based on the keywords identified in the TF-IDF analysis. This lexicon was structured as a Python dictionary where each key represented a high-level category. The value for each key was a further dictionary containing a set of predefined sub-cluster names, which in turn mapped to a list of specific keywords defining the theme of that sub-cluster.
 	Sub-Category Classification (rule_based_class.py)
A script was written to carry out this function (rule_based_class.py). For each action in the dataset, its high-level category was first identified to select the appropriate set of sub-cluster rules from the lexicon. The algorithm then applied a keyword frequency scoring method. For each potential sub-cluster, the script counted the occurrences of its associated keywords within the text of the action item. 
An action item was assigned to the sub-cluster that achieved the highest keyword match count. In cases where no keywords from any sub-cluster were found, the item was labeled as 'Unassigned'. 
The final output of this process was a new CSV file, with a new column containing the assigned sub-cluster name for each action item. 
Analysis of RAG Assessments (rag_extract_simple12.py, rag_alignment_workflow_unpaired8.py)
Document Acquisition and Multi-Format Extraction
Action Plans and RAG assessment documents were retrieved programmatically from a Google Drive repository using the Google Drive API v3 with OAuth 2.0 authentication. The workflow handled three document formats: native PDF files (extracted using pdfplumber), Microsoft Word documents (.docx files, extracted using python-docx), and Google Docs (exported to DOCX format before processing). Document metadata (institution name, document type, assessment period number) was extracted from filenames using regular expression patterns supporting multiple naming conventions.
Structured Table Extraction Algorithm
For documents containing structured action tables with explicit RAG status columns, we implemented an intelligent column detection system (rag_extract_simple12.py, lines 223-517). The algorithm:
Scanned table headers (rows 0 and 1) to identify columns containing "action", "step", or "what is to be done" for action text, and columns labeled "RAG" for status indicators.
When multiple potential action columns were detected, selected the optimal column by prioritising exact "action" header matches, then choosing the column with the most text content in the first data row.
Tracked column positions across continuation tables (tables lacking headers but continuing data from a previous table).
Merged multi-row actions where text was split across consecutive table rows, using heuristics based on capitalisation patterns and sentence completion markers.
Extracted RAG status using explicit markers (single letters R, A, G, or full words Red, Amber, Green), converting to standardised single-letter format.
Unstructured Text Extraction via Large Language Model
For documents lacking structured tables, action extraction was performed using Google's Gemini 2.0 Flash model (gemini-2.0-flash-001) via the Vertex AI API (rag_extract_simple12.py, lines 520-574). The model was instructed to:
Identify discrete actions or commitments as complete, coherent sentences.
Remove RAG status indicators from the action text itself (to prevent status labels contaminating semantic matching).
Preserve sentence completeness (avoiding mid-sentence splits).
Clean formatting artifacts including excess whitespace and line breaks within sentences.
The Gemini prompt included the first 50,000 characters of each document and returned actions as newline-separated text, which was post-processed to remove remaining RAG indicators using regular expressions.
Action Categorisation
All extracted actions (both from tables and unstructured text) were categorised using the same keyword taxonomy and NLTK-based lemmatisation approach described in section ii(c). The categorisation step populated the "Category" column for every extracted action, enabling category-level trajectory analysis.
Output Format
Extraction produced a CSV file (rag_actions_extraction.csv) with columns: LineID (sequential identifier), Institution (organisation name), Document Type (AP1, AP2, AP3, RAG_AP1, RAG_AP2, RAG_AP3), Action (full action text), RAG value (R, A, or G for RAG documents; empty for Action Plans), and Category (populated by categorisation algorithm).
Three-Tier Action Matching Strategy
Following extraction, actions were matched across documents and time periods using a three-tier strategy (rag_alignment_workflow_unpaired8.py):
Tier 1: Within-Period TF-IDF Matching (same assessment period)
For action pairs from the same period (e.g., AP1 vs RAG_AP1), we used TF-IDF vectorization (sklearn.feature_extraction.text.TfidfVectorizer) followed by cosine similarity calculation. Actions with similarity ≥0.95 were classified as "Identical", ≥0.85 as "Match", and ≥0.70 as "Similar". This fast, deterministic method was appropriate for same-period comparisons where actions were expected to be nearly identical. Matching was performed bidirectionally: Action Plan actions were matched to their corresponding RAG assessment actions (AP->RAG_AP), and vice versa (RAG_AP->AP). Results were stored in columns "Match AP vs RAG_AP (LineID, Status)" and "Match RAG_AP vs AP (LineID, Status)".
Tier 2: Cross-Period Batch AI Comparison (subsequent periods)
For tracking action evolution across periods (e.g., RAG_AP1 -> AP2), we employed Gemini AI semantic analysis with batch optimisation (rag_alignment_workflow_unpaired8.py, lines 289-383). This approach:
Compared one RAG-assessed action against up to 10 subsequent Action Plan actions in a single API call (reducing API costs ~10-fold compared to individual comparisons).
Included category metadata in the comparison prompt to provide semantic context (e.g., "[Categories: Training; Development]").
Requested the model classify relationships as:
"Identical": Essentially unchanged action
"Extended": RAG action evolved into more ambitious version
"Narrowed": RAG action became more focused/specific
"Related": Connected but different focus
"Unrelated": No meaningful connection
Returned matches as JSON with relationship type and confidence score (0.0-1.0), filtering to include only matches with score ≥0.5.
Selected the top 5 matches by score for each source action.
Rate Limiting and API Quota Management
To avoid exceeding Vertex AI free-tier quotas, we implemented conservative rate limiting (15 requests per minute) using a sliding window approach (rag_alignment_workflow_unpaired8.py, lines 177-196). Request timestamps were tracked in a deque structure, with automatic sleeping when approaching the per-minute limit. Additionally, a 4-second pause was enforced between batch comparisons.
Checkpoint Architecture for Resumability
Given the computational intensity of processing hundreds of actions across multiple institutions and periods, we implemented a checkpoint system allowing workflow resumption after interruptions (rag_alignment_workflow_unpaired8.py, lines 94-175). After completing each major task (categorisation, within-period matching) and each institution-period transition in cross-period matching, the workflow saved:
Current dataframe state to CSV (checkpoints/checkpoint_name.csv)
Workflow metadata to JSON (checkpoints/workflow_state.json), including last completed institution, period transition, and progress metrics
On restart, the workflow detected existing checkpoints and offered resumption from the last saved state, preventing redundant API calls and computation.
Trajectory Classification
Following the three-tier matching process, each action was assigned a trajectory code:
C (Continued): Action persists with same focus (from "Identical" or "Extended" relationships)
R (Related): Action has related successor with different focus (from "Related" or "Narrowed" relationships)
S (Stopped): No successor action identified
N (New): Action with no predecessor
U (Unknown): Insufficient data to classify
Derived Metrics for Organisational Analysis
Building on the RAG (Red-Amber-Green) status assessment and trajectory analysis, we developed four composite metrics that integrate completion status with action evolution patterns to provide a nuanced understanding of institutional progress.
Composite Code Generation: For each action in the RAG analysis periods (RAG_AP1 and RAG_AP2), we created composite codes combining RAG status (Green (G, completed), Amber (A, ongoing), or Red (R, delayed/failed)) with Trajectory type (Continued (C), Related (R), or Stopped (S)). These were derived by matching each RAG-assessed action to its corresponding action in the subsequent action plan using the Gemini AI fuzzy matching described above. For example, a "Green" action that was "Extended" in the next period receives the composite code "GC" (Green-Continued), while an "Amber" action that was "Narrowed" receives "AR" (Amber-Related).
Trajectory Classification: Gemini relationship types were classified into three trajectory categories:
Continued (C): Actions classified as "Identical", "Extended", or "Continued", Indicating the action was sustained with similar or expanded scope
Related (R): Actions classified as "Related", "Narrowed", "Partial", or "Replaced", Indicating the action evolved, was refined, or shifted focus
Stopped (S): Actions classified as "Stopped" or "Unrelated" Indicating the action was discontinued or replaced with unrelated work
Composite Metrics: Four complementary metrics were calculated for each action category, aggregating composite codes across both transition periods (RAG_AP1→AP2 and RAG_AP2→AP3):
Successful, Completed (%) - Percentage of RAG-assessed actions that achieved completion (Green status), regardless of whether they continued or stopped. Formula: (All Green codes / Total actions with RAG status) × 100
Interpretation: Measures overall success in completing committed actions. High values indicate strong delivery on commitments. This metric captures both actions completed and sustained (GC, GR) and those completed but discontinued (GS).
Successful, Continued (%) - Among successfully completed actions (Green), the percentage that were sustained or evolved (GC + GR) rather than stopped (GS). Formula: ((GC + GR) / (GC + GR + GS)) × 100
Interpretation: Measures whether successfully completed actions are maintained in subsequent planning cycles. High values suggest completed work is embedded into ongoing practice rather than treated as one-off achievements. Low values may indicate completion of temporary initiatives or shifting strategic priorities.
Adaptability Score (0.0-1.0) - Among challenged actions (Amber or Red status), the proportion that were adapted or refined (AR + RR) rather than continued unchanged or stopped. Formula: ((AR + RR) / (All Amber + All Red)) ÷ 100
Interpretation: Measures the institution's capacity to learn from difficulties and adjust struggling actions. High scores indicate responsive, iterative improvement. Low scores may suggest either abandonment of difficult work or persistence with unchanged approaches despite challenges.
Resilience Score (0.0-1.0) - Among challenged actions (Amber or Red status), the proportion that were sustained unchanged (AC + RC) despite difficulties. Formula: ((AC + RC) / (All Amber + All Red)) ÷ 100
Interpretation: Measures institutional commitment to maintaining difficult actions. High scores indicate persistence and determination. When considered alongside Adaptability Score, provides insight into institutional response to challenges: high resilience with low adaptability suggests rigid persistence, while low resilience with high adaptability suggests responsive refinement.
Metric Interpretation: These metrics are designed to be interpreted together rather than in isolation:
High Success + High Successful Continuity: Evidence-based approach with sustainable embedding of completed work (e.g., strong institutional capacity)
High Success + Low Successful Continuity: Completion-focused approach with frequent strategic pivots (e.g., responsive to changing priorities)
Low Success + High Adaptability: Learning-oriented approach actively refining struggling actions (e.g., developmental phase)
Low Success + High Resilience: Committed approach persisting despite challenges (e.g., addressing systemic barriers)
Low Success + Low Adaptability + Low Resilience: Indicates discontinued work, potentially reflecting resource constraints or changing institutional priorities
The combination of these metrics across action categories enables identification of institutional "learning profiles" - characteristic patterns in how organisations approach, sustain, and evolve their equity, diversity, and inclusion commitments over time.
Fallback Strategy and Robustness
When Gemini AI was unavailable (due to quota limits, network issues, or API errors), the workflow automatically fell back to TF-IDF similarity for cross-period matching, preserving functionality at the cost of reduced relationship-type specificity. All API responses from Gemini were cached to a debug JSON file (rag_actions_extraction_MATCHED_gemini_debug.json) for quality assurance and reproducibility. In practice this was not used in this analysis. 
Final Output
The workflow produced an enriched CSV file (rag_actions_extraction_MATCHED.csv) containing:
All original extraction columns (LineID, Institution, Document Type, Action, RAG value, Category)
Within-period match columns showing bidirectional AP<->RAG_AP correspondences
Cross-period match columns showing RAG_AP(n) -> AP(n+1) trajectories with relationship classifications and scores
Trajectory codes and derived metrics enabling analysis of institutional commitment evolution over time
This comprehensive approach enabled quantitative analysis of how institutional commitments evolved, which action types were sustained or discontinued, and which categories showed dynamic growth versus stability.
Visualisation
To enable intuitive interpretation of category-level action evolution across three assessment periods, we developed a graphical summary output using matplotlib (create_graphical_summary_multi_Institution.py) and an output table of the key computed metrics (create_comprehensive_summary_multi_Institution.py). In the category flow diagrams, these metrics are displayed as horizontal bars to the right of the period columns.
Identification of resources available to the technical staff community
The aim of this part of the analysis was to identify the additional support “Technical staff” can expect to be able to call on outside of their University or employer’s commitments, or between posts. The ten top-scoring sub-categories identified by the analysis in (f) above were used as exemplars to examine the provision of support and resources in each area outside of the signatory Universities. 
To enable a wide search and identify any applicable websites or resources, information on the status of each action area was searched for manually using Google Search. The following search criteria were applied, with the ‘Verbatim’ setting switch to ‘On’, searching from the United Kingdom region:
[subcategory title] AND (UK OR united kingdom) AND (technical staff OR technician OR technologist)
The Google Chrome plugin ‘SERP Sonar’ was used to extract the search results, page titles and summaries to a csv file. The first 50 results for each were examined and relevant information was tabulated. Resources were added to a table and supplemented with other known resources. This allowed identification of support for provision compared to the ambitions of the Action Plan documents, and the identification of gaps in this support.
Back-checking of presence of identified resources in Actions
A script was written to carry out this function (enumerate_resources.py). A lexicon of resource-related keywords was formed to represent the specific entities included in the tables following identification in (g) above. This new dictionary was incorporated into a JSON and applied again to the original extracted list of actions using a python script (enumerate_resources.py), with the aim of enumerating instances where these key resources had been mentioned in the action plans. Mentions were screened for instances of self-referral and remaining mentions put into a table.This was designed to provide a measure of the concordance between planned actions and existing resources, and examine the extent to which existing infrastructure or resources were being cited in action planning. 
Comparison of TALENT report recommendations with actions and resources
The TALENT commission report includes general and sector specific recommendations for implementation of its findings. The 16 recommendations were extracted from the text and manually aligned with the 26 categories resulting from the Action Plan documents. This identified the level of discussion of information pertinent to each recommendation in the Action Plans.