Guideline Elements

Clearly list and define each type of clinical information the algorithm is designed to extract, such as entities (medications, diagnoses, procedures, or laboratory values), relations or document classes. Provide examples for each type, especially when extracting complex, multi-part types and/or concepts.

Describe the underlying clinical information model or clinical framework used to guide the structuring or interpretation of extracted data, if applicable. Furthermore, justify the choice of model and its context. Describe any adaptations or assumptions made and limitations of the information model.

Describe the underlying data model used to guide the structuring or interpretation of extracted data, if applicable.

Explain how extracted entities are organized by mapping to terminologies, ontologies, or thesauri.

If applicable, describe how uncertainty, negation, temporality, ambiguity, inconsistency and coreference resolution (e.g., patient vs. family member) is handled.

Describe the primary information extraction task (s) as well as additional steps performed as part of the study, such as document classification, named entity recognition (NER), relation extraction, template filling, or summarization. For complex workflows (e.g., agentic behaviour, multi-step pipelines), describe each step in detail.

Detail the implementation strategy used to develop the algorithm, including any specific frameworks, pipelines, or approaches. When using existing software, include version specifiers and parameters. If applicable, depict the complete machine learning system architecture including its components. For LLMs, provide the model card.

Describe text normalization or cleaning steps performed prior to model input, including the rationale and details of the approach taken. When using existing software, include version specifiers and parameters.

List all hyperparameters used during model training and provide their values and selection process (manual, grid search, etc.). For incontext learning, provide the exact prompts used and describe the prompt development process.

Report hardware specifications (e.g., GPU device name, vRAM, cluster description, GPU hours) of training and inference environments and costs, if applicable and available.

Provide a detailed description of the data processing workflow, to report on the processing of selecting, excluding, filtering and sampling data from the originally available pool of documents and on the number of resulting documents/sentences/words/tokens for each step. The provision of a visualization is complementary.

Show and explicitly describe the applied data splitting strategy (train, test, evaluation and/or validation set) including amount of documents/sentences/words/tokens, splitting modality (timeline, groups, etc.), and method (random, stratified).

Identify the department (s), institution (s) and country (ies) from which the clinical texts originated. If applicable, describe which (clinical) information system data was extracted from.

Describe the clinical text source (author type, stage in patient treatment where it is created (e.g., at discharge, during stay) and their purpose (e.g., progress monitoring) and content (e.g., daily insights from nurses).

Describe the timeframe and (binned) time-related distribution of the dataset.

Describe the population the data used in the study might be representative for, including any limitations. If applicable, report the class distribution on all levels (source, sample, split) and any methods used to address class imbalance.

If applicable, describe the examination and medical specialty the document corresponds to.

Specify the language (s) of the clinical texts used for algorithm development and evaluation. Describe any manipulation of the original data (e.g. translation).

State whether the study received ethics committee approval and provide the approval reference number if applicable.

Describe whether the textual annotation method is manual, rule-based, machine learning-based, or hybrid (i.e., if automated annotation was combined with manual annotation), including its sequence. Describe the applied tagging format, if applicable.

Detail the annotation process, including annotator background, roles, training, as well as number of annotators and test rounds.

Describe the annotation guidelines used, including how they were developed, tested, and refined. Explicitly state whether the guidelines are publicly available.

Mention any software used to support the guideline development and annotation process.

Describe how annotation consistency was measured and report the agreement metrics used, as well as how disagreements were handled.

Provide detailed performance metrics for each information type as well as aggregated over all types, such as precision, recall, and F1- score. Justify each metric used. Explicitly state the calculation formula for each metric. For continuous variables, show the risk for outliers (e.g., scatterplots). Describe the aggregation method used for calculating averaged performance measures. If available, compare with human-level and baseline performance. Describe (manual) error analysis results. For classification tasks, provide confusion matrices.

Describe any statistical validation procedures applied.

If applicable, report results of validating the information extraction algorithm on an independent dataset from another institution than the training / test data and detail the dataset’s characteristics. If external validation is not possible, it should be acknowledged as limitation.

Indicate whether and how the datasets used in the study are publicly available, accessible upon request, or not available (including a rationale).

Explicitly state whether the source code is made available. If yes, provide it via open repositories (e.g., Zenodo). If applicable, provide links to trained models (either via upload to a platform or direct download).