A simple, but opinionated metadata quality checker and fixer designed to work with CSVs in the DSpace ecosystem (though it could theoretically work on any CSV that uses Dublin Core fields as columns). The implementation is essentially a pipeline of checks and fixes that begins with splitting multi-value fields on the standard DSpace "||" separator, trimming leading/trailing whitespace, and then proceeding to more specialized cases like ISSNs, ISBNs, languages, etc.
Requires Python 3.6 or greater (3.8 recommended). CSV and Excel support comes from the Pandas library, though your mileage may vary with Excel because this is much less tested.
The easiest way to install CSV Metadata Quality is with poetry:
$ git clone https://github.com/ilri/csv-metadata-quality.git $ cd csv-metadata-quality $ poetry install $ poetry shell
Otherwise, if you don't have poetry, you can use a vanilla Python virtual environment:
$ git clone https://github.com/ilri/csv-metadata-quality.git $ cd csv-metadata-quality $ python3 -m venv venv $ source venv/bin/activate $ pip install -r requirements.txt
Run CSV Metadata Quality with the
--help flag to see available options:
$ csv-metadata-quality --help
To validate and clean a CSV file you must specify input and output files using the
-o options. For example, using the included test file:
$ csv-metadata-quality -i data/test.csv -o /tmp/test.csv
You can enable several "unsafe" fixes with the
--unsafe-fixes option. Currently this will attempt to fix invalid multi-value separators and remove newlines.
This is considered "unsafe" because it is theoretically possible for a single
| character to be used legitimately in a metadata value, though in my experience it is always a typo. For example, if a user mistakenly writes
Kenya|Tanzania when attempting to indicate two countries, the result will be one metadata value with the literal text
--unsafe-fixes option will correct the invalid multi-value separator so that there are two metadata values, ie
This will also remove unnecessary trailing multi-value separators, for example
This is considered "unsafe" because some systems give special importance to vertical space and render it properly. DSpace does not support rendering newlines in its XMLUI and has, at times, suffered from parsing errors that cause the import process to fail if an input file had newlines. The
--unsafe-fixes option strips Unix line feeds (U+000A).
Unicode is a standard for encoding text. As the standard aims to support most of the world's languages, characters can often be represented in different ways and still be valid Unicode. This leads to interesting problems that can be confusing unless you know what's going on behind the scenes. For example, the characters
é look the same, but are not — technically they refer to different code points in the Unicode standard:
éis the Unicode code point
éis the Unicode code points
Read more about Unicode normalization.
You can enable validation of metadata values in certain fields against the AGROVOC REST API with the
--agrovoc-fields option. For example, in addition to agricultural subjects, many countries and regions are also present AGROVOC. Enable this validation by specifying a comma-separated list of fields:
$ csv-metadata-quality -i data/test.csv -o /tmp/test.csv -u --agrovoc-fields dc.subject,cg.coverage.country ... Invalid AGROVOC (dc.subject): FOREST Invalid AGROVOC (cg.coverage.country): KENYAA
Note: Requests to the AGROVOC REST API are cached using requests_cache to speed up subsequent runs with the same data and to be kind to the system's administrators.
You can enable experimental support for validating whether the value of an item's
dcterms.language field matches the actual language used in its title, abstract, and citation.
$ csv-metadata-quality -i data/test.csv -o /tmp/test.csv -e ... Possibly incorrect language es (detected en): Incorrect ISO 639-1 language Possibly incorrect language spa (detected eng): Incorrect ISO 639-3 language
-m spdxto enable the spdx module?
This work is licensed under the GPLv3.
The license allows you to use and modify the work for personal and commercial purposes, but if you distribute the work you must provide users with a means to access the source code for the version you are distributing. Read more about the GPLv3 at TL;DR Legal.