The checks every data import passes

Every night we download each source, check it and load it. A source can arrive empty, partly written, in a changed format or not at all, so each import goes through the same checks before it replaces the data the APIs serve.

Raw copy

The file or response is stored exactly as received before any processing, so we can always see what the source sent.

Structure

Every expected field must be present and of the right type. If a source renames a column, the import stops.

Volume

The number of records must be within 30% of the previous import, and no more than 5% of values in key fields may be missing. Changes larger than that are usually a partial download or a faulty export, so the import is held until we have checked it.

Known records

After loading, we look up records whose correct values we already know, such as CVE-2021-44228, which is in CISA's KEV catalogue. If any of them return the wrong answer, the import is treated as failed.

When a check fails

The previous day's data stays live and responses show its date. We receive an alert. Each run sends either a summary or an alert, so a run that reports nothing is also treated as a failure.