Automating Geometry Validity Checks in GDAL

Invalid rings, self-intersections, and orphaned vertices do not throw a row-count error — they pass ingestion silently and only surface hours later when a spatial join returns wrong results, a tile renders with holes, or a compliance threshold is breached. This page is the focused implementation guide for one narrow scenario: wiring deterministic geometry validity checks into a GDAL/OGR ingestion step so structural defects fail fast at the boundary and never inflate the freshness budget downstream. It sits under Tracking Spatial Data Freshness SLAs — because a stalled validation pass eats the same clock a freshness SLA is measured against — and within the broader Spatial Data Freshness & Quality Metrics program. The audience is the data engineers, GIS platform administrators, and SREs who own pipeline mean time to resolution (MTTR).

GDAL geometry validity gate at the ingestion boundary An input dataset in shapefile or GeoPackage form passes through a baseline dry-run that aligns the CRS, then into the GDAL validation gate that runs IsValid on every feature. A decision asks whether the geometry is valid. Valid geometry flows straight to the validated output. Invalid geometry is routed sideways into a quarantine layer tagged with feature ID and timestamp, which then emits metrics to Prometheus or OTel, which in turn drives P1 or P2 alert routing. Input dataset shp · gpkg Baseline dry-run CRS align GDAL validation gate IsValid() per feature Geometry valid? yes Validated output no Quarantine layer FID + timestamp Emit metrics Prometheus · OTel Alert routing P1 · P2

Problem Framing

Geometry validity is coordinate-reference-system (CRS) dependent, and that is exactly why an automated check is hard to get right. A polygon that is valid in a projected CRS measured in metres can flag as invalid in a geographic CRS measured in degrees, because the snapping tolerance that decides whether two near-coincident vertices form a self-intersection scales with projection units. Run the same IsValid test against a feed whose .prj is missing or whose EPSG code was assigned but never transformed, and the invalid count inflates with false positives that have nothing to do with the source data.

The signals that this scenario is biting you are specific. The invalid-feature ratio spikes immediately after an upstream format change (a shapefile vendor switching to GeoPackage, a CRS re-tag, a new tiling step). Spatial joins downstream begin returning fewer rows than the feature count implies, because GEOS silently drops the malformed side of the predicate. Rendering shows slivers or self-overlapping fills. The defect almost always originates at the ingestion / format-conversion stage, before the geometry crosses into a trusted store — which is precisely where the gate belongs, ahead of Coordinate Reference System Validation and the deeper Geometry Validity & Topology Checks that assume a structurally sound geometry. Catch it here and a bad batch is a one-partition skip; catch it after publication and it is a full reprocess.

Before enforcing a gate, capture a baseline so the alert threshold reflects reality rather than a guess. Define tolerances per CRS family and run a dry-run inventory:

# Geographic CRS (degrees) — ~11m at the equator
export GDAL_VALIDATION_TOLERANCE_DEG=0.0001
# Projected CRS (metres)
export GDAL_VALIDATION_TOLERANCE_M=0.1

# Inventory feature distribution and geometry-type signatures before gating
ogrinfo -al -geom=SUMMARY input.gpkg

Cross-reference the summary against your CRS registry, then assign the source SRS without reprojecting so the metadata is correct before validation runs:

# -a_srs assigns the SRS metadata WITHOUT transforming coordinates
ogr2ogr -f GPKG validated_baseline.gpkg input.shp -a_srs EPSG:4326

Implementation

GDAL’s CLI has no VALIDATE_GEOMETRY open option, so the gate must call the OGR Python API directly. The function below opens a layer, tests every feature with IsValid() (which uses GEOS under the hood), and routes failures into a quarantine layer tagged with the original feature ID (FID) and a UTC timestamp — the minimum needed to attribute a defect back to a source row and a moment in the freshness timeline.

#!/usr/bin/env python3
import sys
from osgeo import ogr, gdal
from datetime import datetime, timezone

def validate_layer(layer_path: str, output_path: str) -> int:
    gdal.UseExceptions()                       # turn silent GDAL errors into exceptions
    driver = ogr.GetDriverByName("GPKG")
    src_ds = driver.Open(layer_path, 0)        # 0 = read-only
    if not src_ds:
        raise RuntimeError(f"Failed to open {layer_path}")

    layer = src_ds.GetLayer()
    valid_count = 0
    invalid_count = 0
    invalid_fids = []

    for feature in layer:
        geom = feature.GetGeometryRef()
        if geom is not None:
            if geom.IsValid():                 # OGRGeometry.IsValid() delegates to GEOS
                valid_count += 1
            else:
                invalid_count += 1
                invalid_fids.append(feature.GetFID())
        feature = None                         # release the reference promptly

    # Only materialise a quarantine layer when there is something to quarantine
    if invalid_count > 0:
        dst_ds = driver.CreateDataSource(output_path)
        src_layer = src_ds.GetLayer()
        dst_layer = dst_ds.CreateLayer(
            "invalid_quarantine",
            srs=src_layer.GetSpatialRef(),     # preserve the source SRS, do not default
            geom_type=ogr.wkbUnknown           # mixed defects may span geometry types
        )
        dst_layer.CreateField(ogr.FieldDefn("original_fid", ogr.OFTInteger64))
        dst_layer.CreateField(ogr.FieldDefn("validation_ts", ogr.OFTString))

        for fid in invalid_fids:
            src_feat = src_layer.GetFeature(fid)
            if src_feat is None:
                continue
            dst_feat = ogr.Feature(dst_layer.GetLayerDefn())
            dst_feat.SetGeometry(src_feat.GetGeometryRef().Clone())  # keep the broken geom for triage
            dst_feat.SetField("original_fid", fid)
            dst_feat.SetField("validation_ts", datetime.now(timezone.utc).isoformat())
            dst_layer.CreateFeature(dst_feat)
            dst_feat = None
        dst_ds = None                          # flush + close the quarantine dataset

    src_ds = None
    # Structured line a log scraper can lift into gis.spatial.* metrics
    print(f"VALIDATION_RESULT: valid={valid_count} invalid={invalid_count}")
    return invalid_count

if __name__ == "__main__":
    exit_code = validate_layer(sys.argv[1], sys.argv[2])
    sys.exit(1 if exit_code > 0 else 0)        # non-zero exit lets the orchestrator skip the batch

Wrap the call in a systemd timer or an Airflow task with a hard wall-clock limit. Use ogr2ogr for the bulk conversion, promoting to multi-geometry so a single-vs-multi mismatch does not masquerade as a topology fault:

# -skipfailures continues past unreadable features; drop it to halt on the first error
timeout 45s ogr2ogr \
  -f GPKG validated_output.gpkg input.shp \
  -nlt PROMOTE_TO_MULTI \
  -skipfailures \
  -lco GEOMETRY_NAME=geom

If a pass exceeds roughly 45 seconds per 10,000 features, trip a circuit breaker that pauses the pipeline and emits a P2 so a malformed dataset cannot exhaust compute reserved for healthy workloads. Surface the run as a metric — gis_spatial_invalid_features_total over gis_spatial_features_processed_total — so the alert is a ratio, not a raw count:

groups:
  - name: spatial_validation_alerts
    rules:
      - alert: HighInvalidGeometryRatio
        expr: >
          rate(gis_spatial_invalid_features_total[5m])
          / rate(gis_spatial_features_processed_total[5m]) > 0.05
        for: 2m
        labels: { severity: warning, team: data-engineering }
        annotations:
          summary: "Geometry validity ratio exceeds 5% threshold"
      - alert: ValidationTimeoutCircuitBreaker
        expr: gis_spatial_validation_duration_seconds > 45
        for: 0m
        labels: { severity: critical, team: sre }
        annotations:
          summary: "GDAL validation circuit breaker tripped — pipeline paused"

Verification & Testing

Prove the gate works by injecting a known defect rather than waiting for one. A bowtie self-intersection is the canonical test: a four-corner ring whose diagonal vertices are swapped is structurally invalid but parses cleanly, so it exercises IsValid() without tripping a parser error first.

# Create a deliberately self-intersecting polygon and confirm the gate flags it
ogr2ogr -f GPKG fixture.gpkg -dialect sqlite -sql \
  "SELECT GeomFromText('POLYGON((0 0, 1 1, 1 0, 0 1, 0 0))', 4326) AS geom"

python3 validate_geometry.py fixture.gpkg quarantine.gpkg
# Expect: VALIDATION_RESULT: valid=0 invalid=1   (exit code 1)

For row-level confirmation and a human-readable diagnosis, query the quarantine layer with ST_IsValidReason, which names the defect class and the offending coordinate:

SELECT original_fid, validation_ts, ST_IsValidReason(geom) AS reason
FROM invalid_quarantine
LIMIT 50;
-- e.g. "Self-intersection[0.5 0.5]" pinpoints the exact failure

A green test run is three things together: the fixture produces exactly one invalid feature, the process exits non-zero so the orchestrator skips the batch, and the quarantine table carries one row whose validation_ts lands inside the current run window. If any of those is missing, the gate is not wired in correctly.

Gotchas & Failure Modes

An SRID mismatch passes the validity check while corrupting everything downstream. IsValid() only inspects structure within whatever coordinate space the geometry claims; it cannot tell that the geometry was tagged 4326 but actually holds projected metres. The check returns valid, the layer publishes, and features land in the wrong hemisphere. Pair every validity run with an explicit SRID assertion from Coordinate Reference System Validation — the two faults are orthogonal and neither check covers the other.

ogr2ogr cannot validate during conversion. There is no VALIDATE_GEOMETRY open option, and -skipfailures only skips features that fail to read, not features that are topologically invalid — a self-intersecting polygon converts happily and lands in the output. Validation must run through the OGR Python API as above, or via ST_IsValid after a PostGIS load. Treating -skipfailures as a validity filter is the most common silent gap.

ST_MakeValid can change geometry type or drop area. Repairing a self-intersecting polygon may return a MULTIPOLYGON, a GEOMETRYCOLLECTION, or — for a degenerate sliver — an empty geometry. A naive in-place update can therefore violate a single-type column constraint or quietly delete features. Guard the repair, and re-check coverage afterward so a fix does not introduce a gap:

UPDATE invalid_quarantine
SET geom = ST_MakeValid(geom)
WHERE NOT ST_IsValid(geom)
  AND NOT ST_IsEmpty(ST_MakeValid(geom));   -- never replace a feature with emptiness

Re-ingest repaired features and trigger a Spatial Coverage & Extent Monitoring sweep to confirm no extent was lost in the repair.

FAQ

Why does IsValid() flag geometries that render fine in QGIS?

Rendering tolerates defects that topology operations will not. A self-intersection or a duplicated vertex can draw correctly yet break ST_Intersection, ST_Union, or a spatial join. IsValid() enforces the OGC Simple Features rules a renderer ignores, which is exactly why it is the right gate ahead of analytics.

Should an invalid feature fail the whole batch or be quarantined?

Quarantine it and let the run continue. Routing the bad FIDs into a separate layer and exiting non-zero lets the orchestrator skip just that partition, instead of failing the task, burning retries on the same data, and paging on-call. Reserve a hard failure for unrecoverable faults like a mixed-SRID batch.

How do I tune the 5% invalid-ratio alert?

Start from the baseline invalid ratio of known-clean batches captured during the dry-run, and set the alert a few standard deviations above it. Vendor feeds with chronic minor self-intersections may justify a higher gate paired with an automatic ST_MakeValid step; authoritative cadastral feeds should sit near zero and alert on any breach.

Does the GEOS version affect validity results?

Yes. IsValid() and ST_MakeValid behaviour can differ across GEOS releases, and ST_MakeValid gained a linework-vs-structure algorithm choice in newer builds. Pin the GDAL/GEOS version in your pipeline image so a base-image bump does not silently change which features pass.

Can I validate without writing a quarantine file?

Yes — drop the export block and keep only the counters for a fast pass/fail gate. But without the quarantined FIDs and timestamps you lose the triage trail, so for any feed feeding a freshness SLA, keep the quarantine layer; it is what lets you attribute a defect to a source row and a moment in time.