gcp-emulator-provisioner

Pub/Sub and Cloud Storage state for local GCP emulators — and the same config runs against real Google Cloud.

  • v2.0.1 · latest
  • MIT
  • Go
  • same config runs against real Google Cloud
docker pull ghcr.io/datarocks-ag/gcp-emulator-provisioner:2.0.1

What it does

  • Topics, subscriptions, buckets — Idempotent provisioning of Pub/Sub topics and subscriptions and Cloud Storage buckets.
  • Drift reconciliation — Sends a minimal update mask: only the fields that actually changed.
  • Fixed-at-creation fields — Fields Pub/Sub fixes at creation (filter, enable_message_ordering) are reported as warnings instead of failing the run.
  • Emulator gaps — Gaps in the emulators that Google Cloud does not have are downgraded to warnings, so one config works against both.
  • Dry run — Previews every mutation against live state.
  • Never destructive — Never deletes topics, subscriptions or buckets that are not in the config.
config.example.yaml
project_id: local-dev

# Service account key files for local development.
#
# The key is generated here and authenticates to nothing — Google holds no
# matching public key. It exists because many libraries (Spring Cloud GCP among
# them) parse GOOGLE_APPLICATION_CREDENTIALS eagerly at startup and refuse to
# run without a structurally valid key, even when every call they make is
# answered by an emulator.
#
# The file is never rewritten once it exists: its contents are a fresh keypair,
# so rewriting would rotate the credential underneath whatever already loaded
# it. Set `strategy: update` on the entry to force a new one.
credentials:
  - path: /secrets/fake-sa.json
    # Service account id in the generated address. Default: local-emulator.
    account: local-emulator
    # Points every OAuth URL in the file at one address, so a library that does
    # try to mint a token reaches the local stub instead of accounts.google.com.
    # This is what a JVM application needs: google-cloud-storage for Java has no
    # STORAGE_EMULATOR_HOST equivalent, so it authenticates for real and the
    # exchange has to land somewhere local. Omit to use Google's endpoints.
    token_uri: http://gcp-token-stub:8099/token

pubsub:
  topics:
    - name: orders
      labels:
        domain: commerce
      # How long the topic itself retains messages, so a new subscription can
      # seek back into the past. Omit to leave retention to the subscriptions.
      message_retention: 24h

      subscriptions:
        - name: order-processor
          # 10s-600s, and a whole number of seconds: Pub/Sub stores this as
          # ack_deadline_seconds, so a fractional value would be truncated and
          # then reported as drift on every subsequent run.
          ack_deadline: 30s
          # 10m-31d.
Excerpt from config.example.yaml

Full documentation, configuration reference and how-tos.

Read the docs →

Data Rocks AG

[ONE OR TWO LINES ON WHAT DATA ROCKS DOES AND FOR WHOM]

Visit datarocks.ch →