S3 Compatible Object Storage

Plugin: go.d.plugin Module: s3check

Overview

Monitor S3-compatible object storage with active checks that write, read, list, and delete small probe objects and time their replication between sites.

The collector runs on the Netdata Agent as an ordinary S3 client, so every result includes what a real client experiences on that path: DNS, network, proxies, TLS, authentication, and the service itself. It reports the outcome and duration of each S3 operation, whether the content read back matches what was written, how long replication took compared with your objectives, how many probe objects still await cleanup, and whether new probes are paused because cleanup is backlogged.

One job runs one of three modes:

Mode	What it checks	Bucket requirements
lifecycle	One endpoint. Writes a probe object, reads it back and verifies its content, confirms it is listed, deletes it, and confirms it is gone. Works with AWS S3, Ceph RGW, and other S3-compatible services.	Versioning never enabled
ceph_multisite	One replication direction between two Ceph RGW zones. Writes at the source, waits until the same object is readable at the destination, deletes it at the source, and waits until the destination no longer serves it.	Versioning never enabled on either bucket
aws_replication	One replication direction between two AWS S3 buckets, following AWS replication semantics: the object version created by the write and the delete marker created by the delete are each expected to replicate.	Versioning enabled on both buckets and a replication rule that replicates delete markers

The lifecycle mode checks the life cycle of a probe object; it does not inspect S3 Lifecycle Management rules. Replication modes check one direction only: to validate a bidirectional setup, configure one job per direction.

Every job owns a private key space inside the configured prefix, <prefix><owner>/probe-..., where the owner segment is derived from this Agent and job. The collector writes, reads, lists, and deletes only keys it has recorded in that space. It never scans or touches other objects in the bucket, and Agents or jobs that share a bucket and prefix never interfere with each other.

Before a probe object is written, its key is recorded in a local journal under the Agent's state directory. If a probe is interrupted by a restart, a network failure, or an ambiguous response, the recorded objects are deleted during later collections, a few per collection. Cleanup is best effort: it requires the same job name, mode, endpoints, buckets, prefix, and working credentials, and it cannot remove objects when the service or the permissions are gone. The number of objects awaiting cleanup is bounded. When that bound is reached, the collector keeps observing and cleaning up but stops creating new probe objects until a slot is free, and the mutation_backpressure chart shows the pause.

Replication modes poll the destination once per collection until the configured timeout. Each replication job has objectives, the propagation time you consider healthy, and timeouts, the point at which the probe is declared failed and its object is scheduled for cleanup. Delete propagation is measured as the moment the destination stops serving the object; how the storage system reclaims space behind that is not observed. In aws_replication mode cleanup deletes only the exact object versions and delete markers the probe created.

This collector is supported on all platforms.

This collector supports collecting metrics from multiple instances of this integration, including remote instances.

Grant each endpoint only the operations its role needs, scoped to the configured bucket and, for object operations, to the configured prefix:

Mode	Source endpoint	Destination endpoint
lifecycle	s3:GetBucketVersioning, s3:PutObject, s3:GetObject, s3:ListBucket, s3:DeleteObject	not used
ceph_multisite	s3:GetBucketVersioning, s3:PutObject, s3:GetObject, s3:DeleteObject	s3:GetBucketVersioning, s3:GetObject, s3:DeleteObject
aws_replication	s3:GetBucketVersioning, s3:GetReplicationConfiguration, s3:PutObject, s3:GetObject, s3:GetObjectVersion, s3:ListBucketVersions, s3:DeleteObject, s3:DeleteObjectVersion	s3:GetBucketVersioning, s3:GetObject, s3:ListBucketVersions, s3:DeleteObjectVersion

The collector never writes to a destination; its delete permission is used only to remove the replicated copy of a probe object during cleanup. When assume_role is configured, the base credentials also need sts:AssumeRole on that role.

Default Behavior

Auto-Detection

There is no auto-detection. A job needs a mode, a source region and bucket, credentials or an AWS SDK credential source on the Agent host, and a destination for the replication modes.

Limits

Each job runs one probe at a time and cleans up a few pending objects per collection. The number of objects tracked for cleanup is a fixed internal bound; when it is reached, new probes pause until cleanup frees a slot. Probe objects are 4 KiB. Replication objectives must be at least one collection interval, because the destination is polled once per collection, and objectives and timeouts are capped at 24 hours.

Performance Impact

The impact on the Agent is negligible. On the storage side each collection issues a handful of small requests under the configured prefix: at most one new probe object with its reads, listing, and delete, plus a few cleanup deletes. Use a dedicated bucket or reserved prefix, credentials restricted to it, and a collection interval that matches how often you want to exercise the service.

Setup

You can configure the s3check collector in two ways:

Method	Best for	How to
UI	Fast setup without editing files	Go to Nodes → Configure this node → Collectors → Jobs, search for s3check, then click + to add a job.
File	If you prefer configuring via file, or need to automate deployments (e.g., with Ansible)	Edit go.d/s3check.conf and add a job.

important

UI configuration requires paid Netdata Cloud plan.

Prerequisites

Choose a mode and prepare the buckets

The collector verifies the bucket contract of its mode before every collection and refuses to run when it is not met:

Mode	Buckets	Versioning	Replication policy
lifecycle	one bucket	never enabled; a suspended state is also rejected	none
ceph_multisite	one bucket per zone, names may differ	never enabled on either bucket	the zones must sync the bucket so that a key written at the source appears unchanged at the destination
aws_replication	one bucket per side, names may differ	enabled on both buckets, MFA Delete off	an enabled replication rule on the source bucket that covers the prefix without tag filters, targets the destination bucket, and has delete marker replication enabled; no other enabled rule may replicate the prefix to a different bucket

The destination key is always identical to the source key. Policies that rename or re-prefix objects are not supported. A dedicated bucket is the simplest way to meet these requirements, but any bucket that satisfies them works.

Every probe object is written with a conditional If-None-Match: * request so that a probe can never overwrite an existing key; the service must support conditional writes.

Reserve an object prefix

Pick a prefix, by default netdata-s3check/, and do not store your own data under it. The collector creates and deletes objects only under <prefix><owner>/, where the owner segment is unique to this Agent and job, so several Agents and jobs can safely share a prefix and even a bucket. Keys outside the collector's own namespace are never listed, read, or deleted.

Create credentials with the minimum permissions

Each endpoint authenticates independently. Omit credentials to use the AWS SDK default credential chain on the Agent host (environment variables, shared config and profiles, instance or task roles), or provide static keys, preferably as go.d secret references such as ${env:NAME} rather than plaintext. Optionally add assume_role to switch to a role with those base credentials.

Grant only the operations listed in the permissions table of the Overview, scoped to the bucket and, for object operations, to the prefix. When assume_role is configured, the base credentials must also be allowed to call sts:AssumeRole on that role.

Run the job where your clients run

Results reflect the network path of the Agent that runs the job: DNS, routing, proxies, TLS, and authentication are all part of the measurement. Run each job on the Agent that best represents the clients you care about. Replication checks cover one direction; add a second job with source and destination swapped to check the reverse direction.

Configuration

Options

Set mode and configure only the matching mode_* object; the other two must be absent. Durations accept human-readable values such as 10s, 5m, 1h, or 1d.

Endpoint settings. Every source and destination object takes the same connection settings:

• endpoint, region, bucket, path_style: where the bucket lives. For AWS S3 leave endpoint empty and set path_style: no; for Ceph RGW and other S3-compatible services set the full URL and keep path-style addressing.
• credentials: static access keys. Omit the object to use the AWS SDK default credential chain of the Agent host. Prefer go.d secret references such as ${env:NAME} over plaintext values.
• assume_role: optional. The base credentials call STS AssumeRole and the returned temporary credentials are used for this endpoint only.
• timeout, proxy_url, and the tls_* options apply to every S3 request and to the STS AssumeRole request of that endpoint. An empty proxy_url honors the HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment of the Agent.

Prefix. The collector only writes, reads, lists, and deletes keys under <prefix><owner>/, where the owner segment is derived from this Agent and job. Reserve the prefix for Netdata; several Agents and jobs may share it.

Objectives and timeouts (replication modes). An objective is the propagation time you consider healthy: exceeding it marks the write_visibility_objective or delete_visibility_objective chart as breached while the probe keeps waiting. A timeout is the hard limit: exceeding it fails the probe with reason visibility_timeout or delete_timeout and schedules the object for cleanup. Objectives must be at least update_every, because the destination is polled once per collection, timeouts must be at least their objective, and all four are capped at 24h.

Config options

Group	Option	Description	Default	Required
Mode	mode	Check to run: lifecycle, ceph_multisite, or aws_replication.	lifecycle	no
Mode / Lifecycle	mode_lifecycle	Settings of the lifecycle mode. Required when mode is lifecycle.		no
	mode_lifecycle.prefix	Key prefix reserved for probe objects. Must end with /.	netdata-s3check/	no
Mode / Lifecycle connection	mode_lifecycle.source	The S3 endpoint to check. Required.		no
	mode_lifecycle.source.name	Label for this endpoint on charts (the source label).	source	no
	mode_lifecycle.source.endpoint	Service URL with scheme and host only. Empty uses the AWS regional endpoint.		no
	mode_lifecycle.source.region	Signing region such as us-east-1. Required, also for non-AWS services.		no
	mode_lifecycle.source.bucket	Bucket that holds the probe objects. Required.		no
	mode_lifecycle.source.path_style	Path-style addressing (https://host/bucket/key). Set to no for AWS S3.	yes	no
	mode_lifecycle.source.credentials	Static access keys. Omit to use the AWS SDK default credential chain.		no
	mode_lifecycle.source.credentials.access_key_id	Access key ID. Required when credentials is set.		no
	mode_lifecycle.source.credentials.secret_access_key	Secret access key. Required when credentials is set.		no
	mode_lifecycle.source.credentials.session_token	Session token for temporary credentials.		no
	mode_lifecycle.source.assume_role	IAM role to assume with this endpoint's base credentials.		no
	mode_lifecycle.source.assume_role.role_arn	ARN of the role to assume. Required when assume_role is set.		no
	mode_lifecycle.source.assume_role.external_id	External ID expected by the role's trust policy.		no
	mode_lifecycle.source.timeout	Maximum duration of one S3 or STS request, up to 1m.	10s	no
	mode_lifecycle.source.proxy_url	HTTP proxy for this endpoint's S3 and STS requests. Empty uses the proxy environment.		no
	mode_lifecycle.source.tls_skip_verify	Skip TLS certificate verification. Insecure.	no	no
	mode_lifecycle.source.tls_ca	Absolute path to a CA bundle for a private CA.		no
	mode_lifecycle.source.tls_cert	Absolute path to a client certificate. Requires tls_key.		no
	mode_lifecycle.source.tls_key	Absolute path to the client certificate key. Requires tls_cert.		no
Mode / Ceph multisite	mode_ceph_multisite	Settings of the ceph_multisite mode. Required when mode is ceph_multisite.		no
	mode_ceph_multisite.prefix	Key prefix reserved for probe objects. Must end with /.	netdata-s3check/	no
Mode / Ceph multisite source	mode_ceph_multisite.source	Where probe objects are written and deleted. Required.		no
	mode_ceph_multisite.source.name	Label for this endpoint on charts (the source label).	source	no
	mode_ceph_multisite.source.endpoint	Service URL with scheme and host only. Empty uses the AWS regional endpoint.		no
	mode_ceph_multisite.source.region	Signing region such as us-east-1. Required, also for non-AWS services.		no
	mode_ceph_multisite.source.bucket	Bucket that holds the probe objects. Required.		no
	mode_ceph_multisite.source.path_style	Path-style addressing (https://host/bucket/key). Set to no for AWS S3.	yes	no
	mode_ceph_multisite.source.credentials	Static access keys. Omit to use the AWS SDK default credential chain.		no
	mode_ceph_multisite.source.credentials.access_key_id	Access key ID. Required when credentials is set.		no
	mode_ceph_multisite.source.credentials.secret_access_key	Secret access key. Required when credentials is set.		no
	mode_ceph_multisite.source.credentials.session_token	Session token for temporary credentials.		no
	mode_ceph_multisite.source.assume_role	IAM role to assume with this endpoint's base credentials.		no
	mode_ceph_multisite.source.assume_role.role_arn	ARN of the role to assume. Required when assume_role is set.		no
	mode_ceph_multisite.source.assume_role.external_id	External ID expected by the role's trust policy.		no
	mode_ceph_multisite.source.timeout	Maximum duration of one S3 or STS request, up to 1m.	10s	no
	mode_ceph_multisite.source.proxy_url	HTTP proxy for this endpoint's S3 and STS requests. Empty uses the proxy environment.		no
	mode_ceph_multisite.source.tls_skip_verify	Skip TLS certificate verification. Insecure.	no	no
	mode_ceph_multisite.source.tls_ca	Absolute path to a CA bundle for a private CA.		no
	mode_ceph_multisite.source.tls_cert	Absolute path to a client certificate. Requires tls_key.		no
	mode_ceph_multisite.source.tls_key	Absolute path to the client certificate key. Requires tls_cert.		no
Mode / Ceph multisite destination	mode_ceph_multisite.destination	Where replicated copies must appear. Never written by the collector. Required.		no
	mode_ceph_multisite.destination.name	Label for this endpoint on charts (the destination label).	destination	no
	mode_ceph_multisite.destination.endpoint	Service URL with scheme and host only. Empty uses the AWS regional endpoint.		no
	mode_ceph_multisite.destination.region	Signing region such as us-east-1. Required, also for non-AWS services.		no
	mode_ceph_multisite.destination.bucket	Bucket that holds the probe objects. Required.		no
	mode_ceph_multisite.destination.path_style	Path-style addressing (https://host/bucket/key). Set to no for AWS S3.	yes	no
	mode_ceph_multisite.destination.credentials	Static access keys. Omit to use the AWS SDK default credential chain.		no
	mode_ceph_multisite.destination.credentials.access_key_id	Access key ID. Required when credentials is set.		no
	mode_ceph_multisite.destination.credentials.secret_access_key	Secret access key. Required when credentials is set.		no
	mode_ceph_multisite.destination.credentials.session_token	Session token for temporary credentials.		no
	mode_ceph_multisite.destination.assume_role	IAM role to assume with this endpoint's base credentials.		no
	mode_ceph_multisite.destination.assume_role.role_arn	ARN of the role to assume. Required when assume_role is set.		no
	mode_ceph_multisite.destination.assume_role.external_id	External ID expected by the role's trust policy.		no
	mode_ceph_multisite.destination.timeout	Maximum duration of one S3 or STS request, up to 1m.	10s	no
	mode_ceph_multisite.destination.proxy_url	HTTP proxy for this endpoint's S3 and STS requests. Empty uses the proxy environment.		no
	mode_ceph_multisite.destination.tls_skip_verify	Skip TLS certificate verification. Insecure.	no	no
	mode_ceph_multisite.destination.tls_ca	Absolute path to a CA bundle for a private CA.		no
	mode_ceph_multisite.destination.tls_cert	Absolute path to a client certificate. Requires tls_key.		no
	mode_ceph_multisite.destination.tls_key	Absolute path to the client certificate key. Requires tls_cert.		no
Mode / Ceph multisite objectives	mode_ceph_multisite.write_objective	Healthy propagation time for a new object. Exceeding it flags the objective chart only.	15m	no
	mode_ceph_multisite.write_timeout	Hard limit for a new object to appear at the destination. Exceeding it fails the probe.	30m	no
	mode_ceph_multisite.delete_objective	Healthy time for a source delete to hide the destination copy.	5m	no
	mode_ceph_multisite.delete_timeout	Hard limit for the destination copy to disappear. Exceeding it fails the probe.	15m	no
Mode / AWS replication	mode_aws_replication	Settings of the aws_replication mode. Required when mode is aws_replication.		no
	mode_aws_replication.prefix	Key prefix reserved for probe objects. Must end with /.	netdata-s3check/	no
Mode / AWS replication source	mode_aws_replication.source	Where probe objects are written and deleted. Required.		no
	mode_aws_replication.source.name	Label for this endpoint on charts (the source label).	source	no
	mode_aws_replication.source.endpoint	Service URL with scheme and host only. Empty uses the AWS regional endpoint.		no
	mode_aws_replication.source.region	Signing region such as us-east-1. Required, also for non-AWS services.		no
	mode_aws_replication.source.bucket	Bucket that holds the probe objects. Required.		no
	mode_aws_replication.source.path_style	Path-style addressing (https://host/bucket/key). Set to no for AWS S3.	yes	no
	mode_aws_replication.source.credentials	Static access keys. Omit to use the AWS SDK default credential chain.		no
	mode_aws_replication.source.credentials.access_key_id	Access key ID. Required when credentials is set.		no
	mode_aws_replication.source.credentials.secret_access_key	Secret access key. Required when credentials is set.		no
	mode_aws_replication.source.credentials.session_token	Session token for temporary credentials.		no
	mode_aws_replication.source.assume_role	IAM role to assume with this endpoint's base credentials.		no
	mode_aws_replication.source.assume_role.role_arn	ARN of the role to assume. Required when assume_role is set.		no
	mode_aws_replication.source.assume_role.external_id	External ID expected by the role's trust policy.		no
	mode_aws_replication.source.timeout	Maximum duration of one S3 or STS request, up to 1m.	10s	no
	mode_aws_replication.source.proxy_url	HTTP proxy for this endpoint's S3 and STS requests. Empty uses the proxy environment.		no
	mode_aws_replication.source.tls_skip_verify	Skip TLS certificate verification. Insecure.	no	no
	mode_aws_replication.source.tls_ca	Absolute path to a CA bundle for a private CA.		no
	mode_aws_replication.source.tls_cert	Absolute path to a client certificate. Requires tls_key.		no
	mode_aws_replication.source.tls_key	Absolute path to the client certificate key. Requires tls_cert.		no
Mode / AWS replication destination	mode_aws_replication.destination	Where replicated copies must appear. Never written by the collector. Required.		no
	mode_aws_replication.destination.name	Label for this endpoint on charts (the destination label).	destination	no
	mode_aws_replication.destination.endpoint	Service URL with scheme and host only. Empty uses the AWS regional endpoint.		no
	mode_aws_replication.destination.region	Signing region such as us-east-1. Required, also for non-AWS services.		no
	mode_aws_replication.destination.bucket	Bucket that holds the probe objects. Required.		no
	mode_aws_replication.destination.path_style	Path-style addressing (https://host/bucket/key). Set to no for AWS S3.	yes	no
	mode_aws_replication.destination.credentials	Static access keys. Omit to use the AWS SDK default credential chain.		no
	mode_aws_replication.destination.credentials.access_key_id	Access key ID. Required when credentials is set.		no
	mode_aws_replication.destination.credentials.secret_access_key	Secret access key. Required when credentials is set.		no
	mode_aws_replication.destination.credentials.session_token	Session token for temporary credentials.		no
	mode_aws_replication.destination.assume_role	IAM role to assume with this endpoint's base credentials.		no
	mode_aws_replication.destination.assume_role.role_arn	ARN of the role to assume. Required when assume_role is set.		no
	mode_aws_replication.destination.assume_role.external_id	External ID expected by the role's trust policy.		no
	mode_aws_replication.destination.timeout	Maximum duration of one S3 or STS request, up to 1m.	10s	no
	mode_aws_replication.destination.proxy_url	HTTP proxy for this endpoint's S3 and STS requests. Empty uses the proxy environment.		no
	mode_aws_replication.destination.tls_skip_verify	Skip TLS certificate verification. Insecure.	no	no
	mode_aws_replication.destination.tls_ca	Absolute path to a CA bundle for a private CA.		no
	mode_aws_replication.destination.tls_cert	Absolute path to a client certificate. Requires tls_key.		no
	mode_aws_replication.destination.tls_key	Absolute path to the client certificate key. Requires tls_cert.		no
Mode / AWS replication objectives	mode_aws_replication.write_objective	Healthy propagation time for a new object. Exceeding it flags the objective chart only.	15m	no
	mode_aws_replication.write_timeout	Hard limit for a new object to appear at the destination. Exceeding it fails the probe.	30m	no
	mode_aws_replication.delete_objective	Healthy time for a source delete to hide the destination copy.	5m	no
	mode_aws_replication.delete_timeout	Hard limit for the destination copy to disappear. Exceeding it fails the probe.	15m	no
Collection	update_every	Collection interval in seconds. Replication modes poll the destination once per interval.	120	no
	autodetection_retry	Seconds between retries when the initial bucket check fails. 0 disables retries.	0	no
Virtual Node	vnode	Virtual Node that owns the charts of this job.		no

mode

Select the mode first and configure only its mode_* object. A job whose mode object is missing, or that also sets another mode's object, is rejected. The Overview compares the three modes.

mode_lifecycle

Each collection writes one 4 KiB probe object under the prefix, reads it back and verifies its content, confirms it appears in a listing, deletes it, and confirms it is gone. A successful collection leaves nothing behind.

The bucket must never have had versioning enabled; a suspended state is also rejected, because a plain delete must remove the object completely. Required permissions on the bucket: s3:GetBucketVersioning, s3:PutObject, s3:GetObject, s3:ListBucket, s3:DeleteObject.

mode_ceph_multisite

Writes a probe object at the source zone, polls the destination zone once per collection until the same key is readable there with the same content, deletes it at the source, and polls until the destination no longer serves it. Write lag is measured from the source write, delete lag from the source delete.

Both buckets must be unversioned. The source needs s3:GetBucketVersioning, s3:PutObject, s3:GetObject, s3:DeleteObject; the destination needs s3:GetBucketVersioning, s3:GetObject, s3:DeleteObject. The collector never writes to the destination and deletes there only its own replicated copies during cleanup. Use one job per direction.

mode_aws_replication

Follows AWS S3 replication semantics: the source write creates an object version, the source delete creates a delete marker, and both are expected to replicate. The collector records the exact version and marker IDs on both sides and deletes only those during cleanup.

Both buckets need versioning enabled with MFA Delete off. The source bucket needs an enabled replication rule that covers the prefix without tag filters, targets the destination bucket, and has delete marker replication enabled; no other enabled rule may replicate the prefix to a different bucket. Source permissions: s3:GetBucketVersioning, s3:GetReplicationConfiguration, s3:PutObject, s3:GetObject, s3:GetObjectVersion, s3:ListBucketVersions, s3:DeleteObject, s3:DeleteObjectVersion. Destination permissions: s3:GetBucketVersioning, s3:GetObject, s3:ListBucketVersions, s3:DeleteObjectVersion. Use one job per direction.

via UI

Configure the s3check collector from the Netdata web interface:

1. Go to Nodes.
2. Select the node where you want the s3check data-collection job to run and click the ⚙ (Configure this node). That node will run the data collection.
3. The Collectors → Jobs view opens by default.
4. In the Search box, type s3check (or scroll the list) to locate the s3check collector.
5. Click the + next to the s3check collector to add a new job.
6. Fill in the job fields, then click Test to verify the configuration and Submit to save.
   • Test runs the job with the provided settings and shows whether data can be collected.
   • If it fails, an error message appears with details (for example, connection refused, timeout, or command execution errors), so you can adjust and retest.

via File

The configuration file name for this integration is go.d/s3check.conf.

The file format is YAML. Generally, the structure is:

update_every: 1
autodetection_retry: 0
jobs:
  - name: some_name1
  - name: some_name2

You can edit the configuration file using the edit-config script from the Netdata config directory.

cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config go.d/s3check.conf

Examples

Lifecycle check of an S3-compatible endpoint

Checks one Ceph RGW or other S3-compatible endpoint with static access keys read from environment variables. The bucket must never have had versioning enabled.

Config

jobs:
  - name: s3_lifecycle
    mode: lifecycle
    mode_lifecycle:
      source:
        endpoint: https://s3.example.net
        region: us-east-1
        bucket: netdata-s3check
        credentials:
          access_key_id: ${env:NETDATA_S3CHECK_ACCESS_KEY_ID}
          secret_access_key: ${env:NETDATA_S3CHECK_SECRET_ACCESS_KEY}
        path_style: yes

Lifecycle check through a proxy with a private CA

The proxy and the CA bundle apply to this endpoint's S3 requests (and to STS if assume_role were set). Credentials are omitted, so the AWS SDK default credential chain on the Agent host is used.

Config

jobs:
  - name: s3_lifecycle_proxied
    mode: lifecycle
    mode_lifecycle:
      prefix: monitoring/netdata/
      source:
        endpoint: https://s3.internal.example.net
        region: us-east-1
        bucket: netdata-s3check
        proxy_url: http://proxy.example.net:3128
        tls_ca: /etc/ssl/private-ca.pem
        timeout: 15s

Ceph RGW multisite, one direction

Measures replication from zone A to zone B. Add a second job with source and destination swapped to check the reverse direction. Both buckets must be unversioned, and each zone uses its own keys.

Config

jobs:
  - name: ceph_site_a_to_site_b
    mode: ceph_multisite
    mode_ceph_multisite:
      source:
        name: site-a
        endpoint: https://rgw-site-a.example.net
        region: us-east-1
        bucket: netdata-s3check
        credentials:
          access_key_id: ${env:NETDATA_S3CHECK_SITE_A_ACCESS_KEY_ID}
          secret_access_key: ${env:NETDATA_S3CHECK_SITE_A_SECRET_ACCESS_KEY}
      destination:
        name: site-b
        endpoint: https://rgw-site-b.example.net
        region: us-east-1
        bucket: netdata-s3check
        credentials:
          access_key_id: ${env:NETDATA_S3CHECK_SITE_B_ACCESS_KEY_ID}
          secret_access_key: ${env:NETDATA_S3CHECK_SITE_B_SECRET_ACCESS_KEY}
      write_objective: 15m
      write_timeout: 30m
      delete_objective: 5m
      delete_timeout: 15m

AWS S3 replication with a role per account

Checks replication from a bucket in one region to a bucket in another. Base credentials come from the AWS SDK default credential chain on the Agent host, for example an instance profile, and each endpoint then assumes its own role. Both buckets must be versioned and the source rule must replicate delete markers. path_style is disabled because AWS S3 uses virtual-hosted-style addressing.

Config

jobs:
  - name: aws_replication
    mode: aws_replication
    mode_aws_replication:
      source:
        name: us-east-1
        region: us-east-1
        bucket: source-bucket
        path_style: no
        assume_role:
          role_arn: arn:aws:iam::[ACCOUNT]:role/netdata-s3-source
      destination:
        name: us-west-2
        region: us-west-2
        bucket: destination-bucket
        path_style: no
        assume_role:
          role_arn: arn:aws:iam::[ACCOUNT]:role/netdata-s3-destination

Alerts

The following alerts are available:

Alert name	On metric	Description
Alert name s3check_runtime_failed	On metrics3check.runtime_status	DescriptionThe collector could not validate provider safety or persist ownership state. Inspect the collector log for the raw provider or local-state error.
Alert name s3check_probe_failed	On metrics3check.probe_status	DescriptionThe active or last terminal probe failed with a bounded reason. Raw provider errors are available only in the collector log.
Alert name s3check_payload_mismatch	On metrics3check.payload_mismatch	DescriptionThe verification read returned different bytes for the exact key written by the probe.
Alert name s3check_write_visibility_objective	On metrics3check.write_visibility_objective	DescriptionThe exact source key has not become readable with the expected payload at ${label:destination} within write_objective.
Alert name s3check_delete_visibility_objective	On metrics3check.delete_visibility_objective	DescriptionThe destination current object remains readable after the source delete for longer than delete_objective.
Alert name s3check_mutation_backpressure	On metrics3check.mutation_backpressure	DescriptionThe bounded ownership queue is full. Cleanup and observation continue, but no new probe object is created until a slot is released.

Metrics

Metrics grouped by scope.

The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels.

All charts carry the mode, source, and destination labels (destination is empty in lifecycle mode). Work that did not happen during a collection leaves a gap rather than a zero, so a missing point means "not measured". Call counters count logical S3 operations as the collector issued them; retries performed inside the AWS SDK are not counted separately. The reason label uses a fixed vocabulary: none, request, payload_mismatch, visibility_timeout, delete_timeout, cleanup, ownership, internal. Raw provider errors appear only in the collector log.

Per job status

Outcome of the collection and of the current or last completed probe.

Labels:

Label	Description
mode	Configured mode.
source	Display name of the source endpoint.
destination	Display name of the destination endpoint; empty in lifecycle mode.
reason	Why the outcome failed; none otherwise.

Metrics:

Metric	Description	Dimensions	Unit
s3check.runtime_status	S3 check runtime status	success, failed	status
s3check.probe_status	S3 check probe status	success, waiting, failed	status
s3check.payload_mismatch	S3 payload verification mismatch	mismatch	status

Per replication objectives

How long a source change took to become visible at the destination and whether that exceeded the configured objective. Replication modes only.

Labels:

Label	Description
mode	Configured replication mode.
source	Display name of the source endpoint.
destination	Display name of the destination endpoint.
reason	Why the probe failed; none otherwise.

Metrics:

Metric	Description	Dimensions	Unit
s3check.write_visibility_lag	S3 write visibility lag	lag	seconds
s3check.write_visibility_objective	S3 write visibility objective status	breached	status
s3check.delete_visibility_lag	S3 delete visibility lag	lag	seconds
s3check.delete_visibility_objective	S3 delete visibility objective status	breached	status

Per ownership

Probe objects still awaiting cleanup and whether new probes are paused because the cleanup bound is reached.

Labels:

Label	Description
mode	Configured mode.
source	Display name of the source endpoint.
destination	Display name of the destination endpoint; empty in lifecycle mode.

Metrics:

Metric	Description	Dimensions	Unit
s3check.cleanup_pending_objects	S3 owned objects pending cleanup	pending	objects
s3check.mutation_backpressure	S3 mutation backpressure status	active	status

Per operation

One logical S3 operation class on one endpoint: setup (precondition checks), put, read, list, delete, write_visibility, delete_visibility, reconcile, or cleanup.

Labels:

Label	Description
mode	Configured mode.
endpoint	source or destination.
operation	Logical operation name.
source	Display name of the source endpoint.
destination	Display name of the destination endpoint; empty in lifecycle mode.
reason	Why the operation failed; none otherwise. Not present on s3check.operation_calls.

Metrics:

Metric	Description	Dimensions	Unit
s3check.operation_status	S3 logical operation status	success, failed	status
s3check.operation_duration	S3 logical operation duration	duration	seconds
s3check.operation_calls	S3 logical calls	calls, failures	calls/s

Troubleshooting

Debug Mode

Important: Debug mode is not supported for data collection jobs created via the UI using the Dyncfg feature.

To troubleshoot issues with the s3check collector, run the go.d.plugin with the debug option enabled. The output should give you clues as to why the collector isn't working.

• Navigate to the plugins.d directory, usually at /usr/libexec/netdata/plugins.d/. If that's not the case on your system, open netdata.conf and look for the plugins setting under [directories].

  cd /usr/libexec/netdata/plugins.d/

• Switch to the netdata user.

  sudo -u netdata -s

• Run the go.d.plugin to debug the collector:

  ./go.d.plugin -d -m s3check

  To debug a specific job:

  ./go.d.plugin -d -m s3check -j jobName

Getting Logs

If you're encountering problems with the s3check collector, follow these steps to retrieve logs and identify potential issues:

• Run the command specific to your system (systemd, non-systemd, or Docker container).
• Examine the output for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem.

System with systemd

Use the following command to view logs generated since the last Netdata service restart:

journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep s3check

System without systemd

Locate the collector log file, typically at /var/log/netdata/collector.log, and use grep to filter for collector's name:

grep s3check /var/log/netdata/collector.log

Note: This method shows logs from all restarts. Focus on the latest entries for troubleshooting current issues.

Docker Container

If your Netdata runs in a Docker container named "netdata" (replace if different), use this command:

docker logs netdata 2>&1 | grep s3check

The job fails its initial check

Before writing anything, the collector reads the bucket versioning state of every endpoint and, in aws_replication mode, the replication rules of the source bucket. A failure here means one of:

• the endpoint is not reachable from the Agent (DNS, routing, proxy, TLS, or region);
• the credentials are rejected or lack s3:GetBucketVersioning (or s3:GetReplicationConfiguration);
• the bucket does not meet the mode's versioning contract: never versioned for lifecycle and ceph_multisite, versioning enabled with MFA Delete off for aws_replication;
• in aws_replication mode, no enabled rule covers the prefix and targets the destination bucket with delete marker replication, or another enabled rule replicates the prefix elsewhere.

Chart labels carry only a bounded reason; the exact provider error is in the collector log.

Objects remain pending cleanup or new probes are paused

After an interrupted probe the collector deletes the objects it recorded a few per collection. cleanup_pending_objects shows the backlog and mutation_backpressure turns active when the backlog reaches its bound, which pauses new probes but not observation or cleanup.

Cleanup needs the same job name, mode, prefix, endpoints, buckets, and working credentials that created the objects. If you changed the mode, prefix, endpoint, or bucket while objects were pending, the job refuses to start with an ownership mismatch error; if you renamed the job, the old objects are left behind. In both cases restore the previous configuration under the previous job name until the backlog drains, then change it. A cleanup reason means a probe object could not be confirmed gone, usually a permission problem; an ownership reason means the local state could not be locked or saved. The collector log has the details.

Replication probes stay in waiting or time out

A replication probe reports waiting until the destination serves the object (write) or stops serving it (delete), and fails with visibility_timeout or delete_timeout when the configured timeout passes. Check that the replication policy or zone sync covers the prefix, that the destination credentials can read the bucket, and that the policy delivers the object under exactly the same key. Persistent waits usually mean replication is slower than your objectives and timeouts assume; align them with what the replication setup promises.