Docs menu
On this page
  1. Unit tests
  2. Integration tests: tests/exporter.rs
  3. End-to-end tests: tests/e2e.rs
  4. Coverage
  5. CI
  6. Adding tests

Three layers, from fast and isolated to slow and real:

Layer Where Runs against Count Time Needs
Unit src/** (#[cfg(test)]) Plain functions 120 < 1s Rust
Integration tests/exporter.rs, tests/config_env.rs The real HTTP server and a mock Typesense (wiremock) 77 ~1s Rust
End-to-end tests/e2e.rs A real 3-node Typesense cluster, one exporter, Prometheus and Grafana in Docker 10 ~3–5 min per Typesense version Rust, Docker
make test               # lint + unit + integration
make test-unit          # cargo test --lib
make test-integration   # cargo test --test exporter
make coverage           # unit + integration coverage, fails under 90% of lines
make coverage-html      # HTML coverage report, opens in a browser
make e2e                # starts the stack, runs the e2e tests, tears the stack down
TYPESENSE_VERSION=27.1 make e2e   # the same against an older Typesense (default 30.1)

The config-file unit tests read the real process environment (through CliArgs, which reads TYPESENSE_*), so a developer with TYPESENSE_* variables exported may see failures; run them with those unset (env -u TYPESENSE_HOST ...). tests/config_env.rs sets and clears its own variables and serialises its tests.

A plain cargo test runs unit and integration tests only. The e2e tests sit behind the e2e cargo feature, so they never need Docker by accident.

Unit tests#

Unit tests build a Scrape by hand. Scrape::default() uses the default --metric-names=v1; tests about the 0.4 output set metric_names: MetricNames::Legacy (marked legacy below) or Both.

Metric rendering: src/prometheus_exp.rs#

Shared behaviour and the 0.4 names:

Test Verifies
parses_collection_endpoint GET /collections/books/documents/search splits into method, collection and action; GET /health doesn't match
exports_all_metric_families legacy. A full scrape renders every 0.4 family with the right labels. Per-core CPU covers any core count (core 16). Unnumbered keys, unreported keys and unknown fields are left out. Swap, cache hit ratio, health and raft state are present.
failed_endpoints_are_omitted_not_zeroed both. With nothing fetched, every 0.4 and 1.0 /metrics.json and /stats.json family and typesense_unmapped are absent; health and raft state are 0
unknown_raft_states_are_reported_as_zero Raft state 1 → 1 and 4 → 4; any other state → 0
unparseable_values_are_omitted legacy. A non-numeric value ("n/a") in /metrics.json or a per-endpoint stat is left out, not exported as 0
normalizes_resource_ids_but_keeps_collections_and_actions Document, synonym, key, alias and set IDs become :id; collection names, search/import/export and query strings are handled
merged_endpoints_sum_rates_and_weight_latency Entries that collapse to one endpoint sum their request rates and average latency weighted by rate
exports_scrape_health typesense_scrape_success{endpoint} and typesense_scrape_duration_seconds are rendered
non_collection_paths_skip_by_collection_metrics /health, /collections and /collections/books give no per-collection labels; DELETE …/documents/42 does
output_has_help_and_type_lines Output has # HELP and # TYPE … gauge lines
exports_version_replication_and_collection_documents typesense_info{version}, committed index, queued writes and per-collection document counts are rendered; collections without a count are skipped
new_families_are_omitted_when_their_endpoint_failed No /debug, /status or /collections data means no info, raft index, queued writes or document metrics
status_fields_missing_from_older_typesense_are_omitted A /status without committed_index or queued_writes doesn't produce fake zeros
exports_latency_percentiles_and_cache_counts legacy. p95/p99, max latency and cache hit/miss counts are exported as typesense_stats keys

The 1.0 names (design) and the --metric-names modes:

Test Verifies
default_mode_is_v1 Scrape::default() uses the 1.0 names
maps_every_metrics_json_field Every /metrics.json field lands in its 1.0 family with the expected value: CPU and per-core CPU as ratios, memory, swap, disk, network and allocator values; only the unknown fields go to typesense_unmapped
maps_every_stats_json_field Every /stats.json scalar lands in its 1.0 family: per-operation rates and average latency, all and overloaded rates, the latency quantiles (min is quantile="0", max quantile="1"), pending write batches and cache counts; total is not an operation label
maps_every_quantile_for_every_operation Every quantile field of every operation maps to typesense_request_latency_seconds{operation,quantile}, converted to seconds, with nothing unmapped
maps_routes_and_collections Per-route latency and rate become typesense_route_*{route} and typesense_collection_*{collection,method,action}; non-collection routes stay out of the collection families; no collection_name or endpoint labels
merged_routes_keep_the_normalisation_in_v1 Document IDs still collapse to :id with summed rates and rate-weighted latency
converts_to_base_units Milliseconds become seconds and percentages become ratios; ratios, bytes and rates are passed through
network_bytes_are_counters_and_the_rest_gauges The two network byte families are counters (_total); every other 1.0 family is a gauge
negative_or_unparseable_counters_are_omitted A negative or non-numeric network counter is left out, not exported or sent to typesense_unmapped
unknown_numeric_fields_fall_back_to_unmapped Numeric fields without a 1.0 name become typesense_unmapped{source,key} with the raw value; non-numeric and null fields are left out
unmapped_is_only_emitted_with_v1_names typesense_unmapped appears in v1 and both, never in legacy
v1_families_are_omitted_not_zeroed Failed endpoints, missing fields and non-numeric values produce no 1.0 series; health and raft state are still 0
each_mode_emits_only_its_names v1 has only the 1.0 families, legacy only the 0.4 ones, both has all of them
unchanged_families_are_emitted_once_in_every_mode Health, raft, info, documents and scrape-health families appear exactly once in every mode
both_mode_keeps_legacy_output_byte_for_byte Every legacy line, # HELP and # TYPE included, appears unchanged in both; legacy has the 0.4 shape

Probe targets: src/target.rs#

Test Verifies
bare_host_takes_defaults host takes the default scheme and port
host_with_port host:port for names and IPv4
explicit_schemes http:// and https:// (any case), with or without a port
hosts_are_lowercased Upper-case hosts are normalised
ipv6_in_brackets [::1], [::1]:8108, https://[2001:DB8::1]:443; addresses are written canonically
base_url_brackets_ipv6 The URL the exporter calls brackets IPv6 hosts
surrounding_whitespace_is_trimmed Whitespace around a target is ignored
rejects_bad_targets Empty, other schemes, paths, queries, fragments, credentials, bad or zero ports, unbracketed IPv6, bad characters
a_trailing_dot_is_the_same_host ts. parses as ts; a bare . is invalid
normalizes_and_detects_loopback Brackets, case, a trailing dot and IPv6 spelling (0:0::1 is ::1) are normalised; localhost, 127.0.0.1 and ::1 are loopback
allow_list_matches_host_and_port_ignoring_scheme_and_case An entry allows its host and port in any scheme or case; a different port is refused
allow_list_wildcard_allows_everything * allows any target
empty_allow_list_allows_nothing_until_a_node_is_added Blank entries are skipped; the configured host can be added
allow_list_rejects_bad_entries Entries with a scheme, path, credentials, a zero port or bare IPv6 are errors
has_remote_ignores_loopback_entries Only non-loopback entries (or *) count as remote
lists_matches_exact_nodes_only AllowList::lists matches host (case-insensitive) and port, and * is not a listed node

Logging: src/logging.rs#

Test Verifies
each_level_shows_itself_and_everything_above_it_for_this_crate_only Each --log-level lets through its level and above from this crate; dependencies stay at warn; the default is info
colour_only_on_a_terminal_without_no_color ANSI colours only when stdout is a terminal and NO_COLOR is unset
rust_log_beats_the_log_level RUST_LOG directives replace --log-level, louder or quieter
a_blank_rust_log_is_ignored_and_a_bad_one_is_an_error Unset, empty or blank RUST_LOG falls back to the level; invalid directives are an error naming RUST_LOG
json_format_writes_one_object_per_line --log-format json gives one JSON object per line with level and message

Config file: src/config.rs#

Test Verifies
every_option_can_be_set_in_the_file Every option, a key file, and a cluster with an api_key_env, can come from the file; the cluster inherits the file's port and protocol
relative_paths_are_relative_to_the_config_file A relative key file path resolves from the config file's directory; absolute paths are unchanged
flags_beat_the_file_and_the_file_beats_defaults A flag wins (even one equal to the default); the file replaces built-in defaults
a_flag_key_replaces_the_files_key_settings A key from a flag skips the file's key settings; typesense_api_key_env is read and trimmed
rejects_bad_key_settings_in_the_file Unset or empty typesense_api_key_env, and both key sources, are errors naming the setting
rejects_unknown_keys_and_inline_secrets_without_echoing_them A typo, an inline typesense_api_key or cluster api_key, and unknown tables are errors; the hint points to the file/env options; the secret never appears
invalid_values_name_the_file_and_the_line Wrong types, out-of-range numbers, bad metric_names, probe_max_concurrency = 0 and invalid TOML name the file and line
errors_never_contain_the_values_they_reject A secret-looking string in every typed field, a *_env setting and a protocol never appears in the error; an invalid variable name is reported without echoing it
the_protocol_is_validated_and_lowercased_from_every_source ftp from file, flag or no config is an error; HTTPS/HTTP are stored lowercase and clusters inherit it
the_example_file_loads_once_a_key_is_given deploy/exporter.example.toml loads as-is (clusters commented out)
log_options_follow_flag_then_file_then_default log_level and log_format from the file, a flag (even the default info) beating it, defaults without either
a_missing_config_file_is_an_error_and_no_file_changes_nothing A missing file names its path; without --config nothing changes

Clusters: src/probe_config.rs#

Test Verifies
reads_keys_from_a_file_and_from_the_environment Both key sources, trimming, lookup by host and port (case-insensitive)
protocol_overrides_the_default_and_ports_default A cluster's protocol overrides the default; ports default; IPv6 targets
rejects_missing_empty_or_doubled_keys_naming_the_cluster No key, unset/empty variable, empty or missing key file, and both sources are errors naming the cluster (and file)
rejects_bad_names_targets_protocols_and_empty_configs Cluster names outside [A-Za-z0-9_-]+, bad targets, no targets and a bad protocol
rejects_a_node_in_two_clusters_but_not_twice_in_one A node in two clusters names both; repeating it in one is harmless
debug_never_prints_the_key Debug output redacts the key

API key reload: src/key.rs#

Test Verifies
fixed_key_never_changes A flag or environment key is returned as is; Debug redacts it
changed_contents_give_the_new_key A rewritten key file gives the new key, trimmed
empty_or_missing_file_keeps_the_old_key_and_warns_once An empty or deleted file keeps the old key, records one warning per distinct problem, and a usable file afterwards takes effect and clears it
a_rewrite_with_the_same_length_and_mtime_is_noticed_on_unix A same-length rewrite with the mtime restored (cp -p, rsync -t) is seen through the inode/change time on unix; elsewhere it is not (documented)
a_failed_read_is_retried_so_a_permission_fix_takes_effect An unreadable file keeps the old key and its signature is not remembered, so a later chmod (change time only) takes effect (unix only)
a_key_that_is_not_a_header_value_is_rejected_without_naming_it A BOM-prefixed or control-character key fails at startup and is ignored on reload; the error names the file, never the key
a_slow_file_system_does_not_block_other_callers While one caller is stuck reading the key file (a FIFO), another gets the cached key at once (unix only)
checks_at_most_once_per_interval Within the first second a change is not looked for
symlink_swap_like_a_kubernetes_secret_is_followed Swapping a ..data symlink to a directory with a new key, with the same key length, is noticed (unix only)
startup_fails_on_an_unusable_file A missing or empty key file is an error at startup
reloads_are_counted_and_a_failure_is_flagged_until_the_next_good_read An observed key file: a bad rewrite sets key_reload_failing to 1 and counts a failed attempt per retry, the old key stays in use, a good rewrite counts an ok and sets it back to 0
a_fixed_key_has_nothing_to_observe A key from a flag or environment variable gets no reload series
a_watched_file_is_flagged_without_anyone_using_the_key The background check (10ms here) flags an emptied key file and clears it after a good rewrite, with no call to current(); dropping the watcher stops it
nothing_is_watched_for_fixed_keys watch starts no task when no key is file-backed
nothing_is_watched_without_a_runtime watch returns nothing outside a tokio runtime instead of panicking

Exporter metrics: src/exporter_metrics.rs#

Test Verifies
build_info_carries_the_version typesense_exporter_build_info{version} is 1 and names the crate version
scrapes_count_by_handler_and_result Refused, invalid, ok and not found are counted per handler; every outcome a handler can produce starts at 0
no_series_carries_a_target_or_node_label A refused scrape adds no target, host, port or instance label
requests_count_and_observe_a_duration_per_endpoint A Typesense request adds to its endpoint/result counter and to that endpoint's histogram (count, bucket, sum)
probes_in_flight_goes_up_and_back_and_waits_are_observed The in-flight gauge follows its guards; the waiting gauge is 1 while queued and 0 after; a slot wait lands in the right bucket
a_dropped_guard_counts_a_cancelled_scrape_and_its_wait A ScrapeGuard dropped while queued counts cancelled, observes its wait and clears probes_waiting; a finished one counts its own result (unavailable)
an_unknown_endpoint_is_caught_in_debug_builds An endpoint outside ENDPOINTS trips the debug_assert instead of creating an uninitialised series
key_reload_gauge_flips_on_failure_and_back Failures count and set the gauge to 1; a good reload counts and sets it to 0
a_key_that_is_not_observed_has_no_series Without a file-backed key there are no key reload series

Models: src/typesense/models/#

Test Verifies
exports_every_numbered_core All 32 cores of a 32-core host are exported
ignores_non_core_and_non_string_values system_cpuX_…, keys with extra suffixes, non-string values and unrelated keys are not exported as cores
recognises_core_numbers core_number returns the core of system_cpu12_active_percentage and nothing for the overall CPU, cpuX, suffixed or unrelated keys
fixed_entries_leave_out_cores_and_unknown_fields fixed_entries() gives only the fixed /metrics.json fields, not per-core values or unknown keys
unreported_metrics_are_skipped Only the keys Typesense sent are exported; {} exports nothing
exports_every_numeric_field Every numeric /stats.json field is exported: averages, percentiles, max, cache counts, and fields the exporter has never seen
unreported_null_and_non_numeric_fields_are_omitted null, strings and missing fields are left out instead of becoming 0
per_endpoint_maps_are_not_scalars latency_ms and requests_per_second (per-endpoint objects) aren't treated as scalar stats
parses_name_and_document_count_ignoring_the_schema /collections entries parse to name and num_documents, ignoring the schema; a missing count is None

CLI: src/cli.rs#

Test Verifies
applies_defaults Defaults: protocol http, port 8108, bind 0.0.0.0:8888, timeout 5s, metric names v1
accepts_overrides Every flag overrides its default
probe_concurrency_defaults_to_16_and_must_be_positive --probe-max-concurrency defaults to 16, takes a value, and rejects 0
rejects_invalid_port_and_negative_timeout Port 99999 and a negative timeout are rejected
rejects_two_key_sources --typesense-api-key and --typesense-api-key-file can't both be set
reads_and_trims_the_key_file The key file is read and trimmed; an empty or missing file is an error
never_prints_the_key Debug output redacts the key, and --help hides its environment value
flags_plaintext_keys_to_remote_hosts Plain HTTP to a remote host is flagged; localhost and HTTPS are not
parses_metric_names --metric-names accepts v1, both and legacy and rejects anything else
metric_names_select_the_name_sets v1 selects the 1.0 names, legacy the 0.4 names, both both
metric_names_reads_its_environment_variable --metric-names is also read from TYPESENSE_METRIC_NAMES
parses_probe_allowed_targets --probe-allowed-targets is empty by default and splits on commas
probe_allowed_targets_reads_its_environment_variable Also read from TYPESENSE_PROBE_ALLOWED_TARGETS
host_is_optional_with_an_allow_list --typesense-host can be left out when an allow list is set
config_reads_its_environment_variable --config is also read from TYPESENSE_EXPORTER_CONFIG; with only --config, host and key stay unset until merged
load_from_merges_the_file_and_reports_errors_as_clap_errors CliArgs::load_from fills in the file's values and turns config errors into a clap error; unknown flags still fail
parses_log_level_and_format_and_rejects_anything_else Every level and format is accepted; others are rejected naming the value and the choices
log_options_read_their_environment_variables TYPESENSE_EXPORTER_LOG_LEVEL and TYPESENSE_EXPORTER_LOG_FORMAT
flags_plaintext_keys_sent_by_probe Plain HTTP with * or a non-loopback allowed host is flagged; loopback, no allow list and HTTPS are not

Integration tests: tests/exporter.rs#

Each test starts a wiremock server playing a Typesense node, using fixtures shaped like real responses (tests/fixtures/). It then serves the real exporter router on an ephemeral port and scrapes it over HTTP. The mocks only answer requests that carry the API key.

The exporter runs with the default --metric-names=v1 unless a test says otherwise. "The /metrics.json families" and "the /stats.json families" mean the 1.0 families those endpoints feed (has_metrics_json / has_stats_json in the test file).

Test Scenario Expected
healthy_leader_exports_everything All six endpoints OK, raft leader 200; health 1, raft 1, 1.0 system, swap, per-core CPU, cache and per-collection metrics; no 0.4 names
follower_reports_raft_state_four /debug reports state 4 typesense_raft_state 4
sends_api_key_to_typesense Count the requests Exactly 6 requests, one per endpoint, each with X-TYPESENSE-API-KEY; /collections asks Typesense to leave out schemas
unreachable_typesense_reports_unhealthy_and_omits_metrics Nothing listening on the port 200; health 0, raft 0, the /metrics.json and /stats.json families absent
unauthorized_endpoints_are_omitted 401 on metrics, stats and debug; health OK Health 1, raft 0, the /metrics.json and /stats.json families absent
unhealthy_node_reports_zero_health /health returns 503 {"ok": false} Health 0
health_ok_false_with_200_is_unhealthy /health returns 200 {"ok": false} Health 0
malformed_response_only_drops_that_endpoint /metrics.json returns invalid JSON Only the /metrics.json families are absent; the /stats.json families and health are present
timeout_bounds_a_hung_typesense Every endpoint takes 10s; timeout 1s Answers in under 5s, health 0
zero_timeout_means_no_timeout TYPESENSE_TIMEOUT=0 Requests succeed (0 disables the timeout instead of failing everything)
exports_version_replication_documents_and_percentiles Typesense 30.1-shaped responses from all six endpoints Version, committed index, queued writes, documents per collection, latency quantiles and cache counts
older_typesense_without_status_still_exports_the_rest /status returns 404 and /debug has no version Only the raft index and info metrics disappear; everything else is exported
root_points_to_metrics GET / Mentions /metrics
serve_runs_the_exporter_on_a_given_listener server::serve on a pre-bound listener (no port race) Serves /metrics
scrape_health_reports_each_endpoint_and_the_duration Healthy node typesense_scrape_success 1 for all six endpoints; a scrape duration
scrape_health_flags_the_failing_endpoints 401, malformed JSON and a missing endpoint Those endpoints report 0; the working one reports 1
document_ids_do_not_create_new_series 50 document IDs plus a search in /stats.json Exactly 2 series: documents/:id (rates summed) and documents/search; no key label on the by-collection metrics
reads_the_api_key_from_a_file Key supplied with TYPESENSE_API_KEY_FILE Requests carry the key from the file
a_rotated_key_file_is_used_by_the_next_scrape Scrape, rewrite the key file, scrape again after more than a second, then empty the file and scrape once more The mock sees the first key, then the new key, then the new key again (an empty file keeps the last good key)
a_client_keeps_the_key_it_was_made_with_for_all_its_requests Make a client from the current key, rotate the file, send another request Both requests carry the old key: a scrape never mixes keys
a_rotated_cluster_key_file_is_used_by_the_next_probe Same for a cluster's api_key_file, through /probe The mock sees the old key, then the new one
startup_fails_without_a_readable_key_file Key file missing server::router returns an error
run_fails_when_the_port_is_taken Bind port already in use run returns AddrInUse instead of panicking
v1_names_are_the_default_with_base_units_and_counters Default settings, healthy node Every 1.0 family and no 0.4 family; network bytes are counters; values in seconds and ratios; nothing in typesense_unmapped
legacy_names_reproduce_the_0_4_output --metric-names=legacy Every 0.4 family with the 0.4 keys and labels, no 1.0 family, no typesense_unmapped, only gauges
both_names_emit_old_and_new_and_shared_families_once --metric-names=both Both name sets; shared families (health, raft, info, documents, scrape health) once; every 0.4 series identical to legacy
v1_unknown_fields_fall_back_to_unmapped Fields with no 1.0 name in /metrics.json and /stats.json typesense_unmapped{source,key} with the raw value; fields Typesense didn't send are absent

/probe, using two or three mock nodes and an exporter without --typesense-host:

Test Scenario Expected
probe_scrapes_each_node_with_its_own_labels Probe a leader and a follower Each answer has its own raft state and host/port labels; each node got exactly 6 requests, all with the API key
probe_output_matches_metrics_for_the_same_node /metrics and /probe for the configured node Identical output apart from the scrape duration
a_hung_node_only_affects_its_own_probe One healthy, one hung, one unreachable node, probed together The healthy probe is quick and healthy; the others return 200 with health 0 and no metrics
redirects_are_not_followed_so_the_key_stays_with_the_node An allowed node answers every request with a 302 to another server The probe still returns 200 with typesense_health 0; the redirect target received no request, so the API key didn't follow
probe_refuses_downgrading_an_https_default_to_http Default protocol https, probe http:// an allowed node 403 must be https; the node received no request
probe_concurrency_is_capped --probe-max-concurrency 1, two slow nodes probed at once Both return 200, one after the other
probe_refuses_targets_that_are_not_allowed_without_contacting_them Probe a node, with another port or scheme, that isn't in the allow list 403 not allowed; the node received no request, so no API key left
probe_matches_allowed_targets_case_and_scheme_insensitively Allow LocalHost:port, probe localhost, LOCALHOST and http://LocalHost All allowed
probe_always_allows_the_configured_host --typesense-host set, allow list names another node Probing the configured host works
a_wildcard_allows_any_target --probe-allowed-targets=* Any node can be probed
probe_rejects_malformed_targets Empty, other scheme, path, credentials, bad port, bare IPv6, missing parameter 400 with a reason; nothing is scraped
metrics_is_404_without_a_host_and_points_to_probe No --typesense-host /metrics is 404 and mentions /probe?target=
metrics_is_unchanged_when_a_host_is_set --typesense-host and an allow list /metrics works as before
root_mentions_probe GET / Mentions /metrics and /probe
router_rejects_a_malformed_allowed_target --probe-allowed-targets entry with a path server::router returns an error
router_needs_a_host_or_a_non_empty_allow_list No host, no clusters and an empty allow list (clap accepts an empty TYPESENSE_PROBE_ALLOWED_TARGETS) server::router returns an error
each_cluster_is_probed_with_only_its_own_key Two clusters (two mocks) with different keys, probed through one exporter with no default key Both healthy; each mock received 6 requests, all with its own key and never the other
a_node_in_no_cluster_is_refused_without_a_request Probe a node that no cluster lists 403; the node received no request
a_cluster_with_https_refuses_http_targets Cluster protocol = "https", probe http://node 403 must be https; no request
a_cluster_protocol_overrides_the_default_for_bare_targets Default protocol https, cluster http, bare target 200 and healthy
probe_config_works_alongside_the_default_key_and_host --typesense-host, * and a cluster together /metrics and other nodes use the default key; the cluster node uses only its own, despite *
probe_config_alone_leaves_metrics_404 Only clusters configured /metrics is 404 and points to /probe
router_rejects_a_node_that_has_two_keys A cluster node also in --probe-allowed-targets, or equal to --typesense-host Startup error naming the cluster and the other place
bad_config_files_are_reported_without_leaking_keys Missing file, inline api_key, unreadable key file Errors name the file/cluster; the inline secret isn't echoed
the_exporter_starts_from_a_config_file_alone Host, port, key file and timeout all in the file, no flags /metrics works
router_needs_the_default_key_for_a_host_or_allow_list Clusters plus --typesense-host or an allow list but no default key server::router returns an error; clusters alone are fine
router_rejects_an_ipv6_host_spelled_differently_from_a_cluster_node --typesense-host=0:0::1 and a cluster node [::1] Startup error: one node, one key
router_rejects_an_empty_host_and_ignores_an_empty_key Empty --typesense-host; empty API key server::router returns an error for each
debug_logs_one_line_per_typesense_request Debug capture of one scrape, one endpoint answering 503 Six typesense request lines, each with the endpoint, target=host:port, status (200 or 503) and elapsed_ms, plus a scrape finished line
request_lines_are_hidden_at_the_default_level The same scrape at info No request lines
a_failed_request_is_logged_with_its_error_at_debug Nothing listening, JSON logs Six typesense request failed objects with the error, its cause and elapsed_ms
a_timeout_is_reported_with_the_limit_and_the_setting_to_change Every endpoint hangs, timeout 1s A warn line saying timed out after 1s and naming TYPESENSE_TIMEOUT
startup_logs_the_mode_and_settings_but_never_the_key serve with a key file, an allow list, a /metrics scrape and two probes, captured at trace for every crate Version, bind address, metric names, timeout, the /metrics node and the allow list are logged; the API key appears nowhere
startup_logs_each_cluster_with_its_node_count_and_no_key A cluster with two nodes and a key file, captured at trace cluster prod: 2 node(s); the cluster's key appears nowhere
exporter_metrics_count_metrics_and_probe_scrapes_cumulatively (/exporter/metrics) /metrics, an allowed /probe, a refused one, two malformed ones; then the same again Scrape counts by handler and result; every endpoint read OK twice and then four times; refused and malformed targets, the node address and the API key appear nowhere in the output; no key reload series for a key from a flag
exporter_metrics_never_mix_into_a_node_scrape /metrics and /exporter/metrics of one exporter Neither output contains the other's families
exporter_metrics_count_a_metrics_scrape_without_a_host_as_not_found /metrics on an exporter without a host 404, counted as metrics/not_found
exporter_metrics_split_typesense_requests_into_ok_error_and_timeout Four endpoints hang past a 1s timeout, two return 404 timeout for the four, error for the two, and a duration of about the limit
exporter_metrics_show_probes_in_flight_and_slot_waits Two probes with --probe-max-concurrency 1, a metrics read in between One in flight while one waits, 0 afterwards; two slot waits observed, the second at least 0.3s
exporter_metrics_count_a_probe_cancelled_while_waiting_for_a_slot --probe-max-concurrency 1, a slow node, a second probe whose client is aborted while queued probes_waiting 1 while queued, then 0; scrapes_total{probe,cancelled} 1; the cancelled probe's wait (at least 0.3s) is in the slot-wait histogram; the first probe still finishes ok
exporter_metrics_count_typesense_requests_cancelled_in_flight A probe client aborted while the node is still answering The probe is cancelled, and so are the requests it had in flight (ok stays 0)
exporter_metrics_follow_a_key_file_failing_and_recovering The key file is emptied, then rewritten key_reload_failing{key="default"} 0, then 1 with failed reloads counted, then 0 and one ok; no key in the output
exporter_metrics_name_a_cluster_key_by_its_cluster A cluster's key file is emptied key_reload_failing{key="prod"} is 1 and there is no default series
every_blank_environment_variable_counts_as_unset (tests/config_env.rs) Every option's variable set to a blank value The file's value for every option; without a file, the built-in defaults
empty_environment_variables_fall_back_to_the_file (tests/config_env.rs) Empty TYPESENSE_HOST, TYPESENSE_PROBE_ALLOWED_TARGETS and TYPESENSE_API_KEY The file's host, list and key are used
an_environment_key_beats_the_files_key_settings (tests/config_env.rs) TYPESENSE_API_KEY with a file typesense_api_key_file or _env The environment key is used
list_and_enum_options_follow_the_same_precedence (tests/config_env.rs) TYPESENSE_PROBE_ALLOWED_TARGETS and TYPESENSE_METRIC_NAMES against the file and flags Flag > environment > file
log_options_follow_flag_then_environment_then_file_then_default (tests/config_env.rs) TYPESENSE_EXPORTER_LOG_LEVEL / _FORMAT, flags and the file; blank variables Flag > environment > file > default; blank is ignored
invalid_log_values_are_rejected_with_a_clear_error (tests/config_env.rs) A bad level or format from the environment or the file An error naming the value and the choices (file errors name the line and don't echo the value)
flag_beats_environment_beats_file_beats_default (tests/config_env.rs) Real environment variables, flags and a file Flag > environment > file > default; --config read from TYPESENSE_EXPORTER_CONFIG

Checking that the tests catch bugs. The suite was checked by breaking the code on purpose (re-measured for 1.0.0): - Removing the API key header failed 17 of the 24 integration tests. - Treating any /health response as healthy failed health_ok_false_with_200_is_unhealthy, and nothing else.

End-to-end tests: tests/e2e.rs#

These run against the Docker Compose stack: three Typesense nodes (30.1 by default, any release through TYPESENSE_VERSION; CI runs 30.1 and 27.1) peering by fixed IP, one exporter probing every node (TYPESENSE_PROBE_ALLOWED_TARGETS, TYPESENSE_METRIC_NAMES=v1), Prometheus scraping /probe for each node, with the alert rules, and Grafana with the dashboards provisioned. The checks use the 1.0 names.

Test What it does Checks
cluster_has_one_leader_and_two_followers Probes every node through the exporter and reads Prometheus All nodes healthy; roles are one leader and two followers; each probe reports only its own node; 1.0 system metrics (typesense_system_memory_total_bytes) and no 0.4 typesense_metrics/typesense_stats; Typesense version, committed raft index and queued writes from every node; every probe reads all six endpoints (typesense_scrape_success 1); all 3 probes up
traffic_is_reported_by_the_node_that_served_it Creates a collection through node 1 and searches every node, some searches with use_cache=true Documents replicate to every node; every exporter reports 3 documents in the collection; reads by document ID show up as one typesense_route_requests_per_second series with documents/:id, never raw IDs; each exporter reports the search traffic its own node served; typesense_request_latency_seconds reports search quantiles 0.95, 0.99 and 1 (max); cache_hit_ratio goes above 0
followers_catch_up_and_replication_lag_returns_to_zero Writes 20 documents through node 1 Every node's committed raft index moves past the writes; replication lag (max − min committed index) returns to 0; every node reports 23 documents
prometheus_loads_the_alert_rules Reads Prometheus rules and alerts All eight rules loaded; no alerts while the cluster is healthy (TypesenseExporterProbeRefused on exporter:8888 is ignored: the allow-list scenario refuses targets on purpose)
grafana_dashboards_show_every_node Queries Grafana's data source, the same path the panels use Data source healthy; all four dashboards provisioned; per-core CPU (typesense_system_cpu_core_active_ratio), network received rate, raft state and health return data for all three nodes
leader_failure_is_reported_and_a_new_leader_is_elected Stops the current leader Stopped node reports health 0 and raft 0; a different node becomes leader; TypesenseUnhealthy goes pending; TypesenseNoLeader never fires during the election and clears afterwards
hung_node_and_quorum_loss_raise_no_leader Stops one follower and pauses (freezes) the other, so 2 of 3 nodes are gone The frozen node's probe answers in under 4s (the scrape timeout), reports health 0 and every endpoint as failed; all probes still up; no node reports leader; TypesenseNoLeader goes pending
exporter_health_is_scraped_by_prometheus Reads /exporter/metrics, then asks Prometheus Build info is there and no series carries host or port; no key reload series; the typesense-exporter-self job is up and the /health request counter has grown
probe_refuses_targets_outside_the_allow_list Probes nodes and hosts that aren't allowed, and a target with a path 403 for each target outside the allow list, 400 for the path; /metrics is 404 because no host is configured
parses_exporter_output No Docker needed The test's own Prometheus-format parser handles labels containing spaces

How the scenarios stay independent: - Scenarios that stop or pause nodes restore them, even if the test fails, through a Drop guard. - Every test first waits for a healthy cluster with exactly one leader. - A lock stops scenarios from overlapping within one test process. Under nextest, a single-threaded test group does the same (.config/nextest.toml). - Run one scenario on its own with:

bash cargo test --features e2e --test e2e leader_failure -- --nocapture docker compose down -v

Bugs this suite found during development: - Wrong timeout settings. On Linux, a stopped Typesense makes requests hang. With TYPESENSE_TIMEOUT equal to the scrape timeout, Prometheus abandoned the scrape, so the health alerts never saw data. - A broken alert rule. TypesenseNoLeader was first written with count(...) == 0, which can never fire: in PromQL, count() of nothing returns nothing, not 0. - Cluster peering by hostname. A restarted node couldn't rejoin while a peer was down, because Docker had removed the peer's DNS name. Fixed by peering on fixed IPs.

Coverage#

make coverage        # summary; fails under 90% of lines
make coverage-html   # browsable report

Coverage uses cargo-llvm-cov (cargo install cargo-llvm-cov, plus rustup component add llvm-tools-preview) over unit and integration tests. It is about 98% of lines. The remaining gaps: - main.rs, which the e2e suite exercises as the real binary - the Ctrl-C / SIGTERM shutdown path

CI publishes the figure to the README badge on every push to main.

If the local figure looks wrong (lines marked uncovered that are blank or comments, or more "lines" than the file has), stale build output is mixed in. Run rm -rf target/llvm-cov-target and measure again; cargo llvm-cov clean alone doesn't always clear it.

CI#

.github/workflows/ci.yml runs on every pull request and on pushes to main:

Job Runs
lint cargo fmt --check, cargo clippy --all-targets -- -D warnings
audit cargo audit -D warnings: fails on any known vulnerability or warning in the dependency tree
test Unit and integration tests under cargo llvm-cov (90% minimum); uploads lcov.info; publishes the coverage badge on main
rules promtool check rules on the alert and recording rules, and promtool test rules on alerts.test.yml (each exporter-health alert firing and not firing) and legacy-names.test.yml
kubernetes scripts/validate-kubernetes.sh: renders both deploy/kubernetes folders with kustomize and validates them (and each secret.example.yaml) with kubeconform, including the ServiceMonitor and Probe schemas (CRD schemas pinned to a commit)
e2e A matrix over Typesense 30.1 and 27.1 (fail-fast: false, TYPESENSE_VERSION set from it): cargo nextest run --features e2e --test e2e --profile ci; uploads a JUnit report per version (e2e-junit-<version>); prints exporter logs on failure; always tears the stack down

Two more workflows check every pull request: - rust.yml builds the release binaries: Linux amd64 and arm64 (arm64 on a native ubuntu-24.04-arm runner), macOS amd64 and arm64, and Windows. - docker.yml builds the Docker image for amd64 and arm64, each on a native runner, without pushing. It runs when a pull request touches the Dockerfile, sources or the workflow.

On a version tag, both workflows first run a test job and publish nothing if it fails. The job checks that the tag is v plus the Cargo.toml version (cargo metadata --no-deps, compared with GITHUB_REF_NAME) and fails with a message naming both when they differ, so a release can't ship a binary or image that reports an older version (#5). The check only runs for tag refs, which includes running docker.yml manually and choosing a tag; on a branch nothing is published, so it is skipped. cargo test --locked then fails if Cargo.lock doesn't match Cargo.toml. To fix a failed check, bump the version (and Cargo.lock), merge, and re-create the tag.

Dependabot opens grouped weekly updates for Rust crates, GitHub Actions and the Docker base image, so they go through all of the above. Actions are pinned to commit SHAs, and Dependabot keeps those current.

Adding tests#

  • Pure logic (parsing, rendering): add a unit test next to the code.
  • Anything involving HTTP or Typesense responses: add a case to tests/exporter.rs, mocking Typesense with mount(&mock, route, response).
  • Behaviour that needs a real cluster (raft, replication, alerts, dashboards): add a scenario to tests/e2e.rs.
  • Start with let _stack = stack().await; wait_for_healthy_cluster().await;.
  • Use eventually(...) for anything that takes time to propagate.
  • Restore anything you break with Restore.

This page is generated from docs/testing.md on GitHub. Found a mistake? Open an issue.