Skip to main content
Version: development

Agent

This document describes the Prometheus metrics generated by Aperture Agents.

Flux Meter

Metrics

NameTypeLabelsUnitDescription
flux_meterHistogramagent_group, instance, job, process_uuid, flux_meter_name, decision_type, http_status_code, flow_statusmsFlow's workload duration
invalid_flux_meter_totalCounteragent_group, instance, job, process_uuid, flux_meter_name, decision_type, http_status_code, flow_statuscount (no unit)Count of invalid FluxMeter readings

Labels

NameExampleDescription
agent_groupdefaultAgent Group of the policy that FluxMeter belongs to
instanceaperture-agent-cbfnpHost instance of the Aperture Agent
jobaperture-selfThe configured job name that the target belongs to
process_uuiddc0e82af-6730-4f70-8228-ee91da53ac5fHost instance's UUID
flux_meter_nameservice1-demo-appName of the FluxMeter
decision_typeDECISION_TYPE_ACCEPTED, DECISION_TYPE_REJECTEDWhether the flow was accepted or not
http_status_code200, 503HTTP status code
flow_statusOK, ErrorFlow status. A common label to denote OK or Error across all protocols

Load Scheduler

Metrics

NameTypeLabelsUnitDescription
workload_latency_msSummaryagent_group, instance, job, process_uuid, policy_name, policy_hash, component_id, workload_indexmsLatency summary of workload
incoming_tokens_totalCounteragent_group, instance, job, process_uuid, policy_name, policy_hash, component_idtokenA counter measuring tokens incoming into Scheduler
accepted_tokens_totalCounteragent_group, instance, job, process_uuid, policy_name, policy_hash, component_idtokenA counter measuring tokens admitted by Scheduler

Labels

NameExampleDescription
agent_groupdefaultAgent Group of the policy that Load Scheduler belongs to
instanceaperture-agent-cbfnpHost instance of the Aperture Agent
jobaperture-selfThe configured job name that the target belongs to
process_uuiddc0e82af-6730-4f70-8228-ee91da53ac5fHost instance's UUID
policy_nameservice1-demo-appName of the policy.
policy_hash5kZjjSgDAtGWmLnDT67SmQhZdHVmz0+GvKcOGTfWMVo=Hash of the policy used for checking the integrity of the policy.
workload_index0, 1, 2, defaultIndex of the workload in order of specification in the policy.
component_id13Index of the component in order of specification in the policy.

Rate Limiter

Metrics

NameTypeLabelsUnitDescription
rate_limiter_counter_totalCounteragent_group, instance, job, process_uuid, policy_name, policy_hash, component_id, decision_type, limiter_droppedcount (no unit)A counter measuring the number of times Rate Limiter was triggered

Labels

NameExampleDescription
agent_groupdefaultAgent Group of the policy that Rate Limiter belongs to
instanceaperture-agent-cbfnpHost instance of the Aperture Agent
jobaperture-selfThe configured job name that the target belongs to
process_uuiddc0e82af-6730-4f70-8228-ee91da53ac5fHost instance's UUID
policy_nameservice1-demo-appName of the policy.
policy_hash5kZjjSgDAtGWmLnDT67SmQhZdHVmz0+GvKcOGTfWMVo=Hash of the policy used for checking the integrity of the policy.
component_id13Index of the component in order of specification in the policy.
decision_typeDECISION_TYPE_ACCEPTED, DECISION_TYPE_REJECTEDWhether the flow was accepted or not
limiter_droppedtrue, falseWhether this particular limiter has dropped the request.

Concurrency Limiter

Metrics

NameTypeLabelsUnitDescription
concurrency_limiter_counter_totalCounteragent_group, instance, job, process_uuid, policy_name, policy_hash, component_id, decision_type, limiter_droppedcount (no unit)A counter measuring the number of times Concurrency Limiter was triggered

Labels

NameExampleDescription
agent_groupdefaultAgent Group of the policy that Concurrency Limiter belongs to
instanceaperture-agent-cbfnpHost instance of the Aperture Agent
jobaperture-selfThe configured job name that the target belongs to
process_uuiddc0e82af-6730-4f70-8228-ee91da53ac5fHost instance's UUID
policy_nameservice1-demo-appName of the policy.
policy_hash5kZjjSgDAtGWmLnDT67SmQhZdHVmz0+GvKcOGTfWMVo=Hash of the policy used for checking the integrity of the policy.
component_id13Index of the component in order of specification in the policy.
decision_typeDECISION_TYPE_ACCEPTED, DECISION_TYPE_REJECTEDWhether the flow was accepted or not
limiter_droppedtrue, falseWhether this particular limiter has dropped the request.

Sampler

Metrics

NameTypeLabelsUnitDescription
sampler_counter_totalCounteragent_group, instance, job, process_uuid, policy_name, policy_hash, component_id, decision_type, sampler_droppedcount (no unit)A counter measuring the number of times Sampler was triggered

Labels

NameExampleDescription
agent_groupdefaultAgent Group of the policy that Sampler belongs to
instanceaperture-agent-cbfnpHost instance of the Aperture Agent
jobaperture-selfThe configured job name that the target belongs to
process_uuiddc0e82af-6730-4f70-8228-ee91da53ac5fHost instance's UUID
policy_nameservice1-demo-appName of the policy.
policy_hash5kZjjSgDAtGWmLnDT67SmQhZdHVmz0+GvKcOGTfWMVo=Hash of the policy used for checking the integrity of the policy.
component_id13Index of the component in order of specification in the policy.
decision_typeDECISION_TYPE_ACCEPTED, DECISION_TYPE_REJECTEDWhether the flow was accepted or not
sampler_droppedtrue, falseWhether this particular sampler has dropped the request.

Classifier

Metrics

NameTypeLabelsUnitDescription
classifier_counter_totalCounteragent_group, instance, job, process_uuid, policy_name, policy_hash, classifier_indexcount (no unit)A counter measuring the number of times classifier was triggered

Labels

NameExampleDescription
agent_groupdefaultAgent Group of the policy that Classifier belongs to
instanceaperture-agent-cbfnpHost instance of the Aperture Agent
jobaperture-selfThe configured job name that the target belongs to
process_uuiddc0e82af-6730-4f70-8228-ee91da53ac5fHost instance's UUID
policy_nameservice1-demo-appName of the policy.
policy_hash5kZjjSgDAtGWmLnDT67SmQhZdHVmz0+GvKcOGTfWMVo=Hash of the policy used for checking the integrity of the policy.
classifier_index0, 1Index of the classifier in order of specification in the policy.

Flow Control Summary

Flow Control Metrics

NameTypeLabelsUnitDescription
flowcontrol_requests_totalCountercontrol_point, agent_group, instance, job, process_uuidcount (no unit)Total number of aperture check requests handled
flowcontrol_decisions_totalCountercontrol_point, agent_group, instance, job, process_uuid, decision_typecount (no unit)Number of aperture check decisions
flowcontrol_reject_reasons_totalCountercontrol_point, agent_group, instance, job, process_uuid, reject_reasoncount (no unit)Number of reject reasons other than unspecified
flowcontrol_ends_totalCountercontrol_point, agent_group, instance, job, process_uuidcount (no unit)Total number of flow end calls handled

Flow Control Labels

NameExampleDescription
agent_groupdefaultAgent Group of the Aperture Agent
instanceaperture-agent-cbfnpHost instance of the Aperture Agent
jobaperture-selfThe configured job name that the target belongs to
process_uuiddc0e82af-6730-4f70-8228-ee91da53ac5fHost instance's UUID
decision_typeDECISION_TYPE_ACCEPTED, DECISION_TYPE_REJECTEDWhether the flow was accepted or not
reject_reasonREJECT_REASON_RATE_LIMITED, REJECT_REASON_NO_TOKENS, REJECT_REASON_NOT_SAMPLEDReason why FlowControl Check response rejected the flow.

Distributed Cache Metrics

NameTypeLabelsUnitDescription
distcache_entries_totalGaugeagent_group, instance, job, process_uuid, distcache_member_id, distcache_member_namecount (no unit)Total number of entries in the DMap
distcache_delete_hitsGaugeagent_group, instance, job, process_uuid, distcache_member_id, distcache_member_namecount (no unit)Number of deletion requests resulting in an item being removed in the DMap
distcache_delete_missesGaugeagent_group, instance, job, process_uuid, distcache_member_id, distcache_member_namecount (no unit)Number of deletion requests for missing keys in the DMap
distcache_get_missesGaugeagent_group, instance, job, process_uuid, distcache_member_id, distcache_member_namecount (no unit)Number of entries that have been requested and not found in the DMap
distcache_get_hitsGaugeagent_group, instance, job, process_uuid, distcache_member_id, distcache_member_namecount (no unit)Number of entries that have been requested and found present in the DMap
distcache_evicted_totalGaugeagent_group, instance, job, process_uuid, distcache_member_id, distcache_member_namecount (no unit)Total number of entries removed from cache to free memory for new entries in the DMap
cache_lookup_hits_totalCounteragent_group, instance, job, process_uuid, type, control_pointcount (no unit)Cumulative number of cache lookup hits
cache_lookup_misses_totalCounteragent_group, instance, job, process_uuid, type, control_pointcount (no unit)Cumulative number of cache lookup misses
cache_operation_results_totalCounteragent_group, instance, job, process_uuid, type, control_pointcount (no unit)Cumulative number of cache operation results

Distributed Cache Labels

NameExampleDescription
agent_groupdefaultAgent Group of the Aperture Agent
instanceaperture-agent-cbfnpHost instance of the Aperture Agent
jobaperture-selfThe configured job name that the target belongs to
process_uuiddc0e82af-6730-4f70-8228-ee91da53ac5fHost instance's UUID
distcache_member_id384313659919819706Internal ID of distributed cache cluster member.
distcache_member_name10.244.1.20:3320Internal unique name of distributed cache cluster member.

Scheduler Metrics

NameTypeLabelsUnitDescription
token_bucket_lm_ratioGaugeagent_group, instance, job, process_uuid, policy_name, policy_hash, component_idpercentageA gauge that tracks the load multiplier
token_bucket_fill_rateGaugeagent_group, instance, job, process_uuid, policy_name, policy_hash, component_idtokens/sA gauge that tracks the fill rate of token bucket
token_bucket_capacity_totalGaugeagent_group, instance, job, process_uuid, policy_name, policy_hash, component_idcount (no unit)A gauge that tracks the capacity of token bucket
token_bucket_available_tokens_totalGaugeagent_group, instance, job, process_uuid, policy_name, policy_hash, component_idcount (no unit)A gauge that tracks the number of tokens available in token bucket
workload_requests_totalCounteragent_group, instance, job, process_uuid, policy_name, policy_hash, component_id, workload_index, decision_type, limiter_droppedcount (no unit)A counter of workload requests
request_in_queue_duration_msSummaryagent_group, instance, job, process_uuid, policy_name, policy_hash, component_id, workload_indexmsMetric used for grouping durations for requests by workload in queue of Scheduler
workload_preempted_tokensSummaryagent_group, instance, job, process_uuid, policy_name, policy_hash, component_id, workload_indextokenMetric used for counting tokens preempted per request measured end-to-end in the scheduler across all workloads.
workload_delayed_tokensSummaryagent_group, instance, job, process_uuid, policy_name, policy_hash, component_id, workload_indextokenMetric used for counting tokens delayed per request measured end-to-end in the scheduler across all workloads.
workload_on_time_totalCounteragent_group, instance, job, process_uuid, policy_name, policy_hash, component_id, workload_index, decision_type, limiter_droppedcount (no unit)Metric used for counting requests that are on time, neither preempted nor delayed measured end-to-end in the scheduler across all workloads.
fairness_preempted_tokensSummaryagent_group, instance, job, process_uuid, policy_name, policy_hash, component_id, fairness_indextokenMetric used for counting tokens preempted per request measured at fairness queues within the same workload.
fairness_delayed_tokensSummaryagent_group, instance, job, process_uuid, policy_name, policy_hash, component_id, fairness_indextokenMetric used for counting tokens delayed per request measured at fairness queues within the same workload.
fairness_on_time_totalCounteragent_group, instance, job, process_uuid, policy_name, policy_hash, component_id, fairness_index, decision_type, limiter_droppedcount (no unit)Metric used for counting requests that are on time, neither preempted nor delayed measured at fairness queues within the same workload.

Scheduler Labels

NameExampleDescription
agent_groupdefaultAgent Group of the policy that Scheduler belongs to
instanceaperture-agent-cbfnpHost instance of the Aperture Agent
jobaperture-selfThe configured job name that the target belongs to
process_uuiddc0e82af-6730-4f70-8228-ee91da53ac5fHost instance's UUID
policy_nameservice1-demo-appName of the policy.
policy_hash5kZjjSgDAtGWmLnDT67SmQhZdHVmz0+GvKcOGTfWMVo=Hash of the policy used for checking the integrity of the policy.
component_idroot.13Index of the component in order of specification in the policy.
workload_index0, 1, 2, defaultIndex of the workload in order of specification in the policy.
component_id13Index of the component in order of specification in the policy.
decision_typeDECISION_TYPE_ACCEPTED, DECISION_TYPE_REJECTEDWhether the flow was accepted or not
limiter_droppedtrue, falseWhether this particular limiter has dropped the request.