Metrics

Metrics document the state of the application in a timescale manner. For further analysis, connect your ASAB application to a time-series database. Influx and Prometheus are supported.

Metrics Service

Create new metrics using MetricsService.

example of counter initialization:

class MyApplication(asab.Application):
    async def initialize(self):
        from asab.metrics import Module
        self.add_module(Module)
        self.MetricsService = self.get_service('asab.MetricsService')
        self.MyCounter = self.MetricsService.create_counter("mycounter", tags={'foo': 'bar'}, init_values={'v1': 0, 'v2': 0})

if __name__ == '__main__':
    app = MyApplication()
    app.run()

See the full example here: https://github.com/TeskaLabs/asab/blob/master/examples/metrics.py

Types of Metrics

  • Gauge stores single numerical values which can go up and down. Implements set() method to set the metric values.
  • Counter is a cumulative metric whose values can increase or decrease. Implements add() and sub() methods.
  • Event per Second Counter (EPSCounter) divides all values by delta time.
  • DutyCycle https://en.wikipedia.org/wiki/Duty_cycle
  • AggregationCounter allows to set() values based on an aggregation function. max() function is default.
  • Histogram represents cumulative histogram with set() method.

Counter, AggregationCounter and Histogram come also in variants respecting dynamic tags. (See section Dynamic Tags.)

All methods that create new metrics objects can be found in the Metrics Service reference. You should never initiate a new metrics object on its own, but always through Metrics Service. Metris initialization is meant to be done in the init time of your application and should not be done during runtime.

ASAB Metrics Interpretation

Metrics Service not only creates new metrics but also periodically collects their values and sends them to selected databases. Every 60 seconds in the application lifetime, Metrics Service gathers values of all ASAB metrics. All Counters (and metric types that inherit from Counter) reset at this event to their initial values by default. Interpretation of ASAB Counters is affected by their resetable nature. Even though they monotonously increase, resetting every minute gives them a different meaning. In a long-term observation (that’s how you most probably monitor the metrics in time-series databases), these metrics count events per minute. Thus, resettable Counters are presented to Prometheus database as gauge-type metrics. Set the reset argument to False when creating a new Counter to disable Counter resetting. This periodic “flush” cycle also causes 60s delay of metric propagation into supported time-series databases.

Initial Values

You can initiate your metric instance with or without initial values. When providing initial values, these values are always present and presented to databases even when these values were untouched during the last 60 seconds. You will always see these values in the data, with initial state or changed by occasion. However, the lifetime of values (name and value pairs) added during runtime is only 60 s. Thus, if this value is not set during 60s period, you won’t see it in your database anymore.

Built-in Tags

Tags help you to sort and group metrics in a selected target database, and analyze the data easily. Several “static” tags are provided directly by ASAB.

host

This is a hostname of the server or machine where the application is running

appclass

This is the name of the application. It is literally the name of the class that inherits from the ASAB Application object.

node_id

Present if NODE_ID environmental variable is specified. Meant to specify a node in the cluster. Automatically set by the Remote Control.

service_id

Present if SERVICE_ID environmental variable is specified. Meant to specify a service in the cluster. Automatically set by the Remote Control.

instance_id

Present if INSTANCE_ID environmental variable is specified. Meant to specify an instance in the cluster. Automatically set by the Remote Control.

You can use with convenience the three last tags even without Remote Control by adding the respective environmental variables to Docker containers (or any other technology you use to run ASAB microservices).

Dynamic Tags

Some metric types (Counter, AggregationCounter, Histogram) allow you to use dynamic tags. All metrics in ASAB carry some tags - Hostname by default and others added by custom. Using dynamic tags allows you to create values with a specific tag-set during runtime. Specific tag-sets expire after defined period. This might be spotted in your time-series database like a mysterious disappearance of unused tags. Specify expiration period in confiuration. Default is 60 s.

example configuration

[asab:metrics]
expiration=60

Timestamp

Timestamp contains the record of the precise moment the metric’s value was created or committed to the database. There are two types of metrics: resettable (is_reset = True) and non-resettable (is_reset = False). To reset a metric means to set it back to its initial value (for example, back to 0). The metric’s type is determined by the reset: bool = True parameter of the metric’s constructor at the moment it is created. We measure non-resettable metrics at the time of their creation (there are several possible methods depending on the metric’s general logic), while the resettable ones are measured when we send data to the database (which is also the moment of them being reset).

  Metric’s Type Description / Methods Time is Measured Timestamp Value Appears is_reset
1F Gauge

Stores single numerical values which can go up and down.

add_field() / set()

when metric is created (actual time) set() for actual time False
2F

Counter

Allows dynamic tags

A cumulative metric; values can increase or decrease Never stops.

add_field() / add() / sub() / flush()

when metric is created (actual time) add() or sub() for actual time False
2T

Counter

Allows dynamic tags

A cumulative metric; values can increase or decrease Set to 0 every 60 seconds.

AgregationCounter behavior is based on the resettable Counter.

every 60 seconds flush() - time of the test flush True
3F EPSCounter

There is an adjustable reset parameter in the metric’s constructor.

reset: bool = True reset: bool = False

***** ***** False
3T EPSCounter

Divides the count of events by the time difference between measurements.

flush()

every 60 seconds flush() True
4T DutyCycle

The fraction of one period in which a signal/system is active. A 60% DC means the signal is on 60% and off 40% of the time.

add_field() / set() / flush()

every 60 seconds flush() True
5F

Aggregation Counter

Allows dynamic tags

Keeps track of max or min value of the Counter. Maximum value is a default.

set() / +inherits from the Counter add()/sub() are overwritten

when metric is created (actual time) set() False
5T

Aggregation Counter

Allows dynamic tags

***** every 60 seconds flush() True
6F

Histogram

Allows dynamic tags

Represents cumulative histogram with a set() method.

add_field() / set() / flush()

when metric is created (actual time) set() False
6T

Histogram

Allows dynamic tags

***** every 60 seconds flush() True

InfluxDB

Metrics can be collected in the Influx time-series database.

configuration example:

[asab:metrics]
target=influxdb

[asab:metrics:influxdb]
url=http://localhost:8086/
bucket=my_bucket
org=my_org
token=my_token

InfluxDB 2.0 API parameters:

  • url - [required] url string of your influxDB
  • bucket - [required] the destination bucket for writes
  • org - [required] specifies the destination organization for writes
  • orgid - [optional] specifies the ID of the destination organization for writes
    (NOTE: If both orgID and org are specified, org takes precedence)
  • token - [required] API token to authenticate to the InfluxDB

InfluxDB <1.8 API parameters:

  • url - [required] url string of your influxDB
  • username - [required] name of influxDB user
  • password - [required] password of influxDB user

Prometheus

Prometheus is a “pull model” time-series database. Prometheus accesses asab/v1/metrics endpoint of ASAB ApiService. Thus, connecting ASAB to Prometheus requires APIService initialization. However, no more configuration is needed. ASAB metrics are presented to Prometheus in OpenMetrics standard format.

prometheus.yaml configuration file example:

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s

scrape_configs:
- job_name: 'metrics_example'
  metrics_path: '/asab/v1/metrics'
  scrape_interval: 10s
  static_configs:
  - targets: ['127.0.0.1:8080']

Note

To satisfy the OpenMetrics format required by Prometheus, you should follow the instructions below:

  • Metrics names must fit regex [a-zA-Z:][a-zA-Z0-9_:]*. (Any other characters are replaced by an underscore. Leading underscores and numbers are stripped. These changes proceed without warning.)
  • Metrics names MUST NOT end with “total” or “created”.
  • Tags SHOULD contain items “unit” and “help” providing metadata to Prometheus.
  • Values MUST be float or integer.

Metrics Endpoints

The API Service in ASAB offers several endpoints that monitor internal ASAB functionality. Some of them present the current state of metrics. Check for Swagger documentation of your ASAB Application REST API by visiting the /doc endpoint.

/asab/v1/metrics

  • This endpoint returns metrics in OpenMetrics format and its primary purpose is to satisfy Prometheus database needs.

/asab/v1/metrics.json

  • This endpoint presents metrics data in JSON format.

/asab/v1/watch_metrics

  • Use this endpoint for developing or monitoring your app from the terminal. It returns a simple table of ASAB metrics. You can filter metrics by name using the filter parameter and tags parameter to show or hide tags.

example commands:

watch curl localhost:8080/asab/v1/watch_metrics
watch curl localhost:8080/asab/v1/watch_metrics?name=web_requests_duration_max,tags=True

HTTP Target

For use cases requiring a push model of metrics digestion, there is an HTTP Target. HTTP Target creates a POST request to configured URL with current metrics state sent as JSON body. Configuration is required.

configuration example:

[asab:metrics]
target=http

[asab:metrics:http]
url=http://consumer_example:8080/consume

Web Requests Metrics

ASAB WebService class automatically provides metrics counting web requests. There are 5 metrics quantifying requests to all ASAB endpoints.

  • web_requests - Counts requests to asab endpoints as events per minute.
  • web_requests_duration - Counts total requests duration to asab endpoints per minute.
  • web_requests_duration_min - Counts minimal request duration to asab endpoints per minute.
  • web_requests_duration_max - Counts maximum request duration to asab endpoints per minute.
  • web_requests_duration_hist - Cumulative histogram counting requests in buckets defined by the request duration.

Native Metrics

You can opt out of Native Metrics through configuration by setting native_metrics to false. Default is true.

example configuration

[asab:metrics]
native_metrics=true

Memory Metrics

A gauge with the name os.stat gathers information about memory usage by your application.

You can find several metric values there:

  • VmPeak - Peak virtual memory size
  • VmLck - Locked memory size
  • VmPin - Pinned memory size
  • VmHWM - Peak resident set size (“high water mark”)
  • VmRSS - Resident set size
  • VmData, VmStk, VmExe - Size of data, stack, and text segments
  • VmLib - Shared library code size
  • VmPTE - Page table entries size
  • VmPMD - Size of second-level page tables
  • VmSwap - Swapped-out virtual memory size by anonymous private pages; shmem swap usage is not included

Logs Counter

There is a default Counter named logs with values warnings, errors, and critical, counting logs with respective levels. It is a humble tool for application health monitoring.

Reference

class asab.metrics.MetricsService(app, service_name)[source]

Bases: asab.abc.service.Service

create_gauge(metric_name, tags=None, init_values=None, help=None, unit=None)[source]

Creates Gauge object.

create_counter(metric_name, tags=None, init_values=None, reset: bool = True, help=None, unit=None, dynamic_tags=False)[source]

Creates Counter or CounterWithDynamicTags object.

create_eps_counter(metric_name, tags=None, init_values=None, reset: bool = True, help=None, unit=None)[source]

Creates EPSCounter object.

create_duty_cycle(metric_name, tags=None, init_values=None, help=None, unit=None)[source]

Creates DutyCycle object.

create_aggregation_counter(metric_name, tags=None, init_values=None, reset: bool = True, aggregator=<built-in function max>, help=None, unit=None, dynamic_tags=False)[source]

Creates AggregationCounter or AggregationCounterWithDynamicTags object.

create_histogram(metric_name, buckets: list, tags=None, init_values=None, reset: bool = True, help=None, unit=None, dynamic_tags=False)[source]

Creates Histogram or HistogramWithDynamicTags object.

class asab.metrics.Gauge(init_values=None)[source]

Bases: asab.metrics.metrics.Metric

Argument init_values is a dictionary with initial values and value names as keys.

set(name: str, value)[source]
Parameters:
  • name – name of the value
  • value – value itself
class asab.metrics.Counter(init_values=None)[source]

Bases: asab.metrics.metrics.Metric

Argument init_values is a dictionary with initial values and value names as keys. If reset is True, Counter resets every 60 seconds.

add(name, value, init_value=None)[source]
Parameters:
  • name – name of the counter
  • value – value to be added to the counter

Adds the value to the counter Values specified by name. If name is not in Counter Values, it will be added.

sub(name, value, init_value=None)[source]
Parameters:
  • name – name of the counter
  • value – value to be subtracted from the counter

Subtracts the value from the counter Values specified by name. If name is not in Counter Values, it will be added.

class asab.metrics.EPSCounter(init_values=None)[source]

Bases: asab.metrics.metrics.Counter

Event per Second Counter Divides the count of event by a time difference between measurements. It effectively produces the EPS metric. The type of the metric is an integer (int).

class asab.metrics.DutyCycle(app, init_values=None)[source]

Bases: asab.metrics.metrics.Metric

https://en.wikipedia.org/wiki/Duty_cycle

now = self.App.time() d = now - self.LastReadyStateSwitch self.LastReadyStateSwitch = now
class asab.metrics.AggregationCounter(init_values=None, aggregator=<built-in function max>)[source]

Bases: asab.metrics.metrics.Counter

Sets value aggregated with the last one. Takes a function object as the aggregator argument. The aggregation function can take two arguments only. Maximum is used as a default aggregation function.

Values (their names and init_values) can be provided when initializing the metrics or added with set() method. add() and sub() methods are not implemented.

set(name, value)[source]

Applies aggregation function on recent Counter value and value in argument and sets the result as new value of the Counter.

class asab.metrics.Histogram(buckets: list, init_values=None)[source]

Bases: asab.metrics.metrics.Metric

Creates cumulative histograms.

set(value_name, value)[source]
class asab.metrics.CounterWithDynamicTags(init_values=None)[source]

Bases: asab.metrics.metrics.MetricWithDynamicTags

add(name, value, tags)[source]
Parameters:
  • name – name of the counter
  • value – value to be added to the counter

Adds the value to the counter Values specified by name. If name is not in Counter Values, it will be added there.

sub(name, value, tags)[source]
Parameters:
  • name – name of the counter
  • value – value to be subtracted from the counter

Subtracts the value from the counter Values specified by name. If name is not in Counter Values, it will be added there.

class asab.metrics.AggregationCounterWithDynamicTags(init_values=None, aggregator=<built-in function max>)[source]

Bases: asab.metrics.metrics.CounterWithDynamicTags

set(name, value, tags)[source]
class asab.metrics.HistogramWithDynamicTags(buckets: list, init_values=None)[source]

Bases: asab.metrics.metrics.MetricWithDynamicTags

Creates cumulative histograms with dynamic tags

set(value_name, value, tags)[source]