The logging system within libmtev represents a directed acyclic graph of
input-output chains. Each node in the graph has a unique name and is called
a "log_stream." Log_streams without a
type attribute have output to downstream nodes
("outlets"). Nodes with a
type attribute have additional output
characteristics (like outputting to a file).
Upon startup, the system will establish several built-in log_streams, only one of
which has a type. The "stderr" log_stream has a type of
file and an output
filedescriptor of 2. Other log_stream are setup and and configured to have
the "stderr" log_stream as their outlet. These log_streams are called: "error", "notice",
"debug." The correspond to the global logging symbols in the C API:
For more information on logging via the API, see the development section
of this documentation relaated to logging. The "debug" log_stream is
disabled by default.
Logs are hierarchical in nomenclature as a convenience. If, in your code, you request a log named "error/foo" and no such log exists in the configuration, a new untyped log will be created and its outlet will be set to "error". This is recursive, so "debug/myapp/facility1" will (unless configured otherwise) outlet to "debug/myapp" which will outlet to "debug." This makes it very simple to semantically separate logs into new error and debugging facilities without worrying about them being lost, while providing the flexibility to configure where things go if other outcomes are desired.
All logging configuration exists within the top-level XML node
Individual log_streams are declared using
<log> stanzas and outlets are
<outlet> stanzas. A log_stream uses all
that are its direct child or direct child of any ancestor node.
"1.0" encoding="utf8" standalone="yes" xml version=<application> <logs> <log name="internal" type="memory" path="10000,1000000" require_env="MEMLOG"/> <log name="logfile" type="file" path="/var/log/app.log" rotate_bytes="10000000" retain_bytes="50000000" timestamps="on"/> <log name="http/access" type="jlog" path="/var/log/app-http.feed(*)"/> <console_output> <outlet name="stderr"/> <outlet name="internal"/> <outlet name="logfile"/> <log name="error"/> </console_output> <components> <error> <outlet name="error"/> <log name="error/example"/> <log name="error/sample"/> </error> <debug> <outlet name="debug"/> <log name="debug/example" disabled="true"/> </debug> </components> </logs> </application>
Let's walk through this sample file to understand what's going on. First there
<log> stanzas establishing log_streams named "internal", "logfile",
"http/access", "error", "error/example", "error/sample", and "debug/example".
Starting at the end, the "debug/example" log_stream is declared in a disabled state
disabled="true" attribute. If walk it and its anscestors we find one
<outlet name="debug"/> child. This log_stream has no type, so any messages sent
into this log are only output to its outlet "debug." You'll notice that no
log_stream named "debug" is declared. We rely on the built-in "debug" log which
is setup to output to "stderr". Also, because we did not declare a "debug"
disabled="false", the default state remains disabled.
Outlets may have a
filter attribute that will apply to messages passing
along these graph edges. A filter is a logical expression using
p is a predicate
field OP <value> or
OP is one of
!~ (for regex matching). If the value is a quoted string, comparisons are
made a strings. If the value is not quoted, then comparisons are attempted numerically.
and any other fields added via the extended metadata logging facilities:
The "error/example" and "error/sample" log_stream are similarly configured to output to the "error"
log_stream as its outlet. But, we've declared the "error" log_stream in this configuration so that
we can manipulate its outlets. The
nodes have no special meaning by name; they are simply used as descriptive
hierarchical containers to allow us to share outlet configuration and to
logically isolate our intentions.
The "error" log_stream already exists as a built-in log_stream. The declaration here
is used to set outlets to the three log_streams named: "stderr", "internal", and "logfile".
As before, we use an arbitrarily named node to contiain the declaration logically; this
The "internal" log_stream is of type
memory which uses an in-memory ring buffer
to store recent log lines. We have a limit ot 10000 log lines and 1000000 bytes. It
is also only active if the environment variable INMEM is set.
The "logfile" log_stream is of type
file and will auto-rotate files as they hit
10 million bytes and delete old log files as the cumulative space consumed
exceeds 50 million bytes. Timestamps are turned on for this log_stream.
The "http/access" log_stream is of type
jlog which is create a Jlog
journaled log for external consumption.
This optionally requires conditions around an environment variable. See
If "on"/"true", additional debugging information (like thread ID) is injected into logged lines.
If "on"/"true", the name of the log is injected into logged lines.
If "on"/"true", timestamps are injected into logged lines.
If "on"/"true", the stream is disabled and attempts to log to the facility will result in a single branch instruction.
Can be set to
json. The default is
plain. This option impacts logs that write output (those with the
If "on"/"true", log line counts are exposed as metrics in the mtev/stats.json
The memory log_stream type establishes an internal ring buffer in memory. There are APIs (including REST endpoints) to retrieve the contents of this ring buffer. Additionally, if the process crashes one can examine the contents of the ring buffer with a debugger.
pathattribute takes two numbers comma separated. The first number is the maximum number of log lines to be retained. The second number is the maximum number of bytes to be retained. The implementation will not exceed either limit.
The file log_stream type is used to drive writing to ordinary files using the POSIX API. It provides both time-based and size-based retention management capabilities.
The path is the filename to which log data should be written.
Specifies how many seconds of log data should be written into a file before it is moved aside and a new file is started. (used with
retain_secondsand not with
Specified the number of seconds of data to be retained. If all log data in a rotated file are older than this value, the file will be removed. (used with
rotate_secondsand not with
Specifies how many bytes of log data should be written into a file before it is moved aside and a new file is started. (used with
retain_bytesand not with
Specified the number of bytes of data to be retained. If all log data in a in all rotated files exceed this value, the oldest file will be removed. (used with
rotate_bytesand not with
The jlog log_stream type implements an log output to the Jlog multi-file journalled logging format. Jlog is a segmented write-ahead log that is fast and efficient and supports multiple subscribers with independently maintained process checkpoints.
The path is the Jlog directory to be used. It may optionally be ended with a parenthesized subscriber name. If a name (other than "*") is provided, a subscriber of that name will be added to the Jlog on creation.
The jlog can be further configured using a
<config> stanza supporting the following
Set advisory segment size for the jlog in bytes. By default jlog uses 4Mb. There is a hard-coded limit of 1Gb. If the specified value is out of range, not changes are made to the existing jlog.
Specify a precommit buffer size in bytes. The default is 0 and the maximum is 8Mb. If the specified value is too large, 8Mb is used.
Built-in logging facilities
libmtev uses its own logging, so applications have ample error and debugging
information exposed out of the box. There are three four built-in logging
facilities that serve as the base for most others:
debug. By default,
debug is disabled and both
The following log streams are used within libmtev:
Generic debug logging, by default all
debug/* logs flow through here as an outlet.
Debugging output from the amqp module.
Debugging output from mtev clustering.
Debugging output from the configuration system.
Debugging output from the consul module.
Debugging output from curl operations in the consul module.
Debugging information from the internal dwarf analyzer.
Debugging information from the eventer subsystem.
Debugging information from the fq module.
Debugging information from the http service framework.
Debugging information from the http/2 service framework.
Debugging information from the http_observer module.
Debugging information from the listener subsystem.
Debugging information from the memory subsystem, specfically around safe memory reclamation.
Debugging information from the rest subsystem (sitting atop the http and http/2 service frameworks).
Debugging information from the reverse connection subsystem.
Debugging information from the time subsystem, particularly around timings and thread affinity.
Debugging information from the websocker integration atop the http service framework.
Generic debug capture from the libxml2 framework.
Debugging information from the zipkin_fq module.
Debugging information from the zipkin_jaeger module.
Generic error logging, by default all
error/* logs flow through here as an outlet.
Error logging for the amqp module.
Error logging for clustering operations.
Error logging for the configuration subsystem.
Error logging for the consul module.
Error logging for the eventer subsystem.
Error logging for the fq module.
Error logging for the http_observer module.
Error logging for the listener subsystem.
Error logging for the rest subsystem (sitting atop the http and http/2 service frameworks).
Error logging for the reverse connection subsystem.
Error logging for the websocker integration atop the http service framework.
Error logging for the zipkin_jaeger module.
HTTP access logs (close to the Apache2 common log format).
Debugging information for I/O performed in the http and http/2 service frameworks.
A logging facility that writes to file descriptor 2.