1.18.0

Added

• mar.mar_entry_max_count, which allows for setting the max number of MAR
  entries in your tmp directory. This will default to 1000.
• logs.max_buffered_lines, which sets the max number of lines that will be
  kept in memory before lines are dropped. Note that this is the same as
  fluent-bit.max_buffered_lines, but will apply to both the fluent-bit log
  source as well as the journald log source. If you have previously configured
  fluent-bit.max_buffered_lines that value will be used instead of the general
  config.
• Added the kind field to the MAR POST body. This is a field that is used by
  Memfault to differentiate the source of a MAR entry.
• Added logs.extra_attributes option to configure extra log attributes to keep
  in the log file (similar to
  fluent-bit.extra_attributes
  which still works but will be deprecated)

Changed

• The LogCollector has gone through a refactor to make the concurrency methods
  more consistent with the rest of the code base. This has also allowed us to
  make the following change.
• Moved recovery of logs in the logs directory into the LogCollector thread.
  This prevents a case where memfaultd could be slow to start when there is a
  large number of MAR entries on disk, and depending on systemd configuration,
  could be considered crashed and restarted before booting completely.

Fixed

• When passing a null to disable a feature there was a case where the config
  merging logic would fail. This would result in a case where a subset of
  configs were not possible. Fixed this logic to allow these valid use cases.

1.17.0

This release introduces the new memfaultctl write-metrics command
as well as some important bugfixes.

Added

• memfaultctl write-metrics, which allows users to write
  metrics to memfaultd from a script or shell by specifying
  them as arguments to the command in the form KEY=VALUE.
• 2 new built-in system metrics under the new diskstats
  metric category. These metrics follow the naming pattern
  diskstats/<device name>/reads_per_second and
  diskstats/<device name>/writes_per_second where
  <device name> is the name of a device listed in
  /proc/diskstats. The list of devices monitored
  can be configured with the metrics.system_metric_collection.diskstats
  configuration field.

Changed

• The log message emitted when the MAR cleaner
  encounters an invalid MAR entry in the MAR
  staging area has been lowered from WARN level
  to DEBUG level.
• A log message emitted when the configured
  high_resolution_telemetry.max_samples_per_minute
  rate limit is violated has been lowered from WARN
  level to DEBUG. This is to avoid repeatedly
  logging the same message at a high frequency
  when the system is sending more readings than
  permitted.

Fixed

• All deltas calculated based on a current and previous
  counter from procfs now take potential overflow into account.
• Fixed a sequencing issue with the cleaning of the MAR directory
  concerning an edge case where logs would fail to recover on
  devices that are near their disk space or inode quotas.
  This would result in stranded log files that would not
  get deleted or uploaded to Memfault until there is enough
  space to recover them.

memfaultd 1.16.1

This is a small release that adds coredump capture support for 32-bit ARM
targets using musl libc.

Fixed

• The coredump feature can now be compiled for the
  armv7-unknown-linux-musleabihf target.

memfaultd 1.16.0

This release includes two major new features:

• A new coredump capture mode to process the coredumps directly on device
  resulting in even smaller crash signature and guaranteeing that no user data
  is uploaded;
• High Resolution Telemetry for Linux, storing every single datapoint and
  enabling second-by-second data from Linux devices, for even deeper debugging
  capability.

Added

• Built-in wireless metric capture - wireless interfaces being monitored by the
  metrics.system_metric_collection.network_interfaces configuration will now
  have RSSI captured alongside the existing network metrics
• Added a boottime_duration_ms to Metric Reports that indicates the duration
  of the report independent of whether the CPU was running or not. This is
  intended to be used for debugging purposes for teams whose devices sleep often
  while in use.
• The new on-device stack unwinding option for capturing crash Traces, which can
  be enabled by setting coredump.capture_strategy.type to stacktrace in
  memfaultd.conf.
• High resolution telemetry, which is enabled by default and can be disabled or
  have its rate limiting configured with the metrics.high_resolution_telemetry
  field.
• Support for Yocto Scarthgap in meta-memfault. This support will be extended
  to meta-memfault-example in an upcoming release. Check out the scarthgap
  branch if you are using that version of Yocto!
• The built-in system memory metrics now include additional states such as
  Cached and Buffered.
• The metrics.statsd_server has had a new option, legacy_gauge_aggregation,
  added that averages Gauge metric readings rather than simply storing the
  most recent value. This is meant to allow for a smooth migration from
  collectd's StatsD server to memfaultd's. New integrations should
  use the h (Histogram) metric type for metrics whose readings should
  be aggregated via average.

Changed

• The default on-device rate limit for logs ingested by memfaultd has been
  increased from 500 lines per minute to 1000 lines per minute

Fixed

• Arbitrary ASCII strings between 1 and 128 in length can
  now be used in StatsD metric keys (besides the : delimiter),
  matching the behavior described in the docs.

Removed

Some built-in metrics were removed, as the naming conventions were not
consistent. All of these metrics were added in 1.15.0. The list can
be found below:

• cpu_usage_pct
• connectivity_recv_bytes
• connectivity_sent_bytes
• connectivity_<interface>_recv_bytes
• connectivity_<interface>_sent_bytes
• memory_used_pct
• memory_<process>_pct
• cpu_usage_<process>_pct
• storage_used_<disk>_pct

memfaultd 1.15.1

This is a patch release to fix a bug in our release process that caused the
Cargo lock file to get out of sync with our other published crates. This only
affected users building with --locked. No other changes were made.

Fixed

• Fixed a bug in our release process that allowed the Cargo lock file to get out
  of sync with our other published crates. This caused a build failure when
  building with --locked.

memfaultd 1.15.0

Added

• Add a new per-process operational_crashes metric. This is similar to the
  system wide operational_crashes metric, but is broken down by service. The
  metric will be named operational_crashes_<process name>, where <process name> is the name of the executable binary for the crashed process.
• Add a builtin log level extractor for all logs that are ingested by
  memfaultd. This allows you to define regex for each log level so that the
  level reported to Memfault matches the level inside the log message. This is
  useful for logs from systemd for example that do not always have a matching
  level in the metadata and the message.
• We've added a host of new builtin metrics:
  • cpu_usage_<process>_pct - Percent CPU usage of a specific process.
  • memory_<process>_pct - Percent memory usage of a specific process.
  • storage_used_<disk>_pct - Percent disk usage of a specific disk.
  • connectivity_<sent/recv>_bytes - Bytes sent and received over a network
    interface.
  • connectivity_<interface>_<sent/recv>_bytes - Bytes sent and received over
    a specific network interface.

Changed

• The builtin system metric configuration is now enabled by default. This means
  that memfaultd will now collect metrics on CPU, memory, disk usage, and much
  more out of the box. Note that this means
  some metrics previously reported via
  collectd will now have different keys.
  We hope this will make it more clear what each
  metric actually represents on a device -
  if this causes issues with your Project configuration
  please don't hesitate to reach out to us for
  support!
• memfaultd's internal statsd server is now enabled by default. This allows
  you to send custom metrics to memfaultd using the StatsD protocol. without
  having the first route them through collectd.
• We've made some changes to increase performance in our log processing
  pipeline. This should result in a lower CPU usage when processing logs.
• Previously a small number of log messages written by memfaultd did not have a level. We've
  updated them to now have a level matching the severity of each message.
• The journald log source is now the default log collector. This will use our
  internal processor for grabbing logs directly from the systemd journal. If you
  are not using systemd, you can still use the fluent-bit log source.

Fixed

• Fixed a bug where leading and trailing whitespace was being added to attribute
  names when using memfaultctl write-attributes.
• Previously memfaultd would panic if a MAR attachment was not an absolute
  path or was not a file. This has been changed to simply log an error and
  continue.
• Fixed a bug in which some metrics captured before enabling data
  collection were uploadedto Memfault once data collection was enabled.
  No data captured on devices for which data collection was always disabled
  would be uploaded as a result of this bug.

memfaultd 1.14.0

1.14.0

memfaultd 1.13.0

1.13.0