Mot-clé - gstreamer

Fil des billets - Fil des commentaires

mercredi, avril 24 2019

GStreamer buffer flow analyzer

Gstreamer's logging system is an incredibly powerful ally when debugging but it can sometimes be a bit daunting to dig through the massive amount of generated logs. I often find myself writing small scripts processing gst logs when debugging. Their goal is generally to automatically extract some specific information or metrics from the generated logs. Such scripts are usually quickly written and quickly disposed once I'm done with my debugging but I've been wondering how I could make them easier to write and to re-use.

gst-log-parser is an attempt to solve these two problems by providing a library parsing GStreamer logs and enabling users to easily build such tools. It's written in Rust and is shipped with a few tools that I wrote to track actual bugs in GStreamer elements and applications.

One of those tool is a buffer flow analyzer which can be used to provide various information regarding the buffers exchanged through your pipeline. It relies on logs generated by the upstream stats tracer, so no modification in GStreamer core or in plugins is required.

First step is to generate the logs, this is easily done by defining these env variables: GST_DEBUG="GST_TRACER:7" GST_DEBUG_FILE=gst.log GST_TRACERS=stats

We can then use flow for example to detect decreasing pts or dts:

cargo run --release --bin flow gst.log check-decreasing-pts

Decreasing pts tsdemux0:audio_0_0042 00:00:02.550852023 < 00:00:02.555653845

Or to detect gaps of at least 100ms in the buffers flow:

cargo run --release --bin flow gst.log gap 100

gap from udpsrc0:src : 00:00:00.100142318 since previous buffer (received: 00:00:02.924532910 previous: 00:00:02.824390592)

It can also be used to draw graphs of the pts/dts produced by each src pad over time:

cargo run --release --bin flow gst.log plot-pts

gst-flow-pts.png


These are just a few examples of the kind of information we can extract from stats logs. I'll likely add more tools in the future and I'm happy to hear suggestions about other features that would make your life easier when debugging GStreamer.

lundi, juin 20 2016

GStreamer leaks tracer

Here at Collabora we are pretty interested at improving QA tools in GStreamer. Thibault is for example doing a great job on gst-validate ensuring that a lot of code paths are regularly tested using real life scenarios. Last year I added Valgrind support to gst-validate allowing us to automatically detect memory leaks in test scenarios. My goal was to integrate this as part of GStreamer's automatic QA to prevent memory leak regressions. While this can sometimes be a good approach to track leaks it has a few downsides:

  • Valgrind can be very CPU and/or memory consuming which can be a problem with longer scenarios or on limited hardware such as embedded devices.
  • As a result running the full tests suite with valgrind can take ages.
  • Valgrind checks for any potential memory leak which can lead to a lot of false positives or leaks in low level system libraries on which we have few control. We usually work around this problem using suppression files but they are generally very fragile and depend a lot on the system/distribution which has been used for testing.

I tried to solve these issues by trying a new approach using GstTracer. Tracers are a new mechanism introduced in GStreamer 1.8 allowing tools to hook into GStreamer internals and collect data. So I started by adding tracer hooks when GstObject and GstMiniObject are created and destroyed. Then I implemented a new tracer tracking the lifetime of (mini)objects and listing those which are still alive when the application is exiting. This worked pretty well but I needed a way to discard objects which are intentionally leaked (false positives). To do so I introduced a new (mini)object flag allowing us to mark such objects.

I'm pretty happy with the result, while proof testing this tool I found and fixed dozens of leaks into Gstreamer (core, plugins and tests). Some of those fixes have already reached the 1.8.2 release. It's also very easy to use and doesn't require any external tool unlike Valgrind (which can be tricky to integrate on some platforms).

To use it you just have to load the leaks tracer with your application and enable tracer logs:

GST_TRACERS="leaks" GST_DEBUG="GST_TRACER:7" gst-launch-1.0 videotestsrc num-buffers=10 ! fakesink

You can also filter out the types of GstObject or GstMiniObject tracked to reduce memory consumption:

GST_TRACERS="leaks(GstEvent,GstMessage)"  GST_DEBUG="GST_TRACER:7" gst-launch-1.0 videotestsrc num-buffers=10 ! fakesink

This tracer has recently be merged into GStreamer core and will be part of the 1.9.1 release.

As future enhancements I implemented live tracking and checkpointing support using signals like I already did in gobject-list a while ago. I'd also like to be able to display the creation stack trace of leaked objects to easily spot the leaked instances. Finally, I opened a bug to discuss the integration of the tracer with the QA system.