LiDAR Callback Freezes: Audit ROS 2 Service Deadlocks

If a LiDAR subscription stops progressing when its callback makes a synchronous service request, investigate executor scheduling and callback-group ownership before blaming the sensor. The callback may be waiting for work that the same executor or mutually exclusive group cannot run until the waiting callback returns. Messages can continue arriving while application progress is blocked.

First prove that the intended publisher still produces data and that the affected subscriber received data before the request. The publish-rate diagnostic helps separate acquisition from processing. This article focuses on application execution after delivery, rather than on Ethernet packet reception or topic discovery.

The versioned Jazzy callback-group guide explains both explicit callbacks and callbacks created indirectly by service or action clients. Preserve the installed client-library revision and implementation language. An example written for one client library should not be pasted into another without examining its execution model.

Lidar and computer on a quiet laboratory debugging bench — Generated editorial illustration of software debugging; no actual callback timing is shown.

Table of Contents

Record the last boundary that still progresses

Capture the publisher identity, topic, message type, subscription configuration and the exact operation that precedes the freeze. Add bounded observations at callback entry, request submission, response availability and callback exit. Use an identified monotonic timing source for durations so wall-clock adjustments do not create misleading gaps.

Distinguish a callback that never starts from one that starts and never returns. Also distinguish a missing service response from a response that cannot be processed. These cases can look identical in a visualization, but require different evidence. Preserve the original failure before increasing timeouts or changing executor threads.

Check discovery and endpoint compatibility separately. The QoS checklist addresses transport-policy incompatibility. If a callback repeatedly enters until a particular request occurs, a persistent QoS mismatch is less consistent with that observation, although other delivery problems may coexist.

Draw the execution ownership map

List the node, executor type, thread configuration, callback groups and every entity assigned to each group. Include the LiDAR subscription, timers, service clients, service servers and any action clients used inside the processing path. Record the effective assignments, not only the groups the developer intended to create.

The Jazzy executor overview connects incoming work with executor scheduling. More threads provide an opportunity for concurrent execution; they do not remove callback-group restrictions. A multithreaded executor can still serialize relevant work when it is assigned to one mutually exclusive group.

Keep references to callback-group objects for the lifetime required by the client library. A temporary group declaration that is not retained can undermine the intended arrangement. Review construction and ownership alongside launch composition, because moving nodes into a container can change which executor actually serves them.

Observation	Question to answer	Useful evidence
Callback stops at a blocking request	What work must execute for it to finish?	Caller and completion callback ownership
More executor threads do not help	Are the relevant entities in one exclusive group?	Actual group assignments
Asynchronous request returns but data races appear	Which mutable state is now shared?	State ownership and synchronization review
Only composed deployment freezes	Which executor serves the component?	Container configuration and runtime identity

Trace the request’s completion work

For a synchronous operation, identify the condition that releases the wait. It may depend on a future being completed, a done callback, a response handler or work in another node. Draw a dependency arrow from the waiting callback to that work, then record which executor and group are allowed to run it.

The Jazzy rclpy client implementation is a useful reference for Python service-call behavior. Match the code path to the actual API used. A finite timeout can bound how long a caller waits, but it does not establish that the dependency graph is capable of completing successfully.

Inspect the service server independently. Confirm that the server receives the request and whether it generates a response. An unavailable server, a server blocked on its own dependency, and a local completion callback denied execution are different failure boundaries. Do not label every delayed service call a callback-group deadlock.

A reproducible test should make only the triggering request and preserve its inputs. Start with an idle or recorded-data setup where the application can be stopped safely. Avoid using physical robot motion as the first way to demonstrate a software scheduling failure.

Separate computer work areas prepared for comparing software execution paths — Generated illustration of a controlled comparison workspace; not a captured concurrency result.

Choose a repair that makes progress explicit

An asynchronous request can let the current callback return while another path handles completion. Design the continuation deliberately: decide who owns the request, where results are stored, how failures are reported and how late responses are rejected. Returning immediately is not enough if the application then waits elsewhere on the same blocked dependency.

The callback-group implementation distinguishes mutually exclusive and reentrant execution permissions. Separate groups or reentrant execution may permit the needed work, but they also permit new interleavings. Choose them because the ownership map requires them, not because a tutorial describes more concurrency as universally faster.

Audit shared state before enabling parallel callbacks. Include point buffers, configuration snapshots, request counters, device handles and shutdown flags. A sensor SDK may have its own thread-safety requirements. Do not assume a client-library group setting makes an unrelated SDK or a reused data buffer safe for concurrent access.

Keep long work out of the acquisition callback when the architecture supports a bounded handoff. Define the handoff capacity and overflow behavior. An unbounded queue can turn a freeze into growing latency or memory use. Record what the application should discard, preserve or reject under load.

Test the corrected dependency graph

Repeat the original failing request with the same software revision and data. Record callback entry and exit, response processing, pending work and output timestamps. Confirm progress over repeated requests, not only one successful response after startup. Include a deliberately unavailable service so the error path is exercised.

The versioned executor implementation helps explain which work is scheduled in the Python deployment. Use it to interpret observations, rather than to promise a fixed callback latency. Scheduling behavior and application execution time depend on the deployed workload and platform.

Check cancellation and shutdown while a request is pending. Identify what happens to callbacks that hold references to a node being stopped, and what cleanup waits for outstanding work. The restart-recovery checklist is relevant once the execution repair changes startup or shutdown behavior.

When the failure occurs only with several sensor streams, separate executor blocking from timestamp matching. The synchronizer diagnostic addresses message pairing. A synchronizer waiting for a suitable timestamp pair is not automatically blocked by a service request; preserve evidence of which callback is actually running.

Package the minimal reproducer, ownership map, versions, logs and the failing and corrected observations in a support bundle. State the exact repair and its concurrency assumptions. A useful handoff makes both progress and shared-state responsibility visible to the next maintainer.