How a system goes from raw 3D measurements to a description it can reason about: which surface is the floor, what stands on it, where each thing sits relative to the others, and what stays where when it goes out of view. Every figure on this page computes its answer in front of you.
A depth sensor returns points. Nothing in that list says which points belong together, which surface is the ground, or how many objects there are. Scene understanding is the chain of estimates that turns the list into a description: a dominant plane, the clusters that rise from it, a shape and an extent for each cluster, the relations between them, and a map of which cells of space are filled.
Each link in that chain is a small, well-posed estimation problem with a known answer. The four figures below run those problems on a synthetic scene whose ground truth is known, so every readout can be checked. The same routines run on real sensor data; the difference is the noise, which is why the open problems at the end are about noise.
A plane is a unit normal and an offset. A point lies on it when
\hat{\mathbf{n}} \cdot \mathbf{p} + d = 0, \qquad \|\hat{\mathbf{n}}\| = 1and its signed distance from the plane is \hat{\mathbf{n}} \cdot \mathbf{p} + d. Three points fix a plane, so each RANSAC hypothesis draws three at random and counts the points within \tau. If a fraction w of the points are inliers, the probability that a triple is all inliers is w^3, and the number of draws needed to see at least one clean triple with confidence p is
N = \frac{\log(1 - p)}{\log(1 - w^3)}(Fischler & Bolles 1981). The figure reads w off its best hypothesis, evaluates this bound for p = 0.99, and stops after twice that many draws.
The winning triple is only three noisy points. The refinement fits all the inliers at once: with centroid \bar{\mathbf{p}} and scatter matrix
S = \sum_{i \in \text{inliers}} (\mathbf{p}_i - \bar{\mathbf{p}})(\mathbf{p}_i - \bar{\mathbf{p}})^{\mathsf T}the normal that minimises the sum of squared distances is the eigenvector of S with the smallest eigenvalue, and d = -\hat{\mathbf{n}} \cdot \bar{\mathbf{p}}. The figure computes it by Jacobi rotations on the 3 × 3 matrix. The remaining points are then linked whenever two lie within 0.28 units of each other, which is Euclidean clustering; each cluster is then labelled by two tests: a sphere when the residual of an algebraic sphere fit sits at the noise level, a cylinder when the radius of its side points about the footprint centre depends on neither azimuth nor height, and otherwise a box. A cluster that contains a horizontal patch with points resting on it is split at that patch first.
The state is position and velocity, \mathbf{x} = [p_x, p_y, v_x, v_y]^{\mathsf T}, and between frames it moves at constant velocity with an unknown acceleration of strength \sigma_a:
\hat{\mathbf{x}}^- = F\hat{\mathbf{x}}, \qquad P^- = F P F^{\mathsf T} + Q, \qquad F = \begin{bmatrix} I & \Delta t\, I \\ 0 & I \end{bmatrix}with Q the discrete white-noise acceleration covariance, whose position block grows as \sigma_a^2 \Delta t^4 / 4. This step runs whether or not a measurement arrives. Behind the panel it is the only step that runs, and P grows every frame; the ellipse in the figure is the 2σ contour of its position block.
When a measurement \mathbf{z} of the position arrives, the innovation and its covariance are
\mathbf{y} = \mathbf{z} - H\hat{\mathbf{x}}^-, \qquad S = H P^- H^{\mathsf T} + Rand the gain, state and covariance follow:
K = P^- H^{\mathsf T} S^{-1}, \quad \hat{\mathbf{x}} = \hat{\mathbf{x}}^- + K\mathbf{y}, \quad P = (I - KH)P^-(I - KH)^{\mathsf T} + K R K^{\mathsf T}The figure uses the Joseph form for P so it stays symmetric. The normalised innovation squared, \mathbf{y}^{\mathsf T} S^{-1} \mathbf{y}, is what a filter uses to judge itself: for a consistent filter it is chi-squared with two degrees of freedom, so it averages about 2. The first update after the panel is where it spikes.
| A | left of | B | gap 0.931 |
| B | in front of | A | gap 1.737 |
| C | in front of | A | gap 1.336 |
| C | left of | B | gap 0.867 |
| D | left of | B | gap 1.101 |
| B | in front of | D | gap 2.072 |
| C | in front of | D | gap 1.671 |
| D | on | A | gap 0.010 |
Relations need a frame, and the fitted floor supplies one. From \hat{\mathbf{n}} the figure builds two in-plane unit vectors and expresses every point as
u = \hat{\mathbf{u}} \cdot \mathbf{p}, \qquad v = \hat{\mathbf{v}} \cdot \mathbf{p}, \qquad h = \hat{\mathbf{n}} \cdot \mathbf{p} + dso h is height above the floor and each cluster gets an axis-aligned extent [u_{\min}, u_{\max}] \times [v_{\min}, v_{\max}] \times [h_{\min}, h_{\max}]. The scene is generated with a tilt, so this frame is not the world frame; the relations come out right because they are computed in the fitted one.
Every relation is a test on two extents. With footprints that overlap in both u and v,
\text{on}(A, B): \; |h^A_{\min} - h^B_{\max}| < 2\varepsilon, \qquad \text{above}(A, B): \; h^A_{\min} - h^B_{\max} \ge 2\varepsilonand with disjoint intervals along one axis,
\text{left of}(A, B): \; u^A_{\max} < u^B_{\min}, \qquad \text{in front of}(A, B): \; v^A_{\max} < v^B_{\min}with \varepsilon set to about twice the point noise. This is the Allen interval algebra applied per axis; the graph is the list of tests that pass.
The Amanatides–Woo traversal (1987) visits exactly the voxels a ray crosses, in order. For a ray \mathbf{o} + t\mathbf{d} it keeps, per axis, the parameter at which the ray next crosses a voxel boundary and the step between crossings:
t_{\max,x} = \frac{x_{\text{next}} - o_x}{d_x}, \qquad t_{\Delta x} = \left|\frac{\Delta}{d_x}\right|and at each step advances along whichever axis has the smallest t_{\max}. The figure casts 64 × 26 rays per pass and stops each at the first occupied voxel.
Each voxel keeps the log-odds of being occupied and adds a constant per observation:
l \leftarrow l + \begin{cases} l_{\text{occ}} & \text{ray ends here} \\ l_{\text{free}} & \text{ray passes through} \end{cases}, \qquad p = \frac{1}{1 + e^{-l}}with l_{\text{occ}} = 0.85, l_{\text{free}} = -0.4 and l clamped to ±4 so a cell can change its mind. This is the occupancy-grid update of Moravec and Elfes (1985) in the form OctoMap uses (Hornung et al. 2013). A voxel the sensor has never looked at stays at l = 0, p = 0.5, which is the honest state for unknown space.
R(\hat{\mathbf{k}}, \theta) = I + \sin\theta\, K + (1 - \cos\theta)\, K^2K is the cross-product matrix of the unit axis \hat{\mathbf{k}}. The scene tilt and the orbiting cameras in every figure are this formula.
T = \begin{bmatrix} R & \mathbf{t} \\ \mathbf{0}^{\mathsf T} & 1 \end{bmatrix} \in SE(3)Rotation and translation in one 4 × 4 matrix, so a chain of frames composes by multiplication. Sensor-to-world and plane-to-world are both of this form.
q = \cos\tfrac{\theta}{2} + \sin\tfrac{\theta}{2}\,(u_x \mathbf{i} + u_y \mathbf{j} + u_z \mathbf{k})Four numbers for a rotation, no gimbal lock, and interpolation along the shortest arc. The representation of choice when orientations are estimated over time.
DiagBuddy.AI, now ServiceLens, is the appliance-diagnosis product SOPHIA XT built and runs at diagbuddygo.com: field service and dispatch, with diagnostics grounded in a parts corpus we built rather than licensed.
A technician's photograph of an appliance interior is a scene of exactly the kind above: a few planar surfaces, boxes and cylinders, and components hidden behind others. The routines on this page, a dominant plane, clusters with shapes and extents, relations between them, and an estimate that survives an occlusion, are what a diagnostic model needs in order to read such a photograph as a layout rather than as pixels. That is the research track this page documents.
Real depth sensors have range-dependent, anisotropic error, dropouts on dark and specular surfaces, and flying pixels at edges. RANSAC's threshold \tau then has to vary with range, and the least-squares refinement needs a robust weighting. The figure's single \sigma is the simplest case.
Rooms have walls, tables and shelves. The fit has to be run sequentially, removing each plane's inliers before the next, and the relation frame has to be chosen from among them. Gravity from an inertial sensor settles which plane is the floor faster than geometry does.
The constant-velocity model is consistent until the target turns hard behind the panel. The standard answer is a bank of models with an interacting mixture (Blom & Bar-Shalom 1988); the price is several filters running at once.
A dense 0.125 unit grid over a room is a few hundred thousand voxels; a building at 5 cm is billions. Octrees with pruned uniform subtrees, the OctoMap design, keep the log-odds update while storing only what varies.