GraphDiff-IK: https://arxiv.org/abs/2606.00086

Introduction

• Structure-aware graph diffusion framework for IK
• Robot = kinematic graph constructed from URDF
  • Nodes = actuated joints
  • Edges = kinematic dependencies
• IK formulated as conditional graph diffusion that directly generates joint configurations on the robot graph
• Architecture choices:
  • "Hierarchical stage-wise message passing"
  • Torso-aware conditioning
• Training strategies:
  • Noisy FK feedback
  • Task-space supervision
• Architecture is generalizable to many robot embodiments (single-arm, dual-arm, robots with torso/waist structures, ...)
  • They test on Galaxea R1 Pro, Franka Emika Panda, Unitree G1, UR10, AgileX Piper

Results

Below is a selected subset, see Tables II and III in the paper for full results.

• Evaluation metrics:
  • End-effector position (Euclidean distance; mm) + orientation error (angular distance; degrees)
  • Reported over 100 randomly sampled target poses
  • For dual-arm robotic systems, metrics are reported separately for left and right arms
• Implementation details:
  • Inference: 100 denoising timesteps; DDIM scheduler
  • Model dimensions: $d_\text{hidden} = 512, d_\text{time} = 256$
• Franka Emika Panda: 7-DoF fixed base
  • Position error: 3.47 ± 2.47 mm (mean ± std)
  • Rotation error: 0.87 ± 0.69 deg
• Unitree G1: 14-DoF dual arm with waist
  • Position error: 0.82 ± 0.68 mm
  • Rotation error: 0.33 ± 0.24 deg
• Galaxea R1 Pro: 18-DoF dual arm with torso
  • Position error: 5.99 ± 4.48 mm
  • Rotation error: 0.62 ± 0.40 deg

Next steps

• Decrease inference cost of diffusion (increase inference efficiency)
• Current method does not consider collision avoidance, dynamic constraints, or temporal motion consistency.

Methods

• Data generation: generate IK dataset using FK $$q_i \sim \mathcal{U}(q_\text{min}, q_\text{max})$$
  • Sample joint configurations $q$ uniformly within joint limits defined by URDF → $\text{FK}(q) = y$ to generate data pairs $(q, y)$
  • For robots with multiple kinematic branches (e.g. dual-arm systems, humanoids), FK is computed independently for each end-effector: $$y_i = \left\{\left(p_i^{(k)}, r_i^{(k)}\right)\right\}^K_{k=1}$$where $K$ = number of end-effectors, $p^{(k)}_i$ and $r_i^{(k)}$ represent the position and orientation of the $k$-th end-effector.
  • Takes "a few minutes" to generate datasets on the scale of $10^6$ samples
• Diffusion training: learn the distribution $p(q \mid y)$ using a graph diffusion model
  • Forward: noise joint configurations $q_0 \rightarrow q_T \sim \mathcal{N}(0, I)$
  • Reverse: model learns to denoise $q_T \rightarrow q_0$
  • Architecture: "structure-aware GCN with staged message passing"
  • Objectives: diffusion loss + FK-based task-space supervision
• Inference: $z \sim \mathcal{N}(0, I) \xrightarrow{f_\theta} x^\star \sim p^\star (q \mid y)$, diffusion iterative denoising

Graph representation of robot kinematics

$$G = (V, E)$$where each node $v_i \in V$ = actuated joint & each edge $(i, j) \in E$ = kinematic dependency between two connected joints.

• 💡 TL;DR:
  • Node: what kind of joint am I + what is my current state?
  • Edge: how am I physically attached to my neighboring joint?

Actuated joint = a connection point in a mechanical system that is actively driven to move by a power source, such as an electric, hydraulic, or pneumatic motor.

• Edges connect adjacent actuate joints, and stores the rigid transform between them as an edge feature
  • Fixed joints in between are collapsed into that transform rather than becoming graph nodes themselves.
• 🔖 Example: for a Franka Panda, the physical chain is approx: $$\text{base} \rightarrow J_1 \rightarrow J_2 \rightarrow \cdots \rightarrow J_6 \rightarrow J_7 \rightarrow \text{hand}$$So GraphDiff-IK's graph would have each $J_i$ as a graph node and graph edges $E$: $$E = \{(J_1, J_2), (J_2, J_3), (J_3, J_4), \ldots , (J_6, J_7)\}$$with edge features like: $$e_{34} = \begin{bmatrix} t_{34} & r_{34} & a_4 \end{bmatrix}$$where $t_{34}$ and $r_{34}$ are the relative translation and rotation locating joint 4 w.r.t. joint 3, and $a_4$ is joint 4's rotation axis.
• For a dual-arm robot, we might expect a graph like:

                    ┌── left_shoulder ── left_elbow ── left_wrist
waist ── torso ─────┤
                    └── right_shoulder ─ right_elbow ─ right_wrist

Node representation

For each node $v_i$, the node feature is defined as: $$x_i = [q_i \,\, || \,\, a_i \,\, || \,\, l_i \,\, || \,\, d_i ] \in \mathbb{R}^{d_x}$$where $q_i$ = joint angle, $a_i$ = joint axis, $l_i$ = joint limits, $d_i$ = structural information associated with the joint.

• Joint axis: specifies the direction about which/along which that joint moves
  • e.g. for a revolute joint, the axis is a 3D unit vector expressed in the joint's local frame: $$a_i = \begin{bmatrix} a_x \\ a_y \\ a_z \end{bmatrix}$$For example, a hinge that rotates around the vertical $z$ direction has joint axis $[0, 0, 1]$.
  • A prismatic joint that translates along $x$-axis has joint axis $[1, 0, 0]$
• Structural encoding: "structural info associated w/ the joint"
  • 💡 Captures metadata - "where does this joint live in the robot's kinematic tree?"
  • 3 components:
    1. Joint depth = how many levels away from the root of the kinematic tree is this joint? (e.g. $J_1$ in Panda has depth 0)
       • Kind of like positional encoding in a transformer - token count + where token occurs → joint properties + where joint occurs in robot
    2. Branch embedding: distinguishes different kinematic branches
       • e.g. a left elbow and right elbow might have similar depth, joint limits, and joint axes, but they're not interchangeable → so the network gets something like $b_i \in \{\text{torso}, \text{left arm}, \text{right arm}\}$
    3. (For multi-branch systems only) Node-type embedding: embeds "left arm" vs. "torso" vs. "right arm"
       • Paper doesn't make it too clear how this is different from the branch embedding

Edge representation

For each edge connecting a parent joint to a child joint, the edge feature is defined as: $$e_{ij} = [t_{ij} \,\, || \,\, r_{ij} \,\, || \,\, a_j ] \in \mathbb{R}^{d_e}$$where $t_{ij}$ and $r_{ij}$ denote the relative translation and rotation of the child joint w.r.t. the parent joint, and $a_j$ = joint axis of the child joint.

Graph diffusion for IK

At diffusion step $t$, the robot state is represented as a graph: $$G_t = (X_t, E), \quad X_t = [q_t^{(1)}, q_t^{(2)}, \ldots, q_t^{(N)}]$$where $X_t$ = noisy node features at step $T$ and $E$ = fixed graph topology defined by robot kinematic structure (#Edge representation).

• 💡 Noising / denoising is only applied to the node features corresponding to joint states, while the graph connectivity remains unchanged throughout the denoising process.

Forward process

Iteratively noise a clean joint configuration $q_0$: $$q_t = \sqrt{\bar{\alpha}_t} q_0 + \sqrt{1 - \bar{\alpha}_t} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$s.t. $q_T \sim \mathcal{N}(0, I)$.

Reverse process

Denoising model $\epsilon_\theta (G_t, c, t)$ where $G_t$ is the noisy graph at diffusion step $t$, $t$ is the diffusion timestep, and $c$ denotes conditioning information.

Conditioning information

$$c = \{p, r, \hat{p}_t, \hat{r}_t\}$$where $p$ and $r$ = target EE position and orientation; $\hat{p}_t$ and $\hat{r}_t$ = noisy EE observation obtained via applying FK to the noisy joint configuration: $$(\hat{p}_t, \hat{r}_t) = \text{FK}(q_t)$$For multi-branch systems, the conditioning extends to include branch-specific target poses and noisy FK observations for each EE.

Structure-aware graph convolution

Flat network: $$[q_{1, t}, q_{2, t}, \ldots, q_{N, t}] \rightarrow \text{transformer/MLP} \rightarrow [\hat{\epsilon}_1, \ldots, \hat{\epsilon}_N]$$GraphDiff-IK uses "structure-aware message passing" instead: each joint updates its hidden representation by receiving messages from neighboring joints.

Paper claim: Modeling robot kinematic structural dependencies as a graph achieves better performance than using a flat vector $[q_{1}, \ldots, q_t]$.

Transformer-based graph convolution

Core idea: Transformer attention, restricted to the robot graph.

• For a normal transformer, token $i$ can attend to every token $j$. Here, joint $i$ only attends to its graph neighbors $j \in N(i)$ and the attention uses the physical relation between the two joints through the edge feature $e_{ij}$.
• Node update: $$h_i^{(l+1)} = W_1 h_i^{(l)} + \sum_{j \in N(i)} \alpha_{ij} \left(W_2 h_j^{(l)} + W_e e_{ij}\right)$$where $h_i^{(l)}$ is the feature of node $i$ at layer $l$, $N(i)$ = neighborhood of node $i$, $e_{ij}$ = edge feature between nodes $i$ and $j$. Attention coefficients are computed as: $$\alpha_{ij}^{(l)} = \text{softmax}_j \left(\frac{(W_q h_i^{(l)})^\top (W_k h_j^{(l)} + W_e e_{ij})}{\sqrt{d}}\right)$$where $W_q$ and $W_k$ are learnable projection matrices and $d$ denotes feature dimension.
  • $h_i^{(l)}$ is the model's internal representation of joint $i$ at layer $l$. At layer 0 (i.e. input to the neural network), we have $x_i = [q_i || a_i ||l_i||d_i]$: after the network starts processing that node, it gets transformed into some hidden vector $h_i^{(l)} \in \mathbb{R}^d$.
  • At layer $l+1$, the goal is to produce a better representation $h_i^{(l+1)}$. Conceptually, we have: $$\underbrace{h_i^{(l+1)}}_\text{new me} = W_1 \underbrace{h_i^{(l)}}_\text{old me} + \underbrace{\sum_{j \in N(i)} \alpha_{ij} \left(W_2 h_j^{(l)} + W_e e_{ij}\right)}_\text{messages from my neighbors}$$
  • The attention coefficients $\alpha_{ij}$ answer, "who should I listen to?" -- a query from node $i$ vs. key from node $j$ + edge geometry.
    • Query: $Q_i = W_q h_i$
      • "What kind of information does joint $i$ currently need?"
    • Key: $K_{ij} = W_k h_j + W_e e_{ij}$
      • "What does joint $j$ contain ($W_k h_j$) in the context of its physical relationship to $i$ ($W_e e_{ij}$)?"
    • 💡 If $Q_i^\top K_{ij}$ is large (query and key vectors align strongly) -- the information represented by neighbor $j$ "looks relevant" to what node $i$ needs right now.
• 🔑 This allows the model to "adaptively aggregate neighboring information" based on node features and local kinematic relationships (encoded in edge attributes)

Difference between vanilla transformer and this graph-based transformer: A vanilla transformer on $[q_1, \ldots, q_7]$ might allow $J_1 \leftrightarrow J_7$ directly, i.e. every joint can attend to every joint. Graph TransformerConv instead imposes $j \in \mathcal{N}(i)$, s.t. $J_1 \leftrightarrow J_2 \leftrightarrow J_3 \leftrightarrow \cdots \leftrightarrow J_7$ rather than $J_1 \leftrightarrow \text{everybody}$, $J_2 \leftrightarrow \text{everbody}$, etc. It also explicitly injects $e_{ij}$ to tell the attention mechanism the geometry of each connection.

• Vanilla transformer: treats every pair of tokens as equally directly related
• Graph TransformerConv: communicate first along actual physical kinematic connections

Conditional feature modulation

Task-space conditioning via FiLM. This section specifies how conditioning is injected, (#Conditioning mechanism) specifies what the conditioning actually contains, and where each part is used.

Given a conditioning vector $c$, a MLP predicts scaling and bias parameters: $$[\gamma, \beta] = \text{MLP}(c)$$and the node features are normalized then modulated as$$\tilde{h}_i^{(l)} = \gamma \odot \text{Norm} \left(h_i^{(l)}\right) + \beta$$where $\odot$ denotes element-wise multiplication.

• 🔑 FiLM enables the graph convolution process to adaptively adjust feature propagation according to the diffusion timestep, target EE pose, and noisy FK observations.
• 🔑 FiLM is how conditioning is generally done in this model: central conditioning pipeline is - $$\text{condition construction} \rightarrow c \rightarrow \text{FiLM} \rightarrow \text{modulated graph features}$$

Stage-wise graph convolution

Stage-wise framework to explicitly model hierarchical dependencies in multi-branch robotic systems: instead of doing message passing uniformly over the entire graph, GraphDiff-IK decomposes the robot graph into structure-aware subgraphs: $$G_\text{torso} \subseteq G, \quad G_\text{arms} \subseteq G$$where $G_\text{torso}$ contains torso-related joints and $G_|text{arms}$ contains arm-related branches.

1. Stage 1: Torso modeling
   • Message passing is performed only on the torso subgraph: $$H_\text{torso} = \text{GNN}(G_\text{torso})$$where $H_\text{torso}$ denotes the torso node features after graph convolution (i.e. the collection of updated hidden features for all torso joints).
   • The torso feature are then aggregated into a shared latent representation: $$z_\text{torso} = f_\text{proj}(H_\text{torso})$$ where $f_\text{proj}(\cdot)$ denotes a learnable projection function.
     • 💡 Conceptually $z_\text{torso}$ is a compact learned summary of the shared torso context (e.g. might encode things like: current noisy torso configuration, which way the torso is oriented, information about the targets that have been injected into the torso features, ...)
2. Stage 2: Branch-aware arm reasoning
   • Message passing is performed on arm-related subgraphs, conditioned on the shared torso latent representation $z_\text{torso}$
   • Each EE arm is processed independently: e.g. for dual-arm setup, the left-arm and right-arm branches have branch-specific conditioning functions: $$\begin{align*} \tilde{h}_{i, \text{left}} &= \text{FiLM} (h_i, c_\text{left}, z_\text{torso}), \\ \tilde{h}_{i, \text{right}} &= \text{FiLM} (h_i, c_\text{right}, z_\text{torso})\end{align*}$$where $c_\text{left}$ and $c_\text{right}$ denote branch-specific conditioning information.
   • Branch-specific conditioning information: timestep, this EE's target pose, noisy FK feedback
   • ❓ Why not let the left arm directly reason about the entire right arm?
     • GraphDiff-IK's inductive bias - most detailed reasoning is local.
     • So they compress shared information through the torso latent instead: $$\text{right-side/global influence} \rightarrow z_\text{torso} \rightarrow \text{left arm}$$This embeds the architectural hint that - the arms are distinct branches, but they coordinate through a shared upstream body.
3. Stage 3: Global refinement
   • Perform message passing over the full graph: $$H_\text{refine} = \text{GNN}(G)$$This lets information flow across the complete robot so that final representations can reconcile cross-branch interactions.

• 🔑 Core idea: local specialization first, global reconciliation last
  • ❓ Why isn't Stage 3 alone sufficient - i.e. running $\text{GNN}(G)$ over the entire robot every layer?
    • This paper's central argument is that inductive bias improves model performance on the IK task:
      • Uniform graph message passing says - all edges are graph edges; learn whatever structure matters.
      • GraphDiff-IK's staged architecture says - we already know that torso joints are shared upstream variables; arms are downstream branches → let's force computation to respect that hierarchy.

TL;DR: First reason about the shared body, then let each limb solve its own task given that body, then check everything together.

Conditioning mechanism

Hierarchical conditioning mechanism that incorporates both task-level objectives & state-dependent geometric feedback.

Conditioning consists of 3 components:

1. Diffusion timestep encoding $c_t$ - "where am I in the diffusion process?"
2. Target end-effector pose encoding $c_\text{pose}$ - "where should the hand end up?"
3. Noisy FK feedback $c_\text{FK}$ - "where would the current noisy joints put the hand right now?"

These conditioning signals are encoded into latent representations and injected into the graph diffusion network through the FiLM-based feature modulation mechanism described in (#Conditional feature modulation).

Timestep encoding

Sinusoidal positional embedding: $$c_t = \phi_t(t)$$Provides the network with global information regarding the current diffusion stage.

Target pose encoding

$$c_\text{pose} = [\phi_p (P) \,\,||\,\, \phi_r(r)], \quad \text{position } p \in \mathbb{R}^3, \,\, \text{orientation } r \in \mathbb{R}^6$$For multi-branch robots, branch-specific target pose conditions are constructed independently for each EE.

Noisy FK feedback

Intuition: telling the network "move the hand to $x = 0.7$" merely informs it of the goal → but if we don't tell the network where the hand currently is, it doesn't know which way to move.

• Target gives $y_\text{target}$
• Current noisy config gives $q_t$
• Run FK on $q_t$: $\hat{y}_t = \text{FK}(q_t) = (\hat{p}_t, \hat{u}_t)$ = current noisy EE position and EE orientation
• Encode: $$c_\text{FK} = [\phi_{\hat{p}} (\hat{p}_t) \,\, ||\,\, \phi_{\hat{u}} (\hat{u}_t)]$$

This is what the paper means by "state-dependent geometric feedback" - the condition depends on the current state $q_t$, not merely the task.

Multi-arm case

If there are two end effectors $y_L$ and $y_R$, construct branch-specific conditions: $$\begin{align*}c_L &= [c_t \,\,||\,\,c_\text{pose}^L \,\,||\,\, c_\text{FK}^L] \\ c_R &= [c_t \,\,||\,\,c_\text{pose}^R \,\,||\,\, c_\text{FK}^R]\end{align*}$$i.e.,

• left arm condition = current diffusion stage + left hand's desired pose + left hand's current noisy pose
• right arm condition = current diffusion stage + right hand's desired pose + right hand's current noisy pose

Hierarchical conditioning formulation

• Global latent representation extracted from the torso subgraph: $$z_\text{torso} = f_\text{proj}(H_\text{torso})$$
• Augment each local condition (global + local - hence "hierarchical")$$\tilde{c}^L = [c^L \,\,||\,\,z_\text{torso}, \quad \tilde{c}^R = [c^R \,\,||\,\, z_\text{torso}]$$

Stage-wise conditioning usage

Conditioning mechanism is injected hierarchically throughout the structure-aware graph reasoning process - i.e. global and local conditioning enter at different stages in (#Stage-wise graph convolution).

1. Stage 1: global conditioning
2. Stage 2: branch-specific conditioning vectors
3. Stage 3: global conditioning + local info → jointly integrated in full-graph refinement

Training objective

Losses used (summed at optimization):

• Diffusion loss: noise prediction loss $$\mathcal{L}_\text{noise} = \mathbb{E}_{t, q_0, \epsilon} [ ||\epsilon - \hat{\epsilon}||^2]$$
• FK loss: use predicted noise to reconstruct $\hat{q}_0$ to apply FK task-space supervision: $$q_t \xrightarrow{\text{network}} \hat{\epsilon} \rightarrow \hat{q}_0, \quad \hat{y}_0 = \text{FK}(\hat{q}_0)\quad \Rightarrow \mathcal{L}_\text{FK} = \mathcal{L}_{\text{pos}} + \lambda_\text{rot} \mathcal{L}_\text{rot} = ||\hat{p}_0 - p||^2 + \lambda_\text{rot}\mathcal{L}_\text{quat}(\hat{u}_0, u)$$

Multi-end-effector version: compute this independently for each branch; branch losses are summed during optimization. $$\mathcal{L}_\text{FK} = \mathcal{L}_\text{FK}^L + \mathcal{L}_\text{FK}^R$$ $$\mathcal{L}_\text{FK}^L = ||\hat{p}_L - p_L||^2 + \lambda_\text{rot} \mathcal{L}_\text{quat}(\hat{u}_L, u_L), \quad \quad \mathcal{L}_\text{FK}^R = ||\hat{p}_R - p_R||^2 + \lambda_\text{rot} \mathcal{L}_\text{quat}(\hat{u}_R, u_R)$$

Late-step FK supervision

FK loss is only active during the later stages of the denoising process, because joint configurations are too noisy s.t. FK observations are not geometrically meaningful at large $T$. So, they gate FK supervision by an indicator function of $t$: $$\mathbb{I}_\text{FK}(t) = \mathbb{I}(t < \tau_\text{FK})$$where $\tau_\text{FK}$ denotes the FK supervision threshold.

Combined training objective

$$\boxed{\mathcal{L} = \mathcal{L}_\text{noise} + \lambda_\text{FK} \mathbb{I}_\text{FK} (t) \mathcal{L}_\text{FK}}$$

Inference

Iterative denoising, DDIM 100 steps