It might be possible to just initialize the recursions using formulas like these. In particular, the DLMF has
$$P^{\mu}_{\nu} \left(x\right) = \left(\frac{x+1}{x-1}\right)^{\mu/2} \mathbf{F} \left(\nu+1,-\nu;1-\mu;\tfrac{1}{2}-\tfrac{1}{2}x\right), $$
where $\mathbf{F}$ is "Olver’s scaled hypergeometric function: $F\left(a,b;c;z\right) / \Gamma\left(c\right)$", and I guess $F$ means ${}_2F_1$. (Also note that $\mathsf{P}$ and $P$ mean different things in the DLMF; it looks like $\mathsf{P}_\nu^\mu = (-1)^{\mu/2}P_\nu^\mu$.) Wikipedia claims that these forms obey all the same recursion formulas as for the integer arguments.
The harder problem is that the (interesting) $\mathfrak{D}^{(\ell)}_{m',m}$ matrices have $m',m \in -\ell, -\ell+1, \ldots, \ell$ — odd half integers, never including whole integers — which means that $m'=0$ is not a useful value, which means we can't immediately use $P_{\ell,m}$ to initialize the recursions. However, Choi et al. show how to derive recursions for $\mathfrak{D}$ in terms of recursions for the $Y_{\ell,m}$. Their Eq. (5.11) is the crucial idea; to extend this to half-integer values, we would have to generalize the integral implicit in that equation from extending over the sphere to extending over Spin(3).
Maybe, with a little luck, we would get recursions that look like those we already have for $H$. Maybe we need to use general Jacobi polynomials and their recursions.
