The function compoly returns a pair $p\left(x\right)$, $x=q\left(x\right)$ such that subs(x=q(x), p(x)) is equal to r, the input polynomial. If such a pair cannot be found, it returns FAIL. $p\left(x\right)$ and $q\left(x\right)$ are nonlinear polynomials and $q\left(x\right)$ has a low-degree in x greater or equal to 1.

When compoly is called without additional arguments, it is equivalent to being called with a second argument which is indets(r). When the second argument is a set with more than one variable, then a multivariate composition is attempted. In this case, the result has the same form, but the second polynomial is multivariate, that is, $p\left(x\right)$, $x=q\left(x,...\right)$.  For the multivariate case, the second polynomial, $q\left(x\right)$, may be of degree 1.

Note that the composition may not be unique. In particular, if $p\left(x\right)$, $x=q\left(x,...\right)$ is a composition, then we can find another $p\left(x\right)$ such that replacing $q\left(x,...\right)$ by  $cq\left(x,\dots \right)+b$ will also result in a valid composition. This non-determinacy is eliminated by selecting c and b such that the q polynomial has integer content 1 and its independent term is 0.

$\mathrm{compoly}\left(x^6-9x^5+27x^4-27x^3-2x^{2}y+6xy+1\,x\right)$

$x^3-2xy+1,x=x^2-3x$

$\mathrm{compoly}\left(x^4-3x^3-x+5\,x\right)$

$\mathrm{FAIL}$

$\mathrm{compoly}\left(x^2+2xy-7x+y^2-7y+16\right)$

$x^2-7x+16,x=x+y$

$\mathrm{compoly}\left(x^4+4x^3{y}^3+6x^2{y}^6+4x{y}^9+y^{12}+x+y^3-1\,\left\{x\,y\right\}\right)$

$x^4+x-1,x=y^3+x$