For the corresponding homogeneous ODE,
[tex]y'''-2y''+y'=0[/tex]
the characteristic equation is
[tex]r^3-2r^2+r=r(r-1)^2=0[/tex]
which admits the characteristic solution,
[tex]y_c=C_1+C_2e^x+C_3xe^x[/tex]
Assume a particular solution of the form
[tex]y_p=ax+bx^2e^x+ce^{5x}[/tex]
([tex]ax[/tex] because a constant solution is already accounted for by [tex]C_1[/tex]; [tex]x^2e^x[/tex] because both [tex]e^x[/tex] and [tex]xe^x[/tex] are accounted for)
[tex]\implies{y_p}'=a+(2x+x^2)be^x+5ce^{5x}[/tex]
[tex]\implies{y_p}''=(2+4x+x^2)be^x+25ce^{5x}[/tex]
[tex]\implies{y_p}'''=(6+6x+x^2)be^x+125ce^{5x}[/tex]
Substituting the derivatives of [tex]y_p[/tex] into the ODE gives
[tex]((6+6x+x^2)be^x+125ce^{5x})-2((2+4x+x^2)be^x+25ce^{5x})+(a+(2x+x^2)be^x+5ce^{5x})=2-24e^x+40e^{5x}[/tex]
[tex]a+2be^x+80ce^{5x}=2-24e^x+40e^{5x}[/tex]
[tex]\implies\begin{cases}a=2\\2b=-24\\80c=40\end{cases}\implies a=2,b=-12,c=\dfrac12[/tex]
So the particular solution is
[tex]y=C_1+C_2e^x+C_3xe^x+2x-12x^2e^x+\dfrac12e^{5x}[/tex]
With the given initial conditions, we find
[tex]y(0)=\dfrac12=C_1+C_2+\dfrac12\implies C_1+C_2=0[/tex]
[tex]y'(0)=\dfrac52=C_2+C_3+2+\dfrac52\implies C_2+C_3=-2[/tex]
[tex]y''(0)=-\dfrac{11}2=C_2+2C_3-6+\dfrac{25}2\implies C_2+2C_3=-12[/tex]
[tex]\implies C_1=10,C_2=-10,C_3=8[/tex]
and so
[tex]\boxed{y(x)=10-10e^x+8xe^x+2x-12x^2e^x+\dfrac12e^{5x}}[/tex]
