Salt flows into the tank at a rate of
(5 lb/gal) * (6 gal/min) = 30 lb/min
The volume of solution in the tank after t min is
600 gal + (6 gal/min - 12 gal/min)*(t min) = 600 - 6t gal
which means salt flows out at a rate of
(A(t)/(600 - 6t) lb/gal) * (12 gal/min) = 2 A(t)/(100 - t) lb/min
Then the net rate of change of the salt content is modeled by the linear differential equation,
[tex]A'(t)=30-\dfrac{2A(t)}{100-t}[/tex]
Solve for A:
[tex]A'+\dfrac{2A}{100-t}=30[/tex]
Multiply both sides by the integrating factor, [tex]\frac1{(100-t)^2}[/tex]:
[tex]\dfrac{A'}{(100-t)^2}+\dfrac{2A}{(100-t)^3}=\dfrac{30}{(100-t)^2}[/tex]
[tex]\left(\dfrac A{(100-t)^2}\right)'=\dfrac{30}{(100-t)^2}[/tex]
Integrate both sides:
[tex]\dfrac A{(100-t)^2}=\dfrac{30}{100-t}+C[/tex]
[tex]\implies A(t)=30(100-t)+C(100-t)^2[/tex]
The tank starts with no salt, so A(0) = 0 lb. This means
[tex]0=30(100)+C(100)^2\implies C=-\dfrac3{10}[/tex]
and the particular solution to the ODE is
[tex]A(t)=30(100-t)-\dfrac3{10}(100-t)^2=\dfrac3{10}t(100-t)[/tex]