It's not obvious to me that the path C is a closed loop, so you'll have to compute either 3 or 4 line integrals for part (a). I'll assume that it is indeed closed.
(a) Let C = C₁ U C₂ U C₃ (and possibly taking the union of C₄ as well). Parameterize each side of the rectangle by
C₁ : x = t, y = 0, with 0 ≤ t ≤ 5
C₂ : x = 5, y = t, with 0 ≤ t ≤ 1
C₃ : x = 5 - t, y = 1, with 0 ≤ t ≤ 5
C₄ : x = 0, y = 1 - t, with 0 ≤ t ≤ 1
The integral along the whole path C is then the sum of the integrals over each component path:
[tex]\displaystyle \int_C xy \, dx + x^2 \, dy = \sum_{i=1}^4 \int_{C_i} xy \, dx + x^2 \, dy[/tex]
Over the first path, we have the differentials
x = t ⇒ dx = dt
y = 0 ⇒ dy = 0 dt
so that
[tex]\displaystyle \int_{C_1} xy \, dx + x^2 \, dy = \int_0^5 t\cdot0\cdot dt + t^2 \cdot 0 \, dt = \int_0^5 0 \, dt = 0[/tex]
Over the second path,
x = 5 ⇒ dx = 0 dt
y = t ⇒ dy = dt
and the integral is
[tex]\displaystyle \int_{C_2} xy \, dx + x^2 \, dy = \int_0^1 5\cdot t\cdot 0\,dt + 5^2 \cdot dt = 25 \int_0^1 dt = 25[/tex]
Over the third path,
x = 5 - t ⇒ dx = - dt
y = 1 ⇒ dy = 0 dt
and the integral is
[tex]\displaystyle \int_{C_3} xy \, dx + x^2 \, dy = \int_0^5 (5-t)\cdot 1\cdot 0\,dt + (5-t)^2 \cdot (-dt) = -\int_0^5 (5-t)^2 \, dt = -\frac{25}2[/tex]
Over the fourth path,
x = 0 ⇒ dx = 0 dt
y = 1 - t ⇒ dy = -dt
and the integral is
[tex]\displaystyle \int_{C_3} xy \, dx + x^2 \, dy = \int_0^1 0\cdot (1-t)\cdot 0 \, dt + 0^2 \cdot (-dt) = \int_0^1 0 \, dt = 0[/tex]
Then the integral along all of C (including C₄ to close the loop) is
0 + 25 - 25/2 + 0 = 25/2
(b) We first observe that neither xy nor x² have any singularities over the rectangle or its boundary C, so Green's theorem applies. Denoting the rectangle by
R = {(x, y) : 0 ≤ x ≤ 5 and 0 ≤ y ≤ 1}
we have
[tex]\displaystyle \int_C xy \, dx + x^2 \, dy = \iint_R \frac{\partial(x^2)}{\partial x} - \frac{\partial(xy)}{\partial y} \, dx\, dy \\\\= \int_0^1 \int_0^5 x \, dx \, dy = \frac{25}2\int_0^1dy = \boxed{\frac{25}2}[/tex]
If the path C₄ *is not* to be considered, we can still use Green's theorem as above. In this case, neither answer changes, since the contribution of the integral over C₄ is 0. But if it were non-zero, and we did not want to include it, we would have
[tex]\displaystyle \int_C = \int_{C_1} + \int_{C_2} + \int_{C_3}[/tex]
and by Green's theorem,
[tex]\displaystyle \int_C = \iint_R - \int_{C_4}[/tex]
