Given line equation of 2-order:
$$3 x^{2} + 10 x y + 3 y^{2} - 2 x - 14 y - 13 = 0$$
This equation looks like:
$$a_{11} x^{2} + 2 a_{12} x y + a_{22} y^{2} + 2 a_{13} x + 2 a_{23} y + a_{33} = 0$$
where
$$a_{11} = 3$$
$$a_{12} = 5$$
$$a_{13} = -1$$
$$a_{22} = 3$$
$$a_{23} = -7$$
$$a_{33} = -13$$
To calculate the determinant
$$\Delta = \left|\begin{matrix}a_{11} & a_{12}\\a_{12} & a_{22}\end{matrix}\right|$$
or, substitute
$$\Delta = \left|\begin{matrix}3 & 5\\5 & 3\end{matrix}\right|$$
$$\Delta = -16$$
Because
$$\Delta$$
is not equal to 0, then
find the center of the canonical coordinate system. To do it, solve the system of equations
$$a_{11} x_{0} + a_{12} y_{0} + a_{13} = 0$$
$$a_{12} x_{0} + a_{22} y_{0} + a_{23} = 0$$
substitute coefficients
$$3 x_{0} + 5 y_{0} - 1 = 0$$
$$5 x_{0} + 3 y_{0} - 7 = 0$$
then
$$x_{0} = 2$$
$$y_{0} = -1$$
Thus, we have the equation in the coordinate system O'x'y'
$$a_{11} x'^{2} + 2 a_{12} x' y' + a_{22} y'^{2} + a'_{33} = 0$$
where
$$a'_{33} = a_{13} x_{0} + a_{23} y_{0} + a_{33}$$
or
$$a'_{33} = - x_{0} - 7 y_{0} - 13$$
$$a'_{33} = -8$$
then The equation is transformed to
$$3 x'^{2} + 10 x' y' + 3 y'^{2} - 8 = 0$$
Rotate the resulting coordinate system by an angle φ
$$x' = \tilde x \cos{\left(\phi \right)} - \tilde y \sin{\left(\phi \right)}$$
$$y' = \tilde x \sin{\left(\phi \right)} + \tilde y \cos{\left(\phi \right)}$$
φ - determined from the formula
$$\cot{\left(2 \phi \right)} = \frac{a_{11} - a_{22}}{2 a_{12}}$$
substitute coefficients
$$\cot{\left(2 \phi \right)} = 0$$
then
$$\phi = \frac{\pi}{4}$$
$$\sin{\left(2 \phi \right)} = 1$$
$$\cos{\left(2 \phi \right)} = 0$$
$$\cos{\left(\phi \right)} = \sqrt{\frac{\cos{\left(2 \phi \right)}}{2} + \frac{1}{2}}$$
$$\sin{\left(\phi \right)} = \sqrt{- \cos^{2}{\left(\phi \right)} + 1}$$
$$\cos{\left(\phi \right)} = \frac{\sqrt{2}}{2}$$
$$\sin{\left(\phi \right)} = \frac{\sqrt{2}}{2}$$
substitute coefficients
$$x' = \frac{\sqrt{2} \tilde x}{2} - \frac{\sqrt{2} \tilde y}{2}$$
$$y' = \frac{\sqrt{2} \tilde x}{2} + \frac{\sqrt{2} \tilde y}{2}$$
then the equation turns from
$$3 x'^{2} + 10 x' y' + 3 y'^{2} - 8 = 0$$
to
$$3 \left(\frac{\sqrt{2} \tilde x}{2} - \frac{\sqrt{2} \tilde y}{2}\right)^{2} + 10 \left(\frac{\sqrt{2} \tilde x}{2} - \frac{\sqrt{2} \tilde y}{2}\right) \left(\frac{\sqrt{2} \tilde x}{2} + \frac{\sqrt{2} \tilde y}{2}\right) + 3 \left(\frac{\sqrt{2} \tilde x}{2} + \frac{\sqrt{2} \tilde y}{2}\right)^{2} - 8 = 0$$
simplify
$$8 \tilde x^{2} - 2 \tilde y^{2} - 8 = 0$$
$$- 8 \tilde x^{2} + 2 \tilde y^{2} + 8 = 0$$
Given equation is hyperbole
$$\tilde x^{2} - \frac{\tilde y^{2}}{4} = 1$$
- reduced to canonical form
The center of canonical coordinate system at point O
(2, -1)
Basis of the canonical coordinate system
$$\vec e_{1} = \left( \frac{\sqrt{2}}{2}, \ \frac{\sqrt{2}}{2}\right)$$
$$\vec e_{2} = \left( - \frac{\sqrt{2}}{2}, \ \frac{\sqrt{2}}{2}\right)$$