The chemical reactions and pathways resulting in the breakdown of phytosteroids, steroids that differ from animal steroids in having substitutions at C24 and/or a double bond at C22. Phytosteroids are so named because they occur in higher plants; some, notably ergosterol, are also found in fungi.