What happens if I swap plus and minus on the speaker?

You can swap plus and minus with each other, this causes the polarity of the speaker to be reversed. This causes the speaker to lose volume level and sound imprecise. The correct polarity/setting is always the louder one. Likewise, an incorrectly polarized speaker can affect other correctly polarized speakers, frequencies can be diminished or cancelled. Therefore, always pay attention to the correct polarity.