The field of mathematics does not have a consensus citation style. When writing for a course assignment, consult your professor for guidance on selecting a style.
Mathematicians are typically required to adopt the citation style used by the journal publishing their research article. However, in the absence of a required citation style, a good one to consider is the style used by the American Mathematical Society.
Due to the equation-heavy nature of mathematical publications and the difficulty of properly formatting these equations with applications like Microsoft Word, it is extremely likely that your mathematical writing will be done using LaTeX and your citations will be managed and formatted using the amsrefs LaTeX package. However, even though the formats of both in-text citations and the bibliography are controlled by the amsrefs package, there is a great deal of flexibility and you should consult with your professor regarding formatting.
You can learn about managing your citations with the amsrefs package in Chapter 7: Bibliographic references of the "AMS Author Handbook: Journal Classes" style guide provided by the American Mathematical Society.
In the absence of a required citation style, might consider using one of the widely used citation styles (i.e. Chicago, APA, MLA) described in detail on the Bates Library "Citing Sources" guide.