Like I said, there were lots of things that went into my conclusion. However, the straw that broke my testimony-camel's back was the Book of Abraham.
The Book of Abraham is part of the canonized scripture (also known as the "Standard Works") of the LDS church. It was purportedly translated by Joseph Smith in 1835. It was supposed to have come from Egyptian papyrus that had found its way into the hands of Joseph Smith. The papyrus had been discovered, along with some mummies, by Antonio Lebolo around the year 1820. Lebolo had them sold, and they were bought in New York by Michael Chandler in 1833. Chandler then made a travelling exhibition of the mummies and papyrus, occasionally selling parts of the exhibit.
In July of 1835, Chandler brought the mummies to Kirtland, Ohio, where the Latter-Day Saints were settled. Although the Rosetta Stone had been discovered in 1799, only a few people in Europe and certainly no one in America would have known about it, much less been able to translate Egyptian hieroglyphs. Chandler had brought them to Kirtland because he had heard of Joseph Smith's supposed abilities as a translator of ancient languages.
When the mummies had been brought to Kirtland, the LDS people were very interested in purchasing the papyri. Chandler, however, refused to sell the papyri alone; he insisted on selling the entire exhibit. The LDS members raised the necessary $2400 and purchased the exhibit. For this reason, the Book of Abraham and several related books are called the "Pearl of Great Price", in reference to a verse from the New Testament about a man who went and sold all he had in order to buy a single pearl.
After this, the story goes that Joseph Smith translated the papyrus, and the translation was published as the "Book of Abraham", along with several facsimiles of pictures from the papyri.
There are lots of issues with this version of the story. The question is further complicated by the fact that some of the papyri owned by Joseph Smith was destroyed in the Great Chicago Fire of 1871. In 1966, the surviving papyri were found in the Metropolitan Museum of Art and translated. The translation of the surviving papyri had nothing to do with Abraham.
However, this problem was not the major one for me. After all, some of the papyri were destroyed in the Chicago fire, and it's perfectly plausible that the papyrus that the Book of Abraham supposedly came from was destroyed. The main problem for me lies with the facsimiles in the Book of Abraham. These facsimiles are published in the Book of Abraham, along with commentary on what they are supposed to mean. Two of these facsimiles contain actual hieroglyphics. The third is problematic because it is actually included in the surviving fragments of Joseph Smith papyri, but part of the picture is missing.
The facsimile as published in the LDS Standard Works
A photograph of the surviving papyrus fragment
With this one, the argument goes that the fragments missing from the original papyrus were erroneously redrawn. In the LDS Standard Works, the picture is said to be of Abraham on an altar, being sacrificed by an evil priest. The bird in the picture is representative of the Spirit of the Lord, with the figures under the altar being heathen gods.
However, the contention is that the image is in fact a picture of Osiris being resurrected. The figure to the left is the god Anubis, effecting the resurrection of Osiris; the bird is his spirit, the figures under the altar are canopic jars, and the very bottom figure is a sacred crocodile symbolic of the god Sedet. Below is a professional reconstruction of the image:
As you can see, the indication seems to be that the existing papyrus fragment was already damaged when Joseph Smith acquired it, and that he drew in the missing parts as he saw fit. However, with this particular facsimile, it's perfectly possible that the image on the papyrus really was as pictured in the LDS Standard Works. It seems unlikely to me, but it's a perfectly rational objection.
This type of image, frequently found in Egyptian papyri, is called a hypocephalus. Hypocephali were typically buried with Egyptian mummies, and they represent Egyptian ideas about death and life after death-- further reinforcing the conclusion that the papyri that the Book of Abraham supposedly came from are simply funerary documents. As you can see, the hypocephalus (pictured above) has actual, translatable heiroglyphics on it. In the Pearl of Great Price, Joseph Smith's "translation" of said heiroglyphics is included in the text. However, when these heiroglyphics were translated by Egyptologists, they were found to have nothing to do with Abraham, Kolob, or anything else related to the Hebrew religion or to Mormon doctrine.
The last facsimile in the LDS Standard Works is supposed to picture Abraham sitting on the throne of Pharaoh, with other people (Pharaoh; "Prince of Pharaoh, King of Egypt"; Shulem, a waiter; and Olimlah, a slave) standing about. In the Pearl of Great Price, the interpretation of the figures explicitly states that the names of the several of the figures are written in the hieroglyphics above them. However, when these hieroglyphics were translated, they were also found not to say the things Joseph Smith said they did.
These three facsimiles alone seem to me to be adequate to show that Joseph Smith was not what he said he was. Unfortunately, the Book of Abraham is just the tip of the iceberg when it comes to Mormonism.