I'm typesetting a book for a friend. Initially I had to decide between LaTeX and ConTeXt. Although LaTeX had a WYSIWYM editor in LyX I've always liked ConTeXt for its perceived completeness (their documentation always looked better than the rest) but I haven't touched it since 2002 when I last used it for preparing reports for college. However after checking out LyX 1.61 the choice was easy. LyX has come a long way since the early versions that used the XForms toolkit.
As for the document class to use for the book the popular choices seemed to be KOMA-Script and Memoir. I almost selected KOMA-Script when at the last moment decided to take a look the Memoir documentation and I was pleasantly shocked! The documentation for the memoir class is simply brilliant! (and beautifully typeset too!). It is not only a class documentation but also a complete introduction to typesetting. I'd say it is mandatory reading for any novice getting into typsetting. Google for memman.pdf and give it a good reading.
I ended up using LyX with minimal ERT needed to customize the memoir document class.
Here's an article that presents a good overview on designing a book with LyX:
Also a good set of tips was picked up from
For preparing bibliographies, JabRef is a nice utiility for managing your BibTeX databases.
I haven't really looked at indexing options but the book is mainly an internal monograph so the index can be omitted.