Tyler Cowen cited briefly why he doesn't expect a Serenity sequel: low box office. Since its release date (September 30, 2005), Serenity's total box office sales are $22.1 million domestic, $3.6 overseas. That's a $25.7 million revenue versus a $39 million production costs.
These numbers, however, are not perfect as they don't include earnings from preview screenings, and there was a lot of them. Too many, in my opinion; Mike, for example, saw the movie twice before it was even released. Combine this with the low rewatching value (thanks to the death of a certain character), I'm sadly not surprised by the numbers.
Presumably, the powers that be are including preview revenues and Joss will point out the (hopefully) strong DVD sales, but I'm not optimistic.