Wow, I have to disagree with pretty much all of you on this one. I don't think its really that big of a deal to come back within a pre-designated hour that you can usually plan for far in advance. Disney parks are huge and everything, but when I went with my friends last Spring we only had three days to get through all four parks, and yet we still did it no problem and still abiding by the FastPass and riding basically everything.
And the FastPass system wasn't created entirely for the guests' benefit you know. Disney wants as few people in line for attractions as possible at any given time because the longer you aren't in line for a ride, the more likely you are to spend money in the park.
Oh, and it wasn't created so the people who just wanted to ride rides could do so, it was created so everyone could ride what they wanted to without needing to give up time just walking around the park and stuff, which families do quite a bit. By skipping your time and going back later you are screwing up their system and making other guests wait (possibly much) longer when they wouldn't have had to if everyone came back when they were supposed to. I just think it's a bit inconsiderate I guess.