This is fine, really -- none of today's work is sensitive to the time I do it, but it can often be difficult to get myself to leave the office "early" on the days I come in early.
So I put an appointment in my calendar to go home at 4pm, and see if that helps.
Is this a problem you have? How do you manage it?