Chances are if you are traveling in Oman your first port of call will be Muscat. A word of warning to start, taxis in Oman don't have meters and Muscat is fairly spread out so research what you want to see and choose the area you stay in carefully. Always bargain with the taxis before setting off. We found we could get most places in the city for about 3 Omani Riyals after haggling.
Many people choose to stay in Mutrah, an area with a lot of the attractions such as the corniche, souk and one of the forts. It definitely had its appeal, but due to the sheer number of tourists and cruise ships, we found the area to be overpriced and the food served in the restaurants to be poor. We chose instead to stay in Al Khuwair, a part of town with numerous midrange hotels, a hypermarket, and some restaurants. It's also a short taxi ride to the magnificent Sultan Qaboos Mosque, which is worth visiting twice: once during the day and once at sunset.
Non-Muslims can visit the inside of the mosque from 8-11am daily except Fridays. You need to be properly covered and although you can hire abaya, thobes and headscarves at the mosque, it'll cost you. It's generally easier to just come appropriately attired to start with. This means both men and women need to be covered to their ankles and to their wrists, and women need to cover their hair. Once you enter, you are free to wander around and take photos. We went at 8am and had the place to ourselves until the tour buses rolled in at 9am.
If you can't make it to the mosque during the times it is open, it's still worth going and walking around the outside. It's in a quite striking setting with the Hajar mountains behind it, and if you go at sunset, the light hits it beautifully.