Orgasm issues for him
While it's generally accepted that women may struggle with orgasm, men are often expected to be ready to go at any point. In reality, men can have just as many issues as women.
Just because you want to be up and ready for action doesn't mean it'll happen on demand. Stress and performance anxiety can take their toll and that's before you even get into the myriad other things that can affect an erection, including alcohol, prescribed and non-prescribed drugs and an unhealthy lifestyle. Sometimes it can be your body's way of saying you're not ready to have sex with a partner too.
Whether it's down to inexperience, insecurity or simply extreme arousal, there's no reason to worry about it unless it happens every time. Delay condoms can help, or if you're able to climax more than once, wait for your body to recover then try again. The second erection will often last longer than the first (which is why some guys masturbate before a date - another useful coping mechanism) There are plenty of ways to pleasure your partner with your hands or mouth - and a love ring can help speed her orgasm too.
Similarly, there may well be a time when you really want to come but you can't. This can be due to stress, certain medications, alcohol or a host of other reasons. Again, unless it's a regular problem, don't be too concerned - though your partner may need reassurance that they are attractive.
Sensitivity after orgasm
Once you've come, the penis may well be too sensitive to touch for a while. This is known as the refractory period, but it you give it a little time, the sensitivity should die down. Using lube can help make the second time more comfortable by helping you avoid chafing - but the golden rule with sex is that if it hurts then don't do it.
Just because you want to be up and ready for action doesn't mean it'll happen on demand.
Regardless of the issue, it doesn't make you less of a man, and if a partner judges you harshly it's worth considering whether they have your best interests at heart. However, rather than ignoring any problems, it's always worth visiting the doctor if you're concerned about what's going on down below. Erection issues can indicate medical disorders such as diabetes, and it's always better to catch these things early.