Hamlet states these observations of himself because he feels tethered to the obligation that he must avenge his father, which he believes hasn't gone so well and that he hasn't done anything in response to his father's words. It is justifiable to say that in learning this secret from the ghost of his father, he really hasn't done anything in turn after hearing this newfound information. He has a slew of emotions building up because of it, but he hasn't acted upon those emotions. Hamlet believes that if the lead player, using his own imagination and a fictional situation to emphatically display a show of emotion, could be this passionate, then he must be a coward given his situation. He begins to insult himself, saying things like "Why, what an ass am I! This is most brave," said with sarcasm. He also believes that he is basically acting like a whore by using his words rather than his actions (Hamlet/Line 539). He's on a crossroads of confusion on what he must do.
Hamlet undeniably wants to seek revenge no one can dispute that, but it seems he's lost the passion to actually do it, which he seems to realize in himself when he states, "O, what a rogue and peasant slave am I"" (Hamlet/Line 505). He is condemning his own inaction in this line and continues on to say, "is it not monstrous that this player here could force his soul so to his own conceit,"" angrily stating that he has the means to exert some form of sentiment but cannot do so (Hamlet/Line 506-508). He observes how the first player is able to aptly do so with an imagined situation and motivation. Hamlet believes that if the first player had the motive that he had, he would "drown the stage with tears"" (Hamlet/Line 517).
He believes that he deserves this onslaught of verbal abuse, saying that he should take it: for it cannot be/But I am pigeon-liver;" meaning he is cowardly (Hamlet/Line 532-533). If he had actually acted out his father's revenge, he would have fed Claudius' guts to all of the region kites of scavenging birds, by now.