I believe that affirmative action is justified to some extent, however these days it's more appropriate to provide affirmative action based on socio-economic class more than race. I say this because there are many well privileged minorities these days who are getting affirmative action even though they don't need it.
The reason Affirmative action came about was to aid minorities in improving their social status, since some earlier social injustices and prejudices hindered them in the past. However, as I will mention, our current notion of "socio-economic class" has led to some distorted perspectives on affirmative action that are counterproductive.
In the recent past, I personally have seen an example of affirmative action gone awry. I know a person who is a minority who got into a very prestigious college. Clearly some affirmative action was in play, for some better qualified white students were not accepted by that school. However, the minority student came from a very wealthy family, and therefore I don't understand why affirmative action would be needed for him to "improve" his socio-economic status. I especially found it unfair that a relatively poorer and more qualified white student was not accepted for this reason.
I am not saying that affirmative action should be banned altogether, and these days it just so happens that many families of relatively low economic class are minorities, the many remnants of social discrimination that occurs today in some places. However, basing affirmative action solely on race has allowed for some people to abuse the system. We can't blind ourselves to the fact that there are poor white students as well. .
Additionally, I believe that other forms of affirmative action exist that are counterproductive. In many major competitive NCAA schools, academic standards are lowered for minority students who participate in sports. I believe that this is detrimental to the social improvement of some minorities because younger children of those minorities buy into the image of becoming a professional athlete as one of the few high paying jobs that are "appropriate" for their ethnic group.