The basic problem with Gibson's film I think is that he chose the Passion Play format--a format that does away with context and with Jesus' more controversial acts (his scourging of the Temple moneylenders and entrance into Jerusalem on an ass help explain Caiaphas' actions a lot). Without that context, the anti-Jewish elements come out more clearly.
It's the ultimate violent flick--all violence, no or little story. We're suppose to hate--what? Who? That indistinct rage and hatred built up by the film can be directed at anyone, easily. Easily at people who are skeptical of the film. There have been threats at people who've spoken out.