It’s very helpful to ponder this prayer from two perspectives. The first from the soldiers’ perspective and the other from the Savior's perspective. Now from the soldiers’ perspective, it's important to keep in mind that Jesus clearly meant more with this prayer than just the soldiers who were crucifying Him. His prayer also included forgiving the Sanhedrin Jews who brought Him to this point. His prayer also included forgiving Judas who betrayed Him, and Peter who denied Him three times. It included His followers who did not follow Him all the way to the cross. His prayer also included you and me and all humanity, the humanity that's guilty of the offenses that had brought Jesus to this point of crucifixion.
So from the soldiers’ perspective, when they would have heard those words, “Forgive them, for they know not what they do,” I wouldn't be surprised if a number of them thought, “We know exactly what we're doing! This is our job as trained torturers in the Roman army. We know exactly what we're doing. We are executing you as a criminal in an excruciating way. We know exactly what we're doing by humiliating you, by stripping you naked before all Jerusalem and the crowds that are coming to Jerusalem for this Passover feast. We know what we're doing when we're scourging you with nasty instruments until most of your blood comes out of your body. We know exactly what we're doing by crowning you with thick thorns because you claim kingship, King of the Jews, which is foolish especially in the face of Caesar’s kingship.”
So probably from their own perspective they thought, “We know exactly what we're doing." Ahh, but from the Savior's perspective, they knew not what they were doing.
Think about it, “Father, forgive them, they know not what they do. They know not that I am the eternal God made man who is purposely confronting every man's enemy—death. Death itself at this very moment. Father, forgive them, for they know not that I am the all-powerful Creator who humbled Himself before His creation. They know not that I am fulfilling all the Hebrew scriptures pointing to this moment. They know not that I am the Passover lamb of sacrifice who is delivering My people out of slavery and bondage through this very act. They know not that through this crucifixion, I am confronting Satan, our archenemy, head on; I am confronting him head on. They know not that through this redemptive suffering, I'm about to launch a revolution, for My Spirit will be poured out on all humanity, on all those who turn to Me in repentance and faith. And I will transform their lives such that they will bring My Spirit to every nation, every tribe, every tongue, and bring them My freedom that I am bringing about through now. They know not that I will be resurrected bodily and then enthroned as King of other kings, including Herod and Pilate and Caesar, and any other king who comes after. They know not that I will come again in glory to judge the living and the dead."
"Father, forgive them, for they know not what they do."