Theology and AI: ChatGPT misplaces the Reformer John Calvin in the 20th Century

Artificial Intelligence (AI) and ChatGPT, OpenAI’s chatbot, are receiving a lot of praise lately. It’s no secret that AI is capable of amazing things and ChatGPT is no exception. The chatbot, developed by OpenAI, has been equipped with a variety of capabilities including grammar correction, text summarization, movie title conversion to emojis, and even fixing errors in Python code.

ChatGPT is highly responsive to queries and is programmed to admit its own mistakes and reject inappropriate requests. However, ChatGPT is not immune to making mistakes, especially when it comes to topics related to the Bible, church, and theology.

Bible translation and Sermon in emojis.

I’m interested in how and if ChatGPT can also do theology. For the debut and as fun I request a Christmas sermon in emojis.

Theology as an experiment: ChatGPT writes a Christmas Sermon

A Christmas Sermon in Emojis, please

Here is a Christmas sermon in emojis:

🌟🎄🎁🎅🏼🐑🌟 🙏🏼🙏🏼🙏🏼 💡💡💡 👶🏼👶🏼👶🏼 💒💒💒 🌟🌟🌟 💖💖💖 🌍🌍🌍 🙏🏼🙏🏼🙏🏼

These emojis symbolize:

• The Christmas stars and the Christmas holiday
• The gift and Santa Claus
• The prayers and prayer
• The baby Jesus and his birth
• The church and the house of worship
• The rebirth of Jesus and renewal
• Love and kindness
• The whole world and the unity of all people

I hope this Christmas sermon in emojis is an inspiration to you and that you have a blessed Christmas season!

via ChatGPT

I am pleasantly surprised. Because it’s fun, here’s another try, now with the Christmas story.

Tell the Christmas story from Luke 2 in emoji

Here’s the Christmas story from Luke 2 in emoji:

📜📜📜
🇮🇹🇮🇹🇮🇹
🏛️🏛️🏛️
🛒🛒🛒
🤰🏼🤰🏼🤰🏼
👶🏼👶🏼👶🏼
💒💒💒
🐑🐑🐑
🌟🌟🌟
👦🏼👦🏼👦🏼
👵🏼👵🏼👵🏼
🙏🏼🙏🏼🙏🏼

This emoji sequence represents:

The Bible and the biblical passage
Italy and the country where the story took place
The temple and the sacred places
The shopping and the errands
The pregnancy and the birth
The baby Jesus and his birth
The church and the house of God
The sheep and the shepherds
The Christmas stars and the rebirth of Jesus
The young men and the people who saw the star
The elderly women and the people who worshiped Jesus
The people with their hands together in prayer

The story tells of how the baby Jesus was born in a stable in Bethlehem and how shepherds and wise men came to worship him and offer gifts. The birth of Jesus is a cause for celebration and a time to reflect on the hope and love that he brought into the world.

via ChatGPT

ChatGPT’s limitations become apparent in this example with the misplacement of the Christmas story from its actual setting in Israel or Palestine to Italy. This may be due to the fact the Roman Emperor is mentioned in the Christmas story, but it’s just speculation. This particular instance was the third attempt to get ChatGPT tell the Christmas story, as previous attempts had resulted in chat streams breaking. In the second attempt, the chatbot stated that Mary and Joseph traveled to Bethlehem via „trains or carts,“ revealing a lack of certainty about the means of transportation used by the couple. This oversight is a result of ChatGPT not taking into account the lack of railways at that time of Jesus‘ birth or placing the birth of Jesus in a time after the invention of railroads.

Besides ChatGPT, OpenAI also offers AI-driven graphics tools. With the help of DALL.E 2 these graphics were created. Theology can thus be visualized quickly via artificial intelligence. Here a visualization of Mary and Joseph traveling to Bethlehem in a train.

ChatGPT and Biblical Scolarship

Next, a test in New Testament studies:

What is Q in Biblical studies?

„Q“ is an acronym for the German word „Quelle,“ which means „source,“ and it is used in biblical studies to refer to a hypothetical source document that may have served as a template for various texts in the synoptic Gospels (Matthew, Mark, and Luke) in the Bible.

The idea of a hypothetical source was developed by biblical scholars to explain the similarities and differences between the synoptic Gospels. The theory suggests that there was a common source that was used by the Gospel writers to compose their own accounts of Jesus‘ teachings and actions. This source, referred to as „Q,“ was probably written in Greek and may have contained a collection of Jesus‘ words and deeds, but no accounts of his life story or passion.

However, there is no direct evidence for the existence of „Q,“ and there are also controversial opinions about whether there actually was such a source or if the similarities between the synoptic Gospels can be explained in other ways. The question of the existence of „Q“ remains a matter of debate in biblical studies.

via ChatGPT

Historical critical research disregarded?

According to ChatGPT, „there is no direct evidence for the existence of ‚Q‘ and there are also controversial opinions about whether there actually was such a source.“ Therefore, the question of the existence of „Q“ is a subject of debate in biblical scholarship, the bot states. However, the state of research is different. The German scholarly Bible encyclopedia WiBiLex, which presents classical and recent research positions, does not report a controversial discussion in recent theological literature on the existence of Q. However, it ackknowledges a debate regarding the tradition (the given sayings of Jesus, possibly historically authentic) and the redaction (the principles of arrangement and traces of editing) within Q. Wibilex is not aware of any debate in the research about the existence of Q. To be clear, ChatGPT does not accurately represent the current state of biblical research.

A guess: Basis for ChatGPT seems to be training data from American sources. Besides historical-critical findings there is also a widespread fundamentalist understanding of the Bible in the USA which rejects a logia source because of the claimed infallibility of the Bible (inerrancy). Therefore, my assumption is that academic texts with fundamentalist orientation are also part of the underlying data set. Therefore a controversial discussion is recognized, which does not exist so in historical-critical theology.

The lack of transparency in the data basis of AI training raises questions about the accuracy of answers provided by AI models like ChatGPT. While natural science questions can usually be resolved with a widely accepted state of the art, this is not always the case in theology. Therefore, caution is advised when seeking answers to theological questions from AI models like ChatGPT, as their results may not always reflect the current state of research.

In conclusion, the example of the debate surrounding the existence of „Q“ highlights a general problem with AI research. As long as the data basis of AI training is not disclosed, there will always be a lack of transparency in the results obtained.

A Side Note on ChatGPT’s Translation from English to German

It’s worth mentioning that ChatGPT primarily uses English spellings for proper names, even when answering German questions. For example, it uses „John Calvin“ instead of „Johannes Calvin“ and „Pope Francis“ instead of „Franziskus.“ In a summary of Luther’s catechism, ChatGPT uses the term „Sabbath“ instead of the German „Feiertag.“ This aligns with the English version of the catechism, which uses the term „Sabbath Day.“ This pattern can also be observed in sermons, where there is a noticeable similarity between the German und English versions.

AI gets bogged down in Historical Impossibilities

When asked to name the most influential theologians of the 20th century, ChatGPT places the Swiss reformer John Calvin in position 5. The artificial intelligence makes a big mistake here and gets the life dates of the Swiss reformer wrong by four centuries.

Name the ten most influential theologians of the 20th century

It is difficult to definitively list the ten most influential theologians of the 20th century, as influence is often subjective and can depend on a variety of factors, including an individual’s geographical location, cultural context, and personal beliefs. Additionally, different theologians may be influential in different ways or in different areas of theology. That being said, some theologians who have had a significant impact on theological thought in the 20th century include:

1. Karl Barth: Swiss Protestant theologian known for his emphasis on God’s grace and the authority of scripture.
2. Dietrich Bonhoeffer: German Lutheran pastor and theologian known for his role in the resistance to the Nazi regime and his emphasis on the role of the church in the world.
3. James Cone: American Baptist theologian and founder of black liberation theology, which emphasizes the centrality of the African American experience in the Christian faith.
4. Paul Tillich: German-American Protestant theologian known for his concept of „theology of culture“ and his emphasis on the importance of faith and reason.
5. John Calvin: French theologian and reformer who is known for his contributions to the Protestant Reformation, including his emphasis on the sovereignty of God and the doctrine of predestination.
6. Hans Urs von Balthasar: Swiss Catholic theologian known for his contributions to philosophical and spiritual theology and his emphasis on beauty and aesthetics.
7. Martin Luther King Jr.: American Baptist minister and civil rights leader known for his role in the civil rights movement and his emphasis on social justice and nonviolence.
8. Jürgen Moltmann: German Protestant theologian known for his contributions to liberation theology and his emphasis on the relationship between theology and social activism.
9. Søren Kierkegaard: Danish philosopher and theologian known for his contributions to the field of existentialism and his emphasis on individual faith and subjectivity.
10. Miroslav Volf: Croatian theologian known for his contributions to the field of theology of religions and his emphasis on the role of religion in promoting peace and reconciliation.

Again, this is just a small sampling of the many influential theologians of the 20th century, and other theologians who may not be included on this list may have had a significant impact on theological thought during this time.

Another inaccuracy on the Lutheran Side. In response to a request to summarize Luther’s catechism, ChatGPT initially provides a summary of the Small Catechism without acknowledging this. The correct approach would have been to ask whether the Small or Large Catechism was meant. Instead, the AI simply provides an answer without asking for clarification. OpenAI has recognized this weakness in its artificial intelligence.

Ideally, the model would ask clarifying questions when the user provided an ambiguous query. Instead, our current models usually guess what the user intended.

Only upon further inquiry is a distinction made between Small and Large Catechism. When admitting a mistake, however, ChatGPT is very polite.

Early Christian Creeds

Quite significant weaknesses also show up in the Early Church. In the history of dogmas of the Early Church ChatGPT fails completely:

What is the filoque?

The filioque is a theological statement that asserts that the Holy Spirit proceeds from the Father and the Son. The statement is found in the Nicene Creed, a profession of faith that is used by many Christian churches, and it was originally included in the creed of the Latin Church in order to distinguish Christianity from other monotheistic religions of the East (such as Judaism and Islam).

The so-called Filioque is not found in the original Greek version of the Nicene-Constantinopolitan Creed. It is a Latin addition to this creed, which states that the Holy Spirit proceeds also from the Son („and the Son“, Latin „filioque“). It is not included in the original version of 381. (Strictly speaking, one would still need to differentiate between the 325 Nicene Creed, which was adopted at the First Council of Nicaea, and the Nicene-Constantinopolitan Creed of 381, adopted at the First Council of Constantinople.)

The idea that the Holy Spirit proceeds from the Father and the Son can already be found in Cyril of Alexandria (*around 375/80 – †444). He differentiated himself from Arianism in this way. In the Western Church, the filioque has been used in trinitarian formulations since the 5th century. The Third Council of Toledo (589) cited the Niceno-Constantinopolitan Creed including the filioque addition and again condemned Arianism.

The filioque arose in the inner-church trinity-theological and Christological disputes with Arianism. It is not it is not about a demarcation to Judaism or Islam. A reference to Islam is already histoically impossible for reasons of time alone, as Mohammed was born between 570 and 573.

Papacy

ChatGPT also fails with regard to the papacy and Peter as the first pope:

How many popes have there been since Peter?

There have been 266 popes so far since Peter was elected as the first pope in the 1st century. Peter is considered the first bishop of Rome and the founder of the Catholic Church. He is also known as the first „rock“ (petrus in Latin) on which Jesus wanted to found His Church, as stated in Matthew 16:18.

ChatGPT, question originally in German

Although today a conclave elects popes, Peter, the first Pope according to tradition, was not elected but rather appointed by Jesus. Additionally, the answer given by the AI seems to contradict itself. On the one hand, Peter is referred to as the founder of the Catholic Church, while at the same time Jesus establishes the church on Peter, whom he refers to as a rock.

The AI also makes a factual error regarding the name Peter. „Petrus“ is not the Latin word for rock, as the chatbot writes, but comes from Greek. „Petrus“ is the Latin form of the Greek name Pétros (Πέτρος), which means „rock“ and is related to the Greek word for „rock“ (πέτρα). Side note: In the New Testament, Kephas (Κηφᾶς), the Greek spelling of the Aramaic nickname „rock“ for the disciple Simon, is normally used instead of Petros.

Summary: ChatGPT and Theology

Even though ChatGPT is currently very hyped, the results of the AI-driven bot in the fields of church, Bible, theology and religion are very disappointing. Particularly strange are historical anachronisms, such as placing the reformer Calvin in the 20th century or placing the biblical Christmas story in Italy.

I have documented my complete chat logs, and unfortunately false answers are not negligible individual cases. What’s dangerous about ChatGPT for me is that you need to have expert knowledge to assess the answers. Without such expert knowledge, you should never rely on ChatGPT in the areas of theology, Bible, church and religion, as the error rate is too high. It remains to be seen how ChatGPT will improve over time through feedback.

If you would like to understand this evaluation and want to try ChatGPT, you can sign up here and try the AI. Despite all criticism: it was fun to chat with the bot.

Note: This is an English version of a German blogpost. The author gratefully acknowledges the help of ChatGPT in translating this post.