What it shows
Twenty questions with known answers are put to an AI chatbot and the answers are checked against textbooks, giving an accuracy percentage and a list of confident mistakes. It shows that AI can be useful but must be verified.
The science
A chatbot predicts likely words from patterns in a huge amount of text, so it can write fluent answers that sound confident even when a fact is wrong. It is usually good with common knowledge and weaker with recent events, very local facts, precise numbers and long calculations. Testing with questions whose answers you can verify measures how far to trust it.
Materials
Cost: free. Many items come from home or from waste; spare parts such as LEDs, motors and wire are in the store.
- A computer, laptop or phone with a browser (free)
- A free AI chatbot that your school or family allows, used with an adult present
- A textbook, atlas or reliable reference for checking
- A spreadsheet program: LibreOffice Calc (free) or Google Sheets (free)
- A notebook and a pencil
Safety
For this project: Use the chatbot with an adult, never type your name, address, school or photos, and follow the age rules of the tool.
Never use mains electricity: power models with cells, batteries or a low-voltage adapter. Sharp and hot tools, flames and lab chemicals need an adult or your teacher beside you. Keep magnets, button cells and small parts away from young children, and wash your hands after making.
All the safety rulesStep by step
8 steps. Do them in order, and test each part before you go on to the next. Write what you notice in a notebook as you go.
Write 20 questions in four groups of five: general science, maths calculations, Indian geography and history, and local facts such as your district.
Write the correct answer for each from a textbook or atlas before asking the chatbot, so you do not get influenced.
Ask each question exactly as written, without hints, and copy the answer into a spreadsheet next to the question.
Mark each answer Right, Partly right or Wrong and note if the chatbot sounded sure.
Count the percentage right in each group with =COUNTIF(range,"Right")/5.
Ask the chatbot to show its working for two maths questions and check each step.
Ask it for a source for one answer and check whether the source really exists; note the result.
Chart the four group scores and write a half-page on when to trust a chatbot and how to check it.
What you should see
A table and chart of accuracy by question group, a list of wrong but confident answers, and a short set of rules for using AI carefully.
Viva questions
A teacher or a judge will ask how it works. Try to answer each question before you open it, then compare.
1How does a chatbot produce text?
It predicts likely next words from patterns learned from large amounts of text.
2Why can it be wrong and still sound sure?
Fluent writing does not mean the facts were checked.
3Which kinds of questions are risky?
Recent events, local facts, exact numbers and made-up sources.
4What should you do with an AI answer for homework?
Check it against a reliable source and understand it yourself.
Take it further
Repeat the test a month later or with a second chatbot and compare accuracy.
Change one thing, measure what happens, and add a table or chart of your readings: that is what turns a model into your own investigation.
Free ebooks for this project
Computer & office
Computer Fundamentals Study Book
The complete study notes, practice questions and glossary for Computer Fundamentals