- Sabrina Ramonov
- Posts
- AI-Powered Infinite Test Prep (Part 1 - DMV)
AI-Powered Infinite Test Prep (Part 1 - DMV)
Gen AI Eats DMV Practice Tests
My mom-in-law failed the DMV test 4 times.
She studied aerospace engineering (Rocket Science).
She worked on project Mir, the first modular space station:
She’s smart.
She knows how to drive.
But English is not her native language.
I speak only English, so I couldn’t fully appreciate her difficulties, studying in a foreign language, taking a test in a foreign language.
Yet, she’s far from alone.
While waiting at the DMV, I saw many others facing the same struggle.
I realized…
This is a real problem Gen AI can solve today.
By combining infinite test question generation and language translation, AI-powered test practice will help non-native English speakers:
study and learn concepts in their native language
understand how these concepts translate to English
practice test questions (often worded to trick you) in English
for each question, see explanations in their native language and English
So, I started hacking on PassMyTests.com to help my mom-in-law succeed.
MVP of DMV Test Prep
According to Driving-Tests, over 90% of people who use their DMV practice tests will pass the exam.
But this success rate only applies after taking 10+ practice tests.
This statistic also includes native English speakers.
My stupid simple MVP:
You choose a language and US state (some answers vary per state)
You get a realistic AI-generated question with 4 choices, all in English
You select an answer
You see an explanation for the correct answer, in both English and translated by AI to your native language
After a few iterations, I arrived at the prompt:
You act as a terminal for written test taking at a DMV in the state of ${params.state}.
Produce a random 4-choice question.
The question should pull from a random set of 500 questions.
The question should be in English.
Return the output in the valid JSON of the following format:
{
"body": "Body of the question",
"choices": ["Choice A", "Choice B", "Choice C", "Choice D"],
"answer": <index of the current choice e.g. A is 0, B is 1, etc.>,
"explanation": "Short explanation of the answer",
"translation": "${translation}"
}
I haven’t done a formal evaluation, but from eyeballing outputs, this instruction seems to increase randomness:
“The question should pull from a random set of 500 questions”
The key term is seems.
I could be fooling myself.
My next post on this project will dive into evaluations.
The variable ${translation}
works like this:
If the user didn’t select a language, then there’s no translation
If the user selected a language, then we ask ChatGPT to translate:
"translation": "Translation of the explanation in Spanish"
Putting it all together, here is my MVP for practicing DMV test questions:
Source: PassMyTests.com - DMV
Below is a sample question where the correct answer varies by state.
In California, the answer is 0.08%.
But in Utah, the answer is 0.05%.
Source: PassMyTests.com - DMV
Challenges
Multiple Modalities
One challenge is is that only certain types of questions are generated.
I couldn’t generate an image of a street sign or show an image as an answer.
This MVP is limited to text-based question and answers.
No support for other modalities yet.
Here’s a sample DMV test question that involves an image:
Complex Situations
In the same vein, I did not generate complex situational questions.
For example, a question could show a picture of two cards behind a school bus, while another car is turning left from the opposite lane.
Then, the question asks you to determine right of way.
Unfortunately, my MVP neither generates images nor complex situations.
Changing Laws and Knowledge
I grew up learning 10 and 2 as the optimal position to keep my hands on the steering wheel.
But knowledge evolves.
Because of airbags, now the optimal recommended position is 9 and 3.
Laws also change.
Some factual data might be year-dependent, presenting another challenge.
Ideas for Future Work
I see multiple vectors for improvement.
I could slice a DMV handbook, making a RAG that answers factual questions directly from the latest handbook, along with the explanation.
This would address the challenge of non-static laws and knowledge.
Another opportunity is to increase the quality of LLM outputs by having 2 interacting agents:
one agent generates test questions
second agent answers test questions
I could have open and closed-book variants, although presumably the test-taking agent should perform better if it’s open-book!
In general, it would be interesting for someone to turn DMV tests into a proper dataset LLMs can be evaluated on.
The non-trivial reasoning tasks are especially intriguing, such as analyzing an image to determine right of way, law violations, and best next actions.
Happy Ending
Relieved to report…
After a week practicing on PassMyTests, my mom-in-law passed her exam!
Her favorite feature — reading the English explanation right above the explanation in her native language.
She already knew most concepts, but often mistranslation caused confusion.
Seeing the English and translated explanations side-by-side helped her understand nuances in the questions, answers, and explanations.
If you have feedback or requests for PassMyTests.com
DM me anytime @Sabrina_Ramonov or Linkedin.