AI and LLM Interview Questions 2026: Freshers to 10 Years, Transformers, RAG, Evaluation, Agents and MLOps

How to use this guide

This guide covers AI and large language model interviews, from machine learning basics to production systems. The answers explain the concepts and the trade-offs that interviewers test. The field changes quickly, so the guide focuses on ideas that remain stable, and it names specific tools only as examples.

The guide does not attribute any experience to a named person or company. The answers reflect standard, widely described methods.

What the interview pattern usually looks like

Applied roles focus on building and evaluating systems. Research roles focus on model design and experiments. Product roles ask about user risk, cost and quality.

Level 1: Machine learning and neural network basics

Q1. What is the difference between AI, machine learning and deep learning?

AI is the broad goal of machines doing tasks that need intelligence. Machine learning learns patterns from data. Deep learning uses neural networks with many layers, which handle images, text and speech well.

Q2. What is a neural network, in one minute?

Layers of weighted connections with non-linear activation functions. Training adjusts the weights to reduce a loss function, using backpropagation and gradient descent.

input -> [layer 1] -> [layer 2] -> ... -> output
          weights     weights           prediction
          loss = how far prediction is from the true answer
          backpropagation computes how each weight should change

Q3. What is gradient descent?

An optimisation method that moves the weights a small step in the direction that lowers the loss. The learning rate sets the step size. Too large and training diverges; too small and it is slow.

Q4. What is overfitting, and how do you handle it in deep learning?

The model memorises the training data. Handle it with more data, data augmentation, dropout, weight decay, early stopping, and a validation set that is kept separate.

Q5. What is the difference between training, validation and test sets?

Training fits the parameters. Validation chooses hyperparameters and checks progress. The test set is used once, at the end, to report the final result honestly.

Q6. What are activation functions, and why are they needed?

Non-linear functions, such as ReLU or GELU, let the network learn non-linear patterns. Without them, stacked layers collapse into one linear function.

Q7. What is a loss function? Name two examples.

A measure of error the training minimises. Cross-entropy for classification and next-token prediction. Mean squared error for regression.

Level 2: Transformers and LLMs

Q8. What is a transformer?

A neural network architecture built on attention. It processes all tokens in parallel, unlike recurrent networks, and learns which tokens relate to which. Most large language models are transformers.

Q9. What is self-attention?

Each token computes a score with every other token to decide how much to draw on each one. The scores come from query and key vectors, and the output mixes value vectors with those weights.

Attention(Q, K, V) = softmax(Q Kᵀ / sqrt(d)) V

Q: what this token is looking for
K: what each token offers
V: the content each token contributes

Q10. What are tokens, and why do they matter?

Text is split into tokens, which are often word parts. Cost, speed and the context limit are measured in tokens. Count tokens with the provider's tokenizer, not by counting words.

Q11. What is a context window?

The maximum number of tokens the model can read and produce in one call. Long inputs cost more and can push out relevant content, so manage what you send.

Q12. What is the difference between pre-training, fine-tuning and instruction tuning?

Pre-training learns general language from a large corpus by predicting the next token. Instruction tuning trains the model to follow instructions with example prompts and answers. Fine-tuning adapts the model further to a task or domain.

Q13. What does temperature control?

The randomness of sampling. Low temperature favours the most likely tokens and gives more consistent output. High temperature gives more varied output. For factual tasks, use a low value.

Q14. What is the difference between an encoder-only, a decoder-only and an encoder-decoder model?

Encoder-only models, such as BERT, suit classification and search embeddings. Decoder-only models, such as GPT-style chat models, generate text one token at a time. Encoder-decoder models, such as T5, map an input sequence to an output sequence.

Q15. Why do LLMs sometimes produce confident wrong answers?

They learn patterns of likely text, not a verified database of facts. Without grounding in a source or a tool, they can produce plausible but false statements.

Level 3: Prompting and structured output

Q16. What is prompt engineering, and what are the basic techniques?

Designing the instructions and examples to get the output you need. Techniques: a clear role and task, explicit constraints, a few examples (few-shot), step-by-step reasoning for hard tasks, and separating instructions from input data with delimiters.

Q17. How do you get structured output from an LLM?

Ask for a specific schema, use the provider's structured output or tool-calling feature where available, validate the result with a schema library, and retry with the error message if validation fails. Never trust unvalidated JSON.

Q18. What is a system prompt?

The instruction that sets the model's role, rules and style for a session, separate from the user's message. Keep it stable and short, and test it when it changes.

Q19. What is prompt injection?

Text in the input, such as a web page or a document, tries to override the instructions. Defend by treating retrieved content as data, limiting what tools the model can call, separating privileges, validating outputs, and never letting untrusted text choose a sensitive action without confirmation.

Q20. Give an example of a prompt that is too vague, and how you would improve it.

"Summarise this." Improve it by stating the audience, the length, the format and what to include and exclude, for example: "Summarise this support ticket for an engineer in five bullet points, listing the error, the steps tried and the next action."

Level 4: Embeddings and RAG

Q21. What is an embedding?

A vector that represents the meaning of a piece of text. Similar texts have vectors that are close in distance, such as by cosine similarity. Embeddings power semantic search.

Q22. What is retrieval-augmented generation?

A pattern where the system retrieves relevant documents for a question and gives them to the LLM as context, so the answer is grounded in your data.

Question
   |
   v
Embed question -> search vector index -> top relevant chunks
   |
   v
Prompt = instructions + chunks + question -> LLM -> answer with sources

Q23. How do you build a RAG pipeline?

1. Load and clean documents, keeping metadata such as title, date and section.
2. Split into chunks, usually a few hundred tokens, with a small overlap.
3. Embed each chunk and store the vectors with metadata in a vector index.
4. At query time, embed the question and retrieve the top K chunks, optionally with keyword search.
5. Rerank the candidates if precision matters.
6. Build the prompt with the chunks and a rule to answer only from them, and to say when the answer is not there.
7. Return the answer with citations to the chunks.

Q24. How do you choose a chunk size?

Test it. Too small loses context. Too large brings in noise and uses the context window. Measure retrieval recall on a set of questions with known answers, and choose the size that retrieves the right chunk most often.

Q25. What is hybrid search, and why use it?

Combining keyword search, such as BM25, with vector search. Keyword search catches exact terms such as product codes, and vector search catches meaning. Merge the two ranked lists.

Q26. What is reranking?

A second model that scores each retrieved candidate against the question more carefully than the first search did. It improves the order of the top results at the cost of extra latency.

Q27. A RAG system gives wrong answers. How do you debug it?

Split the problem. First check retrieval: is the right chunk in the top K? If not, fix chunking, the embedding model or the search. If retrieval is right, check the prompt and the model's use of the context. Log the retrieved chunks with each answer to make this possible.

Q28. When do you not use RAG?

When the data is small enough to fit in the context and rarely changes, when the task is a calculation or a database lookup better done with a tool, or when the answer does not depend on your documents.

Level 5: Fine-tuning and training choices

Q29. When would you fine-tune instead of using prompting or RAG?

Fine-tune for a consistent style or format the prompt cannot reliably produce, for a narrow task with plenty of labelled examples, or to reduce cost and latency with a smaller model. Use RAG for knowledge that changes or must be cited. Start with prompting, then RAG, then fine-tuning only when the earlier steps fall short.

Q30. What is LoRA?

Low-rank adaptation. It freezes the base model and trains small added matrices. It needs far less memory and storage than full fine-tuning, and the adapter can be swapped in and out.

Q31. What is the difference between full fine-tuning and parameter-efficient fine-tuning?

Full fine-tuning updates all weights and needs a lot of compute. Parameter-efficient methods such as LoRA update a small set of added parameters, which is cheaper and easier to deploy.

Q32. What data do you need to fine-tune?

High-quality examples of the input and the desired output, in the format the model will see. Quality matters more than volume. Hold out a test set that the training data never touches.

Q33. What is catastrophic forgetting?

When fine-tuning on a narrow task makes the model lose general abilities. Mitigate with a mix of general data, a smaller learning rate and evaluation on general tasks.

Q34. What is distillation?

Training a smaller student model to imitate a larger teacher model's outputs. It lowers cost and latency while keeping much of the behaviour on the target task.

Level 6: Evaluation and hallucinations

Q35. How do you evaluate an LLM feature?

Build a test set of realistic inputs with expected outputs or rubrics. Score answers for correctness, groundedness to the sources, format, safety and helpfulness. Use automated checks where possible, and human review for the rest. Track the score for each prompt and model change.

Q36. What is an LLM-as-judge, and what are its limits?

Using a model to grade another model's output against a rubric. It scales well, but the judge has its own biases, such as favouring longer answers. Calibrate it against human labels on a sample.

Q37. What are hallucinations, and how do you reduce them?

Confident statements that are not supported by the input or the facts. Reduce them with retrieval and citations, instructions to say "not found" when the source does not answer, lower temperature for factual tasks, tool use for calculations and lookups, and checks on the output.

Q38. How do you measure retrieval quality?

Recall at K: the share of questions for which the right chunk appears in the top K results. Mean reciprocal rank measures how high it ranks. Measure these before you tune the generation step.

Q39. What metrics would you track in production?

Answer quality from sampled reviews, groundedness, user feedback, latency at the 95th percentile, cost per request, token counts, error and refusal rates, and drift in the input topics.

Q40. How do you handle a bad answer reported by a user?

Log the input, the retrieved context and the output. Reproduce it. Find the cause: retrieval, prompt, model, or data. Add the case to the test set, fix it, and check that the other test cases still pass.

Level 7: Agents, tools and production

Q41. What is tool calling?

The model is given a list of tools with names, descriptions and input schemas. It returns a structured request to call one. Your code runs the tool and sends the result back to the model. The model does not run code by itself.

Q42. What is an AI agent?

A system where the model decides the next step in a loop: it plans, calls tools, reads results and continues until the task is done or a limit is reached. Set step limits, budgets and a way for a human to take over.

Q43. What are the main risks of an agent?

Wrong tool calls, loops that spend money, actions that cannot be undone, data leaks through tool output, and prompt injection from retrieved content. Limit permissions, require confirmation for irreversible actions, log every step, and test with adversarial inputs.

Q44. How do you control cost and latency?

Use the smallest model that meets the quality bar. Cache responses for repeated requests, and use prompt caching where the provider supports it. Shorten prompts and retrieved context. Stream the output for a better perceived speed. Route simple requests to a cheaper model.

Q45. How do you deploy an LLM feature safely?

Put it behind a feature flag, roll out to a small share of users, compare against the current behaviour with the evaluation set, monitor the production metrics, and keep a fallback path if the model provider fails.

Q46. What is MLOps, and what does it add for LLM systems?

Practices for versioning data, prompts and models, automated evaluation, deployment, monitoring and retraining. For LLM systems, prompts and retrieval indexes are versioned as carefully as code.

Level 8: Responsible AI and security

Q47. What is bias in a model, and how do you check for it?

Systematic unfair errors for groups of people. Check by measuring performance and error types across groups in the evaluation set, and by reviewing samples. Mitigate with better data, rebalancing, and a decision process that includes people who can override the model.

Q48. What should you avoid putting into an LLM prompt?

Personal data you do not need, secrets, and confidential data without the approval of the owner and a contract that covers it. Check the provider's data handling terms, and redact sensitive fields where possible.

Q49. How would you explain an AI decision to a non-technical user?

Say what the system did, what information it used, how confident it is, and how to appeal or correct it. Avoid overstating certainty.

Q50. When should a company not use AI for a task?

When errors are costly and cannot be reviewed, when a simple rule or database query is more reliable, when the data cannot be protected, or when the people affected have no way to contest the result.

Design questions asked in AI rounds

D1. Design a customer support assistant for an e-commerce company.

Requirements: answer order and policy questions, hand off to a person when needed, stay within policy. Design: retrieve policy and order status through tools with the user's permission, answer from retrieved sources with citations, and escalate on low confidence or sensitive requests. Evaluation: a test set of real questions, a groundedness score, a handoff rate and customer satisfaction. Safety: never change an order without a verified identity and an explicit confirmation step.

D2. Design a search over 10 million internal documents with access control.

Index documents with their permission groups. At query time, filter results by the user's permissions before ranking or before generating an answer, not after. Use hybrid search, reranking and citations. Refresh the index on change events. Log access for audit.

D3. An LLM feature costs too much. What do you do?

Measure cost per request and where tokens go. Shorten the prompt and retrieved context, cache repeated answers, route easy requests to a smaller model, and batch background work. Confirm each change does not reduce the quality score below the agreed bar.

How to answer 'explain a project'

Describe the problem, the data, the approach, the evaluation and the result. Name what failed: a bad chunking strategy, a prompt that leaked instructions, or a model that was too slow. Interviewers trust the candidate who can explain the failure and the fix.

Only claim metrics you measured, and say how you measured them.

Common mistakes

• Starting with fine-tuning. Try prompting and retrieval first, and measure.
• No evaluation set. Without one, you cannot tell if a change helped.
• Trusting model output without validation. Validate structure and check facts.
• Ignoring prompt injection. Treat retrieved text as data, not as instructions.
• Giving the model tools with broad permissions. Apply least privilege.
• Quoting a benchmark score as your own result. Report your own measurements.

A four-week preparation plan

Frequently asked questions

Which programming language is needed? Python is the standard for AI work. Know how to call an API, handle JSON, and write tests.

Do I need a GPU to practise? No. Use hosted APIs for LLM work and small open models for experiments. Training large models needs compute, which is not required for most applied roles.

What about fresher roles? Applied AI engineer, data analyst with LLM tools, or ML engineer intern roles are common entry points. Show a project with evaluation and a clear write-up.