Ajay Rathod

What is required to ship production-ready AI agents After evaluating the full lifecycle of agent systems, these are the capabilities most teams need beyond basic orchestration: Parallelization — reduce actual end-to-end latency Streaming — improve perceived responsiveness Task queues — minimize retry volume Checkpointing — lower the cost of each retry Human-in-the-loop — enable controlled collaboration Tracing — understand real-world usage patterns These sit on top of foundational requirements: observability, evaluations, deployment infrastructure, and monitoring. Without them, agents remain demos rather than reliable production systems.

Forward Deployed Engineer Q&A Experience at Cisco These are questions asked in the actual tech round. Hope this helps anyone preparing for AI Engineer / FDE roles. 1. Agent Frameworks & Tooling Why LangGraph and why LangSmith? LangGraph: Agent workflows are not linear — they loop, branch, retry, and call tools. LangGraph models the agent as a stateful graph (nodes = steps/tools, edges = transitions) with built-in state, cycles, and human-in-the-loop checkpoints. This provides control and determinism compared to a black-box agent. LangSmith: Used for observability and evaluation. It traces every step of the agent (prompts, tool calls, tokens, latency, cost), helps debug failures, and supports running evaluation datasets. LangGraph runs the agent. LangSmith tells you why it did what it did and whether it’s improving. How do you make LangGraph performant? - Run independent nodes in parallel instead of sequentially - Cache tool results and use semantic caching for repeated LLM calls - Right-size the model per node (small/fast model for routing, larger model only for hard reasoning) Stream tokens - Keep state minimal in the graph - Use conditional edges to short-circuit and avoid unnecessary LLM calls 2. Prompting Patterns Prompt chaining Break a complex task into a sequence of smaller prompts, where each step’s output feeds the next (e.g., extract → transform → summarize). This is more reliable, debuggable, and often cheaper than one large prompt. Each step can be tested independently. Parent and child prompt splitting A parent (orchestrator) prompt decomposes a task into subtasks and delegates each to a specialized child prompt (or sub-agent), then aggregates the results. This is the orchestrator-worker / hierarchical-agent pattern. It helps isolate context, enable parallelism, and keep each prompt focused. 3. Retrieval & Vector Databases Why a vector database, and why Pinecone or Weaviate? Vector databases store embeddings and perform fast similarity search (ANN). They are the backbone of Retrieval-Augmented Generation (RAG). Pinecone: Fully managed and serverless. Good for scale with minimal operational overhead. Weaviate: Open-source. Supports hybrid search (vector + keyword), has built-in modules, and can be self-hosted (useful for data residency or on-prem requirements). Selection depends on managed vs self-hosted needs, hybrid search requirements, and metadata filtering needs. When to use RAG vs agentic development? RAG: Best when the task is fundamentally “answer grounded in my data” — Q&A, search, or summarization. Typically one retrieve-then-generate step. Agentic: Best when the task requires multi-step reasoning, tool use, planning, and decision-making. The model plans, calls tools/APIs, reacts to results, and can loop. RAG is a retrieval step. Agentic is a reasoning loop. They are often combined — RAG can be one tool an agent is allowed to call. 4. Guardrails & Safety Guardrails enforce structure and safety on both inputs and outputs? Frameworks such as Guardrails AI provide many validators, including: - Toxic language detection - PII detection and redaction - Profanity filters - Competitor mention checks - Valid JSON / schema enforcement - Groundedness / no-hallucination checks - Topic restriction A toxicity guardrail screens generated output (and sometimes user input) for harmful content and can block or rewrite it before it reaches the user. Guardrails typically run on both input (prompt injection, PII, off-topic) and output (toxicity, format, groundedness), with configurable fail actions such as block, retry, or fix. 5. LLM Gateway What is an LLM Gateway and why use one? An LLM Gateway is a single entry point in front of multiple model providers (similar to an API gateway for LLMs). It typically provides: - Model routing (to the right or cheapest model) - Rate limiting and quotas - Caching - Retries and failover across providers - Authentication - Centralized logging of tokens and cost It decouples the application from any single provider and becomes the place to enforce cost control, reliability, and security. 6. MLOps & Lifecycle Design an MLOps pipeline — what are you trying to achieve? The goal is to automate and govern the full model/LLM lifecycle so that releases are repeatable, safe, and observable. Typical stages: Data ingestion → validation → feature engineering → training / experiment tracking (e.g., MLflow) → model registry + versioning → automated evaluation gate in CI → deployment (canary / blue-green) → monitoring for drift, quality, latency, and cost → retraining trigger. For LLMs, also include prompt/version management and evaluation suites before promotion. Product development lifecycle for an AI product Problem / scenario definition → data & feasibility → prototype (prompt / RAG / agent) → offline evaluation → guardrails & security → deploy behind a gateway → online evaluation + human feedback → monitor and iterate. Evaluation and feedback loops should be continuous rather than one-time. How do you evaluate an LLM or agent? Offline: Curated test sets scored for accuracy, groundedness/faithfulness, relevance, toxicity, and hallucination rate. Can use LLM-as-judge plus human review. Online: A/B tests, user feedback, and production traces (e.g., via LangSmith). Gate every prompt or model change behind an automated evaluation suite in CI. 7. System Design Scenarios Design a 10-million file upload system High-level flow: Client → CDN → Load Balancer → API Gateway → Upload Service Use pre-signed URLs or multipart / chunked uploads directly to object storage (e.g., S3). Publish an event to a message system such as Kafka. Async workers transform/enrich the data and write metadata to a database. Index the content for search. Key considerations: scale, retries, deduplication (checksums), and idempotency. Message Queue vs Kafka A traditional message queue delivers a message once and then deletes it. Kafka retains an ordered, replayable log. It is highly scalable and partitioned, allowing multiple independent consumers to read and replay history. A queue is typically used to deliver work to a worker. Kafka is used as a durable event stream. 8. Production Agent under Burst Traffic Scenario: You have an agentic AI system in production. Non-technical users hit it, keep typing, and cause traffic bursts. How do you keep it from crashing and ensure it continues to work? Protect, scale, and degrade gracefully: Rate limiting and throttling per user / API key at the gateway, plus quotas. Queue + backpressure — buffer bursts and process at a controlled rate (async for longer tasks) Autoscaling — horizontal scaling of the agent service (and GPU pool) behind a load balancer, based on QPS or queue depth Semantic caching — many non-technical users ask similar questions; cache to avoid inference entirely Timeouts, retries with backoff, and circuit breakers — fail fast and isolate a slow model or provider; failover to a backup model via the LLM gateway Graceful degradation — if overloaded, return a cached/lite response or a “busy, try again” message instead of crashing Observability + alerting — track latency, error rate, tokens/cost, and saturation; alert before the system is overwhelmed Cost and loop guards — cap maximum steps/tokens per request so a runaway agent loop cannot spike cost or hang Design for bursts, not averages. Rate-limit at the edge, absorb spikes in a queue, autoscale, cache aggressively, and degrade gracefully. The agent should get slower under load, never crash. If you give the agent to non-technical users, how does it behave? Assume input will be messy and unbounded: Input guardrails — validate and clean vague, off-topic, or malicious input (prompt-injection defense, PII redaction) Constrain scope — system prompt + tool allow-list so it only does what it should; refuse out-of-scope requests politely Clarify instead of guessing — ask follow-up questions when input is ambiguous Deterministic output guardrails — no toxic content, structured/safe responses, and a human escalation path Handle continuous typing — debounce/deduplicate, rate-limit, and keep conversation state bounded For non-technical users, assume messy, unbounded input. Harden the input side with guardrails, keep the agent scoped, make it ask instead of hallucinate, and never let free-form typing translate into unbounded cost or unsafe actions.

AI Engineer - Interview Topics Agentic architecture ReAct agent Deep agent Ambient agent Evals RBAC and safety Why AI, not rules LangGraph, StateGraph, and reducers MCP versus REST for many tools Transformer attention formula DSA