Technical deep dive

1. What an LLM actually is, for a statistical audience

An LLM is a very large neural network (a transformer) trained to predict the next token (~¾ of a word) given all preceding tokens, over trillions of words of text. Post-training steps (instruction tuning, reinforcement learning from feedback) shape this raw predictor into an assistant that follows instructions.

Three consequences follow directly from this design, and they explain almost every strength and weakness you'll observe:

1. It is a conditional distribution over text, not a knowledge base. Output is sampled, so it is stochastic: the same prompt can yield different answers. A temperature parameter controls sampling dispersion. Even at temperature 0, exact reproducibility across model versions is not guaranteed, a material point for actuarial documentation (§6).
2. Fluency and truth are different objectives. The model was optimized to produce plausible text. When it lacks knowledge, it produces plausible text anyway: a hallucination. Frequency varies by task: low when summarizing a document it's been given, higher when recalling specific facts (citations, table values, policy language) from memory.
3. Parameters encode compressed knowledge with a training cutoff. The model knows nothing after its cutoff and nothing about your company unless you put that information in the prompt. This is the single most important practical insight: an LLM is only as good as the context you give it.

2. Core concepts, precisely

• Context window: the maximum tokens processed per request (currently 200K–1M+, i.e., hundreds of pages). Everything the model "knows" about your problem lives here. Long-context recall is good but degrades subtly; don't assume perfect retrieval from page 400.
• System prompt: standing instructions that frame every request (role, constraints, output format). In applications, this is where you encode business rules.
• RAG: instead of hoping the model memorized your underwriting guidelines, you (a) chunk and index documents as embeddings (dense vectors where semantic similarity ≈ geometric proximity), (b) retrieve the most relevant chunks per query, and (c) paste them into the prompt with instructions to answer only from them, with citations. The workhorse architecture for grounded internal Q&A. Its failure modes include retrieval failures, not just generation failures; evaluate both stages.
• Fine-tuning: further gradient training on your examples. Good for style and format consistency at scale; generally the wrong tool for injecting knowledge (RAG wins there), and it raises governance burden. Most insurance teams need it rarely or never. Start with prompting + RAG.
• Structured output: modern APIs can force output to conform to a JSON schema. This is how you turn "read this loss run" into reliable columns: {claim_date, cause, incurred, paid, open_flag}. For extraction pipelines, always use schema-constrained output rather than parsing prose.
• Tool use / agents: the model can be given tools (code execution, search, database queries) and can loop: plan → act → observe → act. Crucial implication: never let the LLM do arithmetic in its head; let it write and execute code. An LLM asked to sum a loss triangle will often be slightly wrong; an LLM asked to write pandas code to sum it will almost always be right, and the code is auditable.
• Reasoning models: variants that generate extended internal chains of thought before answering. Materially better on multi-step quantitative problems; slower and costlier. Use them for hard analysis, standard models for bulk extraction and summarization.

3. Good at / bad at, calibrated for actuarial work

Strong today: use freely, with review

• Code generation and review (R, Python, SQL, VBA translation): for most actuaries the single highest-ROI use, including explaining inherited legacy code
• Summarizing and comparing documents: reinsurance treaties, policy forms, regulatory correspondence, prior analyses
• Extracting structured data from unstructured sources: loss runs in inconsistent broker formats, exposure schedules, contracts
• Drafting: report language, documentation, memo sections, filing narrative (ASOP 41 review still applies; see §6)
• Regulation orientation: "explain this state's rate filing requirements" as a starting point to verify, never a citation source

Weak or dangerous: control tightly

• Mental arithmetic and multi-step numeric manipulation without code execution
• Factual recall of specific numbers, citations, ASOP text, or policy language from memory; always retrieve the source document instead
• Anything requiring guaranteed reproducibility without an eval harness around it
• Judgment calls that are actually yours: assumption setting, method selection, reserve picks. The model can argue positions usefully (ask it to critique your assumptions, which is genuinely valuable) but it cannot own them

3A. AI agents around pricing and costing model development

The right mental model is an agent-supported model-development lifecycle, not an autonomous pricing actuary. The predictive model can still be classical and deterministic. Agents accelerate the work around it: gathering evidence, producing reproducible code, running independent challengers, testing, documenting, and monitoring.

Keep costing separate from transaction pricing. The costing layer estimates expected loss and risk costs. The pricing/rater layer then adds expenses, commission, reinsurance, capital and profit provisions, rules, and permitted underwriting judgment. An agent may trace and reconcile those components; it should not silently mix them or choose assumptions.

The useful multi-agent pattern

Freeze the data snapshot, holdout, metric definitions, and model charter first. Then fan out independent workers by model formulation or task. One agent builds the baseline GLM, one tests a GBM residual, one builds frequency/severity, one researches peer filings, and one validates every output. The orchestrator compares artifacts; it does not average recommendations, and no builder grades its own model.

This pattern compresses iteration while preserving effective challenge. Deloitte describes agentic pricing analysis that reverse-engineers complex regulatory manuals and reduces competitor analysis from weeks toward minutes (Deloitte). Aviva is testing an Actuarial Agent to enhance pricing tools with new insights and optimize existing model code in Global, Corporate & Specialty business (Aviva/hx). Hyperexponential demonstrates agents generating pricing schemas, rating logic, interfaces, parameter tables, profiling suggestions, and documentation (hx product demonstration; vendor-reported).

Non-negotiable controls

• Agents call code; they do not perform actuarial arithmetic in prose. Every result must be reproducible from versioned code and data.
• One approved model charter. Target, exposure, offsets, weights, time split, loss treatment, constraints, and success criteria are fixed before fan-out.
• One frozen holdout. All challengers are compared on identical future-period data and converted to the same loss-dollar basis.
• No autonomous assumption or rate changes. Trend, credibility, expense, reinsurance, profit, and judgment selections require named actuarial approval.
• Independent validation. A separate reviewer challenges data lineage, leakage, variable justification, stability, fairness, and implementation.
• Production is gated. The agent may draft rater code and tests; deployment requires version control, approvals, parallel-run reconciliation, and rollback.

Thin specialty data raises the value of workflow automation but lowers the credibility of unconstrained model search. Agents should help the actuary test what the current plan misses, document uncertainty, and maintain repeatable diagnostics; they should not manufacture confidence from sparse claims. The SOA's 2026 agentic-workflow research scope includes rate development, assumption development and benchmarking, model governance, documentation maintenance, and regulatory reporting, with explicit attention to reliability, explainability, bias, and human oversight (SOA Research Institute).

4. A hands-on learning path (4–6 weeks of spare-time effort)

Stage 1: Fluent prompting (week 1). Everything in Hands-on use, practiced daily on real work. Deliberately try to make the model hallucinate: ask for citations to actuarial papers, then check them. Calibrating your skepticism is the point.

Stage 2: API basics (weeks 2–3). Call a model from Python/R rather than a chat window: system prompts, temperature, structured JSON output against a schema. First project: extract a standardized claims table from three differently-formatted loss-run PDFs, with schema-constrained output, and reconcile totals against the source in code.

Stage 3: Evaluation discipline (weeks 3–4). Before trusting any pipeline, build an eval set: 30–100 examples with known correct answers, a scoring script, and a tracked accuracy metric. This is the LLM analogue of a holdout sample, and the skill that separates production work from demos (§5).

Stage 4: RAG and agents (weeks 5–6). Build a small RAG prototype over a document set you know well (e.g., your own past reports), so you can judge retrieval quality personally. Then experiment with tool use: an assistant that answers loss-development questions by writing and running code against a triangle you supply.

Stage 5: ongoing. SOA/CAS materials (see Resources), and pick one real workflow to instrument end-to-end with a colleague as reviewer.

5. Validation: treat LLM pipelines like models, because they are

For any LLM use beyond personal drafting assistance, apply model-risk thinking:

1. Define the task narrowly and the failure cost. "Summarize claim files" is not evaluable; "extract these 8 fields from loss runs with ≥98% field-level accuracy" is.
2. Build a labeled eval set from real data (with permissions). Measure before deploying, and re-measure on every model version change, prompt change, or vendor update. Model upgrades silently change behavior; pin model versions and re-run evals on upgrade.
3. Measure the right failure modes: field-level accuracy for extraction; faithfulness (does every claim in the summary trace to the source?) for summarization; retrieval hit rate and answer groundedness for RAG.
4. LLM-as-judge (using a second model to grade outputs) scales evaluation cheaply but inherits its own biases; calibrate it against human grading on a subsample first.
5. Keep humans in the loop proportionate to consequence, and monitor for automation bias: spot-audit accepted outputs, not just rejected ones.
6. Log everything: prompts, model version, outputs, reviewer decisions. This is your exam-readiness evidence under the NAIC bulletin regime (see Governance).

6. Professional standards: how this touches your credential

Nothing about the ASOPs is suspended because a model drafted the text or the code:

• ASOP 56 (Modeling): an LLM pipeline used for actuarial work is a model; you're responsible for understanding its intended purpose and limitations, and for appropriate validation and reliance. Reliance on an LLM is reliance; document it as such.
• ASOP 23 (Data Quality): LLM-extracted data is processed data; you must consider its quality. Your eval-set accuracy measurements (§5) are exactly the documentation this calls for.
• ASOP 41 (Actuarial Communications): you own every word in an actuarial communication regardless of who (or what) drafted it. Consider disclosure of material AI reliance in your documentation.
• Reproducibility: outputs are stochastic and vendors update models, so archive prompts, model versions, and outputs used in any work product supporting an actuarial opinion. Where feasible, have the LLM produce code (deterministic, re-runnable) rather than answers.

7. Data handling rules that are never optional

• Use only company-sanctioned tools and APIs (enterprise terms: no training on your data, defined retention). Never paste policyholder, claimant, or confidential company data into consumer AI tools.
• Prefer de-identified or minimal-necessary data in prompts even on sanctioned tools.
• Treat model outputs derived from confidential inputs as confidential.
• Documents from outside the company (broker submissions, claimant correspondence) can carry prompt-injection payloads: hidden instructions to the model. Pipelines that act on external documents need injection-aware design and constrained tool permissions.

Full rules are in the policy template under Governance; these are the ones analysts hit daily.