Fine-Tuning LLMs: LoRA to AWS Bedrock

LoRA with Hugging Face, how SFT compares to continued pre-training, DPO and RLHF, and a deep dive into AWS Bedrock’s reinforcement fine-tuning, supervised fine-tuning, and distillation workflows.

Fine-Tuning LLMs: LoRA to AWS Bedrock

There are two practical ways to fine-tune an LLM: train a LoRA adapter yourself with Hugging Face, or hand the job to a managed service like Bedrock. This post covers the minimum you need for the first, how supervised fine-tuning compares to continued pre-training, DPO and RLHF, how to get a LoRA model into Bedrock, and then the three Bedrock customization workflows (reinforcement fine-tuning, supervised fine-tuning, distillation) with the data format, launch call and evaluation for each.

Full Fine-Tuning vs PEFT

Full fine-tuning updates every weight in the model. It gives the most control, but it’s expensive in GPU memory and easy to overfit on small datasets. Parameter-efficient fine-tuning (PEFT) methods like LoRA (Hu et al., 2021) freeze the base model and train small low-rank adapter matrices instead, cutting trainable parameters to around 1% of the model while staying close to full fine-tuning quality on most instruction-tuning tasks.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model

model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules="all-linear",
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

target_modules="all-linear" adapts every attention and MLP projection rather than only q_proj/v_proj. See the Llama 3 8B walkthrough for why, what r and lora_alpha actually control, and a full training loop with TRL.

Before you train

• Keep examples in a consistent instruction/response format.
• Favor quality over quantity. A few thousand well-curated examples usually beat a large noisy dataset.
• Deduplicate aggressively; repeated examples skew the model more than expected.
• Track validation loss, but don’t stop there. Run side-by-side comparisons against the base model on real prompts, because loss going down doesn’t always mean the outputs are better for your task.

Start small with LoRA on a subset of the data before committing to a full fine-tune or a larger training run.

Choosing a Training Signal: SFT, Continued Pre-Training, DPO, RLHF

LoRA answers which weights get updated. It says nothing about what signal drives the update. Everything above is supervised fine-tuning (SFT): next-token loss on labeled completions. The same LoRA adapter can just as well be trained on raw text, on preference pairs, or on a reward, and the choice of signal usually matters more than the choice between LoRA and a full fine-tune.

Method	Training data	Objective	What it changes	Relative cost	Bedrock customizationType
Continued pre-training	Raw, unlabeled domain text (millions to billions of tokens)	Next-token loss on every token	Vocabulary, facts, domain style	High (data volume)	CONTINUED_PRE_TRAINING
SFT	Prompt/completion pairs (hundreds to tens of thousands)	Next-token loss on completion tokens only	Output format, task behavior, tone	Low	FINE_TUNING
DPO	Prompt + chosen + rejected responses	Logistic loss on the log-prob margin vs a frozen reference	Which of several plausible answers the model prefers	Low to medium	None
RLHF / RL	Preference comparisons for a reward model (or a programmatic grader), then prompts	Maximize reward with a KL penalty to the starting model	Same as DPO, plus behavior on prompts outside the preference set	High	REINFORCEMENT_FINE_TUNING (GRPO)

Distillation, the fourth Bedrock workflow, isn’t a separate signal: it’s SFT on responses generated by a larger teacher model.

Continued pre-training

Continued pre-training (also called domain-adaptive pre-training, see Gururangan et al., 2020) keeps running the original pre-training objective on a corpus from your domain: legal filings, clinical notes, an internal codebase. There are no prompts or labels, so the model learns the distribution of the text itself. It is the one method in this list that reliably adds knowledge, which matters because a low-rank SFT adapter is a weak tool for teaching new facts.

There are two catches. It needs far more data than SFT to move the needle, and it erodes general and instruction-following ability if the corpus is narrow. The usual recipe mixes some general-domain text back into the corpus, then runs a short SFT pass afterwards to restore chat behavior. Bedrock exposes it with {"input": "..."} records, but only for Amazon Titan text models.

DPO

Direct Preference Optimization (Rafailov et al., 2023) trains on triples of a prompt \(x\), a preferred response \(y_w\) and a rejected one \(y_l\). It starts from the same KL-constrained objective as RLHF and shows that the optimal policy can be fit directly with a single loss, with no reward model and no sampling:

\[\mathcal{L}_\text{DPO} = -\log \sigma\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_\text{ref}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_\text{ref}(y_l \mid x)}\right)\]

\(\pi_\text{ref}\) is a frozen copy of the starting model (normally your SFT checkpoint) and \(\beta\) controls how far the policy may drift from it. With LoRA you don’t need a second copy in memory: TRL computes the reference log-probs by disabling the adapter on the same base model.

from datasets import load_dataset
from peft import LoraConfig
from trl import DPOConfig, DPOTrainer

# each line: {"prompt": ..., "chosen": ..., "rejected": ...}
dataset = load_dataset("json", data_files="preferences.jsonl", split="train")

trainer = DPOTrainer(
    model=sft_model,  # start from the SFT checkpoint, not the raw base
    args=DPOConfig(output_dir="dpo-out", beta=0.1, learning_rate=5e-6, num_train_epochs=1),
    train_dataset=dataset,
    processing_class=tokenizer,
    peft_config=LoraConfig(r=16, lora_alpha=32, target_modules="all-linear", task_type="CAUSAL_LM"),
)
trainer.train()

DPO fits when it’s easier to say “this answer is better than that one” than to write the ideal answer: tone, verbosity, refusing the right things, picking the better of two summaries. It is offline, though. The model only learns from the pairs you collected, and it can lower the likelihood of both responses as long as the margin grows, so watch the chosen-response log-probs during training, not only the loss.

RLHF

The original recipe (Ouyang et al., 2022) has three stages: SFT, then a reward model trained on human rankings of sampled responses, then PPO, which samples fresh responses from the policy, scores them with the reward model, and updates the policy with a KL penalty that keeps it close to the SFT model. Because it samples on-policy, it can improve on prompts and failure modes that no preference pair covered. The price is four models in memory (policy, reference, reward, value), many more hyperparameters, and training runs that are notoriously unstable.

Most teams now reach for DPO first and keep online RL for cases where the reward can be computed rather than learned. That is the niche Bedrock’s reinforcement fine-tuning targets. It uses GRPO (Shao et al., 2024), which drops PPO’s value model by scoring each response against the other samples for the same prompt, and the reward comes from a verifiable grader (RLVR) or an LLM judge (RLAIF, the AI-feedback variant of RLHF) instead of a reward model trained on human labels.

Picking one

• The model doesn’t know your domain: continued pre-training, then SFT.
• You can write the ideal outputs: SFT.
• You can only rank outputs, not write them: DPO on top of SFT.
• You can score outputs automatically (tests pass, answer matches, schema validates): RL with GRPO, on Bedrock or with TRL’s GRPOTrainer.

These stack in that order, and LoRA works at every stage.

From a LoRA Adapter to Bedrock

If you trained the adapter yourself, Custom Model Import is how it reaches Bedrock. It accepts Llama, Mistral, Qwen and a few other architectures as Hugging Face safetensors (fp16, bf16 or fp32), and it takes full weights, so the adapter has to be merged into the base first:

merged = model.merge_and_unload()
merged.save_pretrained("llama3-8b-support", safe_serialization=True)
tokenizer.save_pretrained("llama3-8b-support")
# then: aws s3 sync llama3-8b-support s3://my-bucket/cmi/llama3-8b-support/

client = boto3.client("bedrock")
client.create_model_import_job(
    jobName="import-llama3-8b-support",
    importedModelName="llama3-8b-support",
    roleArn="arn:aws:iam::<account-id>:role/BedrockImportRole",
    modelDataSource={"s3DataSource": {"s3Uri": "s3://my-bucket/cmi/llama3-8b-support/"}},
)

Once the job completes, call the imported model’s ARN through bedrock-runtime like any other model. The merge section of the Llama 3 post covers the merge step in more detail. The alternative is to skip local training and let Bedrock run it, which is what the rest of this post is about.

AWS Bedrock Model Customization

Bedrock model customization exposes several workflows through one API, CreateModelCustomizationJob, selected by customizationType: FINE_TUNING, CONTINUED_PRE_TRAINING, DISTILLATION and, since December 2025, REINFORCEMENT_FINE_TUNING. This section covers RFT, SFT and distillation. Training data is read from S3 as .jsonl. The resulting custom model is stored by Bedrock in your account; your output S3 prefix only receives training metrics (and, for distillation, samples of the synthetic data).

Reinforcement fine-tuning (new)

How it works

The model generates several candidate responses per prompt, a reward function scores each one, and Bedrock updates the model with GRPO to prefer higher-scoring responses (see RLHF above for how this differs from PPO). There’s no gradient step against a single “correct” answer; the signal is relative quality across the sampled candidates. That suits tasks where correctness is graded rather than binary: multi-step reasoning, code generation, tool use, or open-ended writing where several answers are valid but some are clearly better.

Reward functions come in two flavors:

• RLVR (verifiable rewards): a rule-based grader for objective tasks (math, code execution, format checking).
• RLAIF (AI feedback): an LLM-as-judge grader for subjective tasks (instruction following, tone, RAG faithfulness). The console offers built-in judge prompt templates and turns them into a Lambda for you; through the API, both flavors are a lambdaGrader.

At launch (December 2025), RFT supports Amazon Nova 2 Lite in us-east-1.

Data format

Each record is an OpenAI-style chat-completion object with a reference_answer that the reward function grades against. It can be any structure your grader understands, not just text. A minimum of 100 records is required, and extra fields (e.g. difficulty_level, domain) are passed through to the grader as metadata:

{
  "id": "algebra_001",
  "messages": [
    {"role": "system", "content": "You are a math tutor"},
    {"role": "user", "content": "Solve: 2x + 5 = 13"}
  ],
  "reference_answer": {"solution": "x = 4", "steps": ["2x = 8", "x = 4"]},
  "difficulty_level": "easy",
  "domain": "algebra"
}

Reward function (Lambda grader)

A custom grader receives the full conversation plus your reference_answer in metadata, and must return a normalized reward per sample. Raise the Lambda timeout above the 3-second default if your scoring runs code or calls a judge model:

def lambda_handler(event, context):
    results = []
    for sample in event:
        response_text = sample["messages"][-1]["content"]
        reference = sample["metadata"]["reference_answer"]

        # your scoring logic: exact match, execution check, semantic diff, etc.
        score = score_response(response_text, reference)

        results.append({
            "id": sample["id"],
            "aggregate_reward_score": score,   # normalized, e.g. 0.0-1.0
            "metrics_list": [
                {"name": "accuracy", "value": score, "type": "Reward"}
            ],
        })
    return results

Launching a job (boto3)

import boto3

client = boto3.client("bedrock", region_name="us-east-1")

response = client.create_model_customization_job(
    jobName="rft-math-tutor",
    customModelName="nova-lite-math-rft",
    roleArn="arn:aws:iam::<account-id>:role/BedrockCustomizationRole",
    baseModelIdentifier="amazon.nova-2-lite-v1:0:256k",
    customizationType="REINFORCEMENT_FINE_TUNING",
    trainingDataConfig={"s3Uri": "s3://my-bucket/rft/train.jsonl"},
    validationDataConfig={"validators": [{"s3Uri": "s3://my-bucket/rft/val.jsonl"}]},
    outputDataConfig={"s3Uri": "s3://my-bucket/rft/output/"},
    customizationConfig={
        "rftConfig": {
            "graderConfig": {
                "lambdaGrader": {"lambdaArn": "arn:aws:lambda:us-east-1:<account-id>:function:math-grader"}
            },
            "hyperParameters": {"epochCount": 3, "trainingSamplePerPrompt": 4},
        }
    },
)

Evaluation metrics

Training emits average reward and validation reward curves per checkpoint (validation reward normally trails training reward; if training reward climbs while validation reward falls, you’re overfitting), plus validation episode length to track response verbosity. Beyond the training curves, Bedrock lets you A/B the RFT model against the base model with the same prompt set in the Playground, or run a full Model Evaluation job with your own test set for a systematic before/after comparison.

Supervised fine-tuning

How it works

This is the SFT row of the table above, run on Bedrock’s infrastructure. It’s the right default when you already have good examples of the exact output you want.

Data format

For non-conversational tasks it’s flat prompt/completion pairs:

{"prompt": "Summarize the article about climate change.", "completion": "Climate change refers to the long-term alteration of temperature and typical weather patterns in a place."}

Chat formats differ by model family. Claude 3 Haiku takes a system string plus messages:

{
  "system": "You are a helpful assistant.",
  "messages": [
    {"role": "user", "content": "What is AWS?"},
    {"role": "assistant", "content": "It's Amazon Web Services."}
  ]
}

Converse-format models (Llama 3.2/3.3, Nova) need the bedrock-conversation-2024 schema with content arrays instead, so check the per-model data requirements before you build the file. Minimums also vary by model (32 records for Claude 3 Haiku, 100 for Llama 3.1/3.3), and the default cap is 10,000 training records, adjustable via service quota.

Launching a job (boto3)

Claude 3 Haiku fine-tuning runs in us-west-2, and it uses a learning-rate multiplier rather than an absolute learning rate:

client = boto3.client("bedrock", region_name="us-west-2")

response = client.create_model_customization_job(
    jobName="sft-support-tone",
    customModelName="haiku-support-v1",
    roleArn="arn:aws:iam::<account-id>:role/BedrockCustomizationRole",
    baseModelIdentifier="anthropic.claude-3-haiku-20240307-v1:0:200k",
    customizationType="FINE_TUNING",
    trainingDataConfig={"s3Uri": "s3://my-bucket/sft/train.jsonl"},
    validationDataConfig={"validators": [{"s3Uri": "s3://my-bucket/sft/val.jsonl"}]},
    outputDataConfig={"s3Uri": "s3://my-bucket/sft/output/"},
    hyperParameters={"epochCount": "3", "batchSize": "8", "learningRateMultiplier": "1.0"},
)

Evaluation metrics

Bedrock writes training and validation loss curves to your output S3 prefix. They’re useful for spotting under- or overfitting but say nothing about task quality. Pair them with a Bedrock Model Evaluation job on held-out prompts: its built-in accuracy metrics are BERTScore for summarization and F1 for QA. Compute ROUGE yourself if you need it, and for open-ended output add an LLM-as-judge score and a manual side-by-side against the base model.

Distillation

How it works

A large teacher model generates (or already has, via invocation logs) responses to your prompts, and those responses fine-tune a smaller, cheaper student model. Teacher and student must be one of the pairs AWS supports, all within one model family (e.g. Nova Pro to Nova Lite, Llama 3.1 405B to Llama 3.1 8B); you can’t distill across families. Bedrock can also apply its own data-synthesis techniques, such as paraphrasing prompts, to expand a small seed set to at most 15k prompt-response pairs before training the student. AWS claims distilled models are up to 500% faster and up to 75% cheaper, with less than 2% accuracy loss on use cases like RAG (Bedrock Model Distillation).

Data format

The input is prompts only, in the bedrock-conversation-2024 schema. Add an assistant turn to some records if you want to give the teacher golden examples:

{"schemaVersion": "bedrock-conversation-2024", "system": [{"text": "You classify review sentiment."}], "messages": [{"role": "user", "content": [{"text": "Classify the sentiment of this review: 'The battery life is incredible.'"}]}]}

Alternatively, reuse invocation logs already stored in S3 (with model invocation logging enabled), filtered by the requestMetadata you attached at invoke time:

trainingDataConfig={
    "invocationLogsConfig": {
        "invocationLogSource": {"s3Uri": "s3://my-bucket/invocation-logs/"},
        "requestMetadataFilters": {"equals": {"project": "CustomerService"}},
        "usePromptResponse": True,
    }
}

With usePromptResponse=True, Bedrock trains on the logged responses instead of re-querying the teacher, so you pay no new teacher inference. This only works if the teacher in the job is the model that produced the logs.

Launching a job (boto3)

client = boto3.client("bedrock", region_name="us-east-1")

response = client.create_model_customization_job(
    jobName="distill-support-classifier",
    customModelName="nova-lite-support-distilled",
    roleArn="arn:aws:iam::<account-id>:role/BedrockCustomizationRole",
    baseModelIdentifier="amazon.nova-lite-v1:0:300k",  # student model
    customizationType="DISTILLATION",
    trainingDataConfig={"s3Uri": "s3://my-bucket/distill/prompts.jsonl"},
    outputDataConfig={"s3Uri": "s3://my-bucket/distill/output/"},
    customizationConfig={
        "distillationConfig": {
            "teacherModelConfig": {"teacherModelIdentifier": "amazon.nova-pro-v1:0"}
        }
    },
)

Evaluation metrics

Because the whole point is a favorable accuracy/latency/cost trade, evaluate the student against the teacher on all three axes rather than accuracy alone: run a Bedrock Model Evaluation job with the same test prompts through both models to get a direct accuracy delta, then separately track p50/p90 latency and cost per 1K tokens for the student versus teacher to confirm the trade you’re accepting is the one you intended.

Serving the custom model

None of these jobs leaves you with something you can call yet. A customized model needs either Provisioned Throughput or an on-demand custom model deployment before bedrock-runtime will accept it, and that choice, not the training job, is usually where the bill grows. Check the inference options for custom models before you commit to a workflow.