Prompt Engineering for Local Language Models

Patterns and anti-patterns for gemma3, qwen, llama and similar local LLMs

Article last updated on 25 June 2026

Key insight: Local language models can do pattern matching but only limited inference. Phrase every instruction so that a first-day junior could follow it mechanically without domain expertise.

Cloud LLM vs. Local LLM

Aspect	Cloud LLM (GPT-4, Claude)	Local LLM (gemma3:27b)
Implicit inference	Works	Unreliable
Vague instructions	Usually interpreted sensibly	Ignored or misinterpreted
Zero-shot instructions	Often sufficient	Need few-shot examples
Self-verification	Helps	Often degrades the result
Conditional behavior	Self-inference sufficient	Paths must be spelled out explicitly as IF-THEN
Format adherence	High	Reliable only with explicit templates

IF-THEN Structuring: The most important lever

When the correct output depends on an input parameter, spell out the path explicitly as an IF-THEN rule. Local models cannot derive this decision on their own.

Before: Universal instruction

Schreibe einen Lösungsvorschlag für den Bescheid. Berücksichtige die Anspruchsstruktur und argumentiere passend.

Score: 3.62 (gemma3:27b, 5 runs)

After: IF-THEN structuring

IF nur ein unabhängiger Anspruch betroffen ist: Argumentiere direkt gegen die Entgegenhaltung. IF mehrere unabhängige Ansprüche betroffen sind: Gliedere die Argumentation nach Ansprüchen. IF ein Hilfsantrag nötig ist: Schlage konkrete Einschränkungen aus den abhängigen Ansprüchen vor.

Score: 5.0 (gemma3:27b, 5 runs)

Generic template

IF [condition A]: [Concrete action for A] IF [condition B]: [Concrete action for B] ELSE: [Fallback action]

When IF-THEN is useful

Input parameter	IF-THEN useful?
Number of independent claims	Yes
Presence of figures	Yes
Document language	Yes
Number of prior art references	Yes
Stylistic preferences	No, better via few-shot examples

Conditioning Trade-off

The more strongly a prompt pushes toward a specific source, the more likely the model gives up when the source is missing. The solution: spell out both paths explicitly.

Phrasing	Source present	Source missing
Weak: “Use the description”	Partially uses source	Hallucinates freely
Strong: “Exclusively from the description”	Reliably uses source	Returns empty or minimal output
Conditional: “IF description present: quote from it. ELSE: own formulation”	Reliably uses source	Fallback behavior defined

Recommendation: Always choose the conditional variant. The extra effort is minimal, the robustness significantly higher.

Baseline-First Test

Before any optimization, measure 3 baseline runs. If the score is already high and stable, optimization is not worth it; it can even degrade the result.

Baseline result	Recommendation
Score ≥ 4.5 and variance < 0.3	Hands off, prompt is good enough
Score ≥ 4.0 and variance ≥ 0.3	Stabilization needed (IF-THEN, few-shot)
Score < 4.0	Fundamental revision needed
Score < 3.0	Rebuild prompt structure from scratch

Score ≠ Quality

Heuristic scoring (e.g., by an evaluator LLM) can be deceptive. Always run out-of-distribution tests, i.e., use inputs that differ from the usual test set.

What the heuristic measures	What it misses
Structural completeness	Factual correctness
Template conformity	Patent law plausibility
Keyword matching	Argumentation logic
Length compliance	Technical depth and nuance

Rule of thumb: A score increase without a simultaneous out-of-distribution test is worthless.

Golden References

Attorney-drafted example texts beat any heuristic. The model imitates patterns, and good patterns yield good outputs.

Task	Golden reference
Office action response	Real response from the firm (anonymized)
Claim drafting	Granted claims from comparable patents
Figure description	Description section from published application
Abstract	Abstracts from the same technical field

Golden references are embedded as few-shot examples in the prompt or used as evaluation basis for scoring.

Anti-Repetition for Continuation

During Ctrl+Space completion (continuation), local models frequently repeat the last word or sentence. An explicit instruction prevents this.

Prompt snippet

WICHTIG: Setze den Text nahtlos fort. Wiederhole NICHT das letzte Wort oder den letzten Satz des bisherigen Textes. Beginne direkt mit neuem Inhalt.

This snippet belongs in the system prompt of every continuation function. Without it, gemma3 and qwen produce repetitions in ~40% of cases.

Synchronize Stop Tokens

When the prompt uses section headers (e.g., ## Next Section), these must be added to the Ollama stop list. Otherwise the model generates beyond the desired section.

Symptom	Cause
Model generates next section too	Section header missing from stop list
Output breaks off mid-sentence	Stop token too short / ambiguous
Empty output with new prompt versions	New header accidentally used as stop token at the start

gemma4 Thinking Trap

gemma4 is a thinking model. For structured prompts, thinking must be disabled, but the mechanism differs from other models.

Note: gemma4 requires think: false as an API parameter in the Ollama request. The /no_think prefix from qwen3 does not work with gemma4.

Without think: false, gemma4 frequently produces empty or incomplete outputs with structured prompts because the thinking budget displaces the actual answer.

More on the gemma4 architecture: MoE Architecture →

Explicit knowledge over implicit

Naming a provision is not the same as supplying its content. “Argue under Art. 54, 56, 84, 123(2) EPC” assumes the model knows what each provision requires. Large cloud models usually carry that mapping in their weights; smaller local models swap novelty and inventive step, or invent a plausible-sounding requirement. Name the provision, but spell out the actual requirement next to it.

Before: citation only

You argue precisely under the EPC (Art. 54, 56, 84, 123(2)).

After: citation + requirement

For each objection, name the provision together with its requirement: - Novelty (Art. 54): no single document discloses all features. - Inventive step (Art. 56): not obvious from the prior art. - Clarity (Art. 84): the claim is clear, concise and supported by the description. - No added subject-matter (Art. 123(2)): nothing extends beyond the original disclosure.

Ground rule: Never rely on implicit knowledge alone in a prompt. Definitions, standards and test schemes that the model is meant to apply belong explicitly in the context, not just their citation. This does not replace the duty of evidence: the human still checks the output against the statutory text.

Anti-Patterns: What does not work

Anti-pattern	Why it hurts
Self-verify (“Check your output”)	Filters out real findings instead of correcting errors
Role boilerplate (“You are an experienced patent attorney”)	No measurable effect with local models
Open avoidance rules (“Avoid repetitions”)	Ignored, concrete IF-THEN rules needed
Implicit style instructions (“Write professionally”)	Model knows no conventions, use few-shot examples
Long justifications before the task	Recency bias, model weighs the end of the prompt more heavily

Practical Recommendations

Make every condition explicit. If the result depends on the input, use IF-THEN. Never trust the model to make the right decision on its own.
Measure baseline before optimizing. 3 runs with the unchanged prompt. If the score is fine, don't touch it.
Collect golden references. Every well-drafted piece of firm work is a potential few-shot example. Anonymize and add to the prompt library.
Test out-of-distribution. A prompt that works for mechanical engineering patents can fail for software patents. Always test with divergent inputs.
Place the task at the end of the prompt. Context and examples first, the actual instruction last. This uses the recency bias to your advantage.

Content partially AI-generated, curated by Sebastian Goebel. This is not legal advice but training material for my workshops. No guarantee of accuracy or completeness. No liability. Software provided as-is.