Split Text for Text to Speech in Python: A Sentence-Aware Chunker with Tests

You've got a long narration script in markdown and a text-to-speech API that takes text per request, with a character limit. Cutting the script every N characters breaks sentences, abbreviations and inline code. Below is a small stdlib-only Python chunker that splits at sentence boundaries under a configurable limit, plus unittest tests and real output comparing it with the naive splitter on the same input.

Some links in this post are affiliate links; see our disclosure.

Nothing here calls a TTS service. I didn't generate any audio, so I'm not making claims about how anything sounds. This is the text-preparation step that sits before the API call.

Why a fixed-size split isn't enough for markdown scripts

Deepgram's chunking guide lists splitting at a fixed maximum character count first, and its own example sets MAX_CHUNK_SIZE = 200. It also describes sentence-boundary splitting. What I wanted was sentence-boundary splitting that survives a markdown script: e.g., Dr., initials, decimals like 30.5, and inline code with periods inside. A split on . would cut at every one of those. The tests below pin those cases down.

The limit is a parameter, not a constant

Every provider sets its own per-request cap. I opened the ElevenLabs text-to-speech API reference to look for one. The text field is described only as "The text that will get converted into speech", with no number on that page. So I'm not quoting a limit. Read yours from your provider's docs and pass it in. The function takes limit as an argument, and the demo uses 300 characters only because the sample script is short.

If you're deciding between a hosted API and a local engine, our Piper vs ElevenLabs comparison covers that.

The rules the chunker follows

1. Split paragraphs on blank lines first. Chunks never span a paragraph break.
2. Inside a paragraph, find sentence-end candidates: ., ! or ? (plus closing quotes or brackets), whitespace, then something that looks like a sentence start (capital, digit, quote, bracket, backtick).
3. Reject a candidate if it falls inside a code span (inline backticks or a fenced block).
4. Reject a . candidate if the word before it is a known abbreviation or a single-letter initial.
5. Pack whole sentences greedily into a chunk until the next one would exceed limit.
6. If a single sentence is longer than limit, fall back to splitting on words, keeping code spans whole. If a single code span is longer than limit, raise ValueError rather than silently cut it.

Candidates ending in ! or ? can't be abbreviations, so only periods get the abbreviation lookup.

Python chunker that never splits mid-sentence

The demo uses four files in one directory: chunker.py, test_chunker.py, demo.py and sample_script.md. All commands below run from that directory. I used Python 3.12.10; the code uses nothing newer than f-strings. Save this as chunker.py (75 lines):

"""Sentence-aware chunker for TTS narration scripts. Stdlib only."""
import re

ABBREVIATIONS = {
    "e.g.", "i.e.", "etc.", "vs.", "cf.", "approx.", "no.", "fig.",
    "dr.", "mr.", "mrs.", "ms.", "prof.", "sr.", "jr.", "st.",
}

FENCE = "`" * 3
_PROTECTED = re.compile(re.escape(FENCE) + r".*?" + re.escape(FENCE) + r"|`[^`\n]*`", re.S)
_CANDIDATE = re.compile(r"""([.!?]+["')\]]*)(\s+)(?=["'(\[`A-Z0-9])""")
_WORDS = re.compile(r"`[^`\n]*`\S*|\S+")


def _is_abbreviation(text, end):
    """True if the token that ends at text[:end] is an abbreviation or an initial."""
    token = text[:end].split()[-1].lstrip("(\"'[").lower()
    return token in ABBREVIATIONS or re.fullmatch(r"[a-z]\.", token) is not None


def split_sentences(text):
    protected = [m.span() for m in _PROTECTED.finditer(text)]
    sentences, start = [], 0
    for m in _CANDIDATE.finditer(text):
        punct_end = m.end(1)
        if any(a < punct_end <= b for a, b in protected):
            continue
        if m.group(1) == "." and _is_abbreviation(text, punct_end):
            continue
        sentences.append(text[start:punct_end])
        start = m.end()
    if text[start:].strip():
        sentences.append(text[start:])
    return [s.strip() for s in sentences if s.strip()]


def _split_long(sentence, limit):
    """Last resort for one sentence over the limit: break on words, keeping code spans whole."""
    pieces, current = [], ""
    for word in _WORDS.findall(sentence):
        candidate = f"{current} {word}".strip()
        if len(candidate) > limit and current:
            pieces.append(current)
            current = word
        else:
            current = candidate
    if current:
        pieces.append(current)
    return pieces


def chunk_script(text, limit=500):
    """Pack whole sentences into chunks of at most `limit` characters."""
    if limit < 1:
        raise ValueError("limit must be positive")
    chunks = []
    for paragraph in re.split(r"\n\s*\n", text.strip()):
        current = ""
        for sentence in split_sentences(" ".join(paragraph.split()) if FENCE not in paragraph else paragraph):
            for piece in ([sentence] if len(sentence) <= limit else _split_long(sentence, limit)):
                if len(piece) > limit:
                    raise ValueError(f"unsplittable piece of {len(piece)} chars: {piece[:40]!r}")
                candidate = f"{current} {piece}".strip()
                if len(candidate) <= limit:
                    current = candidate
                else:
                    chunks.append(current)
                    current = piece
        if current:
            chunks.append(current)
    return chunks


def naive_chunks(text, limit=500):
    return [text[i:i + limit] for i in range(0, len(text), limit)]

Three details that cost me thinking time:

• The fence marker is built, not typed. FENCE = "" * 3` keeps the module free of a literal triple-backtick, which would break the markdown if you paste the code into a post.
• Whitespace collapsing flattens fenced blocks. Paragraphs are normalised with " ".join(paragraph.split()) so line wraps in the markdown don't leak into the spoken text. That would also flatten a fenced code block's newlines, so paragraphs containing a fence skip it.
• The abbreviation check looks at the original text up to the period, not the sentence so far, so (e.g. with a leading bracket still matches after lstrip.

A sample script that tries to break it

sample_script.md is 1,215 characters of narration with the usual hazards: e.g., i.e., etc., Dr., Mr., the initials J. R., the decimal 30.5, an inline call and an inline JSON config. I wrote it for this test; it isn't a real episode. Here it is in full:

# Episode 12: Shipping a webhook retry queue

Welcome back. Today we're building a retry queue for failed webhooks, e.g. Stripe or GitHub deliveries that time out. Dr. Chen from the platform group asked for this last week, and it's simpler than it sounds.

First, call `queue.enqueue(job, delay=30.5)` whenever a delivery fails. The value 30.5 is seconds, not milliseconds! Why does that matter? Because most retry libraries, i.e. the ones I've used, default to milliseconds and you'll wait 8 hours instead of 30 seconds.

Next, store the attempt count. Don't retry forever, etc. Cap it at 6 attempts, then move the job to a dead-letter table. Mr. Okafor's rule of thumb: if it failed 6 times, a human needs to look at it.

Here is the config you'll paste in, and it's worth reading slowly: `{"max_attempts": 6, "backoff": "exponential", "base_delay_seconds": 30, "jitter": true}`. Notice that jitter is on. Without it, every failed job retries at the same instant and you get a thundering herd. J. R. Smith wrote about this in 2019.

Finally, add a metric for queue depth. If depth grows for ten minutes straight, page someone. That's the whole system. See the repo for the full example, and thanks for listening.

And demo.py, which prints stats for both splitters and, with -v, dumps every chunk:

import re, sys
from chunker import chunk_script, naive_chunks, split_sentences

text = open("sample_script.md", encoding="utf-8").read()
limit = int(sys.argv[1]) if len(sys.argv) > 1 else 300
ABBR = re.compile(r"\b(e\.g|i\.e|etc|Dr|Mr|J|R)\.?$")

def report(name, chunks):
    ends = sum(1 for c in chunks if re.search(r"[.!?][\"')\]]*$", c.rstrip()))
    odd = sum(1 for c in chunks if c.count("`") % 2)
    abbr = sum(1 for c in chunks if ABBR.search(c.rstrip()))
    print(f"{name:<6} chunks={len(chunks):<3} max_len={max(map(len, chunks)):<4} "
          f"not_ending_at_sentence={len(chunks)-ends:<3} unbalanced_backticks={odd:<3} ending_on_abbrev={abbr}")

print(f"input chars={len(text)} sentences={len(split_sentences(text))} limit={limit}")
smart, naive = chunk_script(text, limit), naive_chunks(text, limit)
report("smart", smart); report("naive", naive)
if "-v" in sys.argv:
    for i, c in enumerate(smart, 1): print(f"--- smart {i} ({len(c)})\n{c}")
    for i, c in enumerate(naive, 1): print(f"--- naive {i} ({len(c)})\n{c!r}")

Run python demo.py 300. A chunk counts as "not ending at sentence" if it doesn't end in ., ! or ?; ending_on_abbrev only checks the abbreviations in the demo's ABBR regex (e.g, i.e, etc, Dr, Mr, J, R):

input chars=1215 sentences=18 limit=300
smart  chunks=7   max_len=271  not_ending_at_sentence=1   unbalanced_backticks=0   ending_on_abbrev=0
naive  chunks=5   max_len=300  not_ending_at_sentence=4   unbalanced_backticks=2   ending_on_abbrev=0

The one "bad" smart chunk is the # Episode 12: ... heading, which has no terminal punctuation. That's legitimate; a heading is its own unit. The naive splitter has fewer chunks and a higher max length, but four of its five chunks end mid-sentence and two cut through an inline code span. These are the first two naive chunks from python demo.py 300 -v, unabridged (Python repr, so \n is a newline):

naive 1 (300)
"# Episode 12: Shipping a webhook retry queue\n\nWelcome back. Today we're building a retry queue for failed webhooks, e.g. Stripe or GitHub deliveries that time out. Dr. Chen from the platform group asked for this last week, and it's simpler than it sounds.\n\nFirst, call `queue.enqueue(job, delay=30.5)"
naive 2 (300)
"` whenever a delivery fails. The value 30.5 is seconds, not milliseconds! Why does that matter? Because most retry libraries, i.e. the ones I've used, default to milliseconds and you'll wait 8 hours instead of 30 seconds.\n\nNext, store the attempt count. Don't retry forever, etc. Cap it at 6 attempts"

The backtick that opens the code span ends request 1 and the one that closes it starts request 2. Whatever you do to code spans (strip them, read them aloud, replace them with a phrase), that now can't work.

And the sentence-aware output, in full:

--- smart 1 (44)
# Episode 12: Shipping a webhook retry queue
--- smart 2 (209)
Welcome back. Today we're building a retry queue for failed webhooks, e.g. Stripe or GitHub deliveries that time out. Dr. Chen from the platform group asked for this last week, and it's simpler than it sounds.
--- smart 3 (264)
First, call `queue.enqueue(job, delay=30.5)` whenever a delivery fails. The value 30.5 is seconds, not milliseconds! Why does that matter? Because most retry libraries, i.e. the ones I've used, default to milliseconds and you'll wait 8 hours instead of 30 seconds.
--- smart 4 (199)
Next, store the attempt count. Don't retry forever, etc. Cap it at 6 attempts, then move the job to a dead-letter table. Mr. Okafor's rule of thumb: if it failed 6 times, a human needs to look at it.
--- smart 5 (271)
Here is the config you'll paste in, and it's worth reading slowly: `{"max_attempts": 6, "backoff": "exponential", "base_delay_seconds": 30, "jitter": true}`. Notice that jitter is on. Without it, every failed job retries at the same instant and you get a thundering herd.
--- smart 6 (37)
J. R. Smith wrote about this in 2019.
--- smart 7 (179)
Finally, add a metric for queue depth. If depth grows for ten minutes straight, page someone. That's the whole system. See the repo for the full example, and thanks for listening.

Dr., Mr., e.g., i.e. and J. R. all stayed inside their sentences, the JSON config stayed whole, and no chunk went over 300. etc. is the exception: the period in "forever, etc." ends that sentence, but the chunker kept "Don't retry forever, etc. Cap it at 6 attempts, ..." together (see Gotchas).

How the limit changes the numbers

I ran python demo.py N at four limits:

The smart splitter ended on an abbreviation zero times at every limit. The naive one did at 200, 500 and 1000. That is the strongest argument in the table: a chunk ending on one of those abbreviations hands the API a sentence cut off right after it.

At 500 and 1000 the smart count is the same 6 chunks: one for the heading and one per paragraph. The whole script already fits sentence-packed per paragraph, so the larger limits add nothing. The naive splitter happened not to hit a backtick at 500 and 1000; that's alignment on this one script, not safety. At 200 and 300 it cut a code span both times.

Smart chunk counts exceed naive counts at small limits. You pay for whole sentences with a few more, slightly shorter requests. Whether that's a cost depends on how your provider bills; check its pricing page. I haven't looked at any prices for this post.

Unit tests for the sentence splitter

Thirteen tests, all passing (python -m unittest, 0.004 s on this machine). Here are the ones that earn their keep, with the file header and the shared TEXT:

import unittest
from chunker import chunk_script, split_sentences, naive_chunks


class SplitSentences(unittest.TestCase):
    def test_abbreviations_do_not_split(self):
        s = split_sentences("Use a queue, e.g. Redis. Dr. Chen agrees.")
        self.assertEqual(s, ["Use a queue, e.g. Redis.", "Dr. Chen agrees."])

    def test_code_span_is_atomic(self):
        text = "Call `a. B. c` now. Then stop."
        self.assertEqual(split_sentences(text), ["Call `a. B. c` now.", "Then stop."])


class ChunkScript(unittest.TestCase):
    TEXT = ("Alpha one is here. Beta two follows it. Gamma three arrives, e.g. now. "
            "Delta `x. Y` four ends.\n\nNew paragraph starts here. It has two sentences.")

    def test_limit_respected(self):
        for limit in (30, 45, 80, 200):
            for c in chunk_script(self.TEXT, limit):
                self.assertLessEqual(len(c), limit)

    def test_no_text_lost(self):
        words = lambda s: " ".join(s.split())
        self.assertEqual(words(" ".join(chunk_script(self.TEXT, 45))), words(self.TEXT))

    def test_unsplittable_code_span_raises(self):
        with self.assertRaises(ValueError):
            chunk_script("See `" + "x" * 60 + "` here.", 40)

The other eight cover basic splitting, decimals, initials, chunks ending at sentence ends, balanced backticks at several limits, the word-level fallback for a long sentence, a non-positive limit, and a check that the naive splitter does leave chunks ending mid-sentence. That last one failed on its first run: I'd written it to look for odd backtick counts at limit 45, and this text doesn't happen to cut a span there. I changed it to test for mid-sentence endings instead.

test_no_text_lost is the one I'd keep if I could keep only one. A chunker can drop a character at a boundary without raising anything, and reading the output won't show it.

Gotchas I'd flag

• etc. followed by a capital is ambiguous. In "Don't retry forever, etc. Cap it at 6 attempts" the period ends the sentence, but my abbreviation list refuses to split there, so the two sentences stay together in one unit. That's the safer error for TTS (a slightly long chunk instead of a clipped one), but it is an error. Fix it by adding etc. handling only when the next word is lowercase, if that matters for your scripts.
• The abbreviation list is English and incomplete. Add your domain's: Inc., Ltd., U.S.. Multi-letter dotted forms like U.S. aren't covered by the initials rule, which only matches one letter plus a period.
• Sentence-start detection needs a capital, digit, quote, bracket or backtick. A sentence that starts lowercase after a period (rare in narration) won't split. It stays in the chunk before it.
• Markdown formatting is passed through. **bold**, links and headings stay in the text. Strip them before chunking if your provider reads them aloud.
• Check what your provider's limit counts. The function counts Python characters; if your provider's docs say bytes or tokens, convert before choosing limit.

When to write your own

Write your own when your input is markdown with code and you want to control the exceptions; it's about 75 lines and the tests are the real asset. I haven't tried any splitting library on this sample, so I can't compare them. Run test_code_span_is_atomic and test_abbreviations_do_not_split against whatever you choose; if it passes both, it handles the two cases that broke the naive splitter here.

What to do next

• Copy chunker.py into your repo, write the tests above into test_chunker.py, and run python -m unittest.
• Look up your TTS provider's per-request limit in its docs, and confirm whether it counts characters or something else. Pass that value (minus some margin) as limit.
• Add your own abbreviations and run the chunker on one real script. Print chunks that don't end in ., ! or ? and read them.
• Decide how code spans should be handled before synthesis: strip, rewrite as words, or skip. Do it after chunking, so the spans arrive whole.
• Send one chunk to your provider and check the result by ear. I haven't done that, so that step is yours. If you don't have an account yet, ElevenLabs is one option.