Rewilding the Index using Gemini
Using generative AI to create diverse back of the book indexes
Two of my favorite reads in the last five years were PARTS OF THE BOOK and INDEX, A HISTORY OF THE, both by Dennis Duncan. The story Duncan tells is that over the course of the modern era, “parts of the book” in the front and back matter such as prefaces, dedications, bibliographies, and indexes, have gradually become systematized, fewer, and less diverse. A similar process has taken place in the body of the book where decorations, borders, dingbats, and sketches are endangered and footnotes are domesticated as end notes. There are many reasons for this process of adverse selection, but among them must be environmental considerations like reducing the cost of book production and making simpler products that appeal to a broader audience. My position is straightforward: these trends are bad, and the diminishing biodiversity of book parts has impoverished the reading experience.
Fortunately, the advent of generative AI offers authors and publishers the opportunity to reverse this trend by “rewilding” book parts such as the abstract, the glossary, and the index. I have begun this process in my AI Lab for Book-Lovers in my ADEPT series, which adds seven to ten novel parts of the book to each new publication: tldrs, critiques, page-by-page-summaries, and more. I am introducing these new critters to the heavily cultivated habitat of today’s book world in a gradual and experimental fashion. Let a thousand flowers bloom! If I am right, book readers will begin to accept new parts of the book, at least in this experimental setting. If I am not, well, the world can keep on with the homogenized monoculture that is today’s interior book design.
The most recent royaltied book I have published by a third party is PT BOAT BASES: VOLUME 1, PACIFIC 1942-1943 by Frank J. Andruss Sr. and Ted Walther. This is quite a technical book with 100+ illustrations of every aspect of life on the bases from repair shops to medical stations.
When I started thinking about the index, my first insight was that, since the book is organized geographically by base, there would be little point in including place names in an index. Since the book is about PT boats, it was also apparent that including PT boat names in a single index would result in a giant section “P”, which would result in readers looking for people spending a lot of time flipping through the “P”s: an exercise in “p”utility, if you like. So I decided on two indexes: one for ships and boats, another for people.
Google’s Gemini 1.5, with its 1M token context window, is my favorite tool for back-of-the-book indexing.
Anyone who has created the index for a book knows it is a time-consuming task which, while not without its elements of intrigue and enjoyment, is also tedious, repetitive and persnickety. There are “pre-AI” software tools for indexing that rely on regexing PDFs (I liked Textract) but they are not free, and they still require quite a bit of user interaction and editing.
Fortunately, it turns out that large language models are great at indexing. I wrote a simple program to provide the model with chunks of text accompanied by page numbers, iterated the prompt a few times to make the output slide into my page layout format, and I was able to finished both indexes and release them from captivity in less than an hour. This is the system prompt I used:
You are an extremely competent copy editor for Nimble Books LLC. You are familiar with specialized vocabulary and style rules in every domain of book publishing. You are thoroughly familiar with the Chicago Manual of Style 17th edition. You are meticulous, accurate, and careful. While your communication with the user is professional in tone, your work output is characterized by wit and flair.
Follow these rules in creating a back-of-the-book index for this document:
Entries are last name, first name.
The first letter of an entry should be capitalized.
There should be no index headings.
Subentries should come on the next line from the entry and be preceded by the /t character. Entries and page numbers are separated by a single /t character.
Page numbers in sequence are expressed as ranges. For example: 1, 2 becomes 1-2; 1, 2, 4 becomes 1-2, 4; 1, 2, 3, 4 becomes 1-4.
In text file uploads, page numbers are normally found at the bottom of a page of text.
It is essential that page numbering in the index is accurate.
Comments:
The bit about “professional in tone” was necessary to counteract the model’s tendency to interpret “wit and flair” as an invitation to “talk like a pirate”. “Ar ar ar, here are the index entries” was a bit much. But I do want it to provide some flavor notes in the entries.
When telling an LLM to create a book index, I always use the term “back of the book index” because the scholarly literature for “indexing” is dominated by stuff about search indexes in computer science; back of the book indexing is the term that computer scientists use to talk about the indexes at the back of books. I don’t know for a fact that the model gives greater weight to scholarly literature, but I hope it does.
“No index headings” per letter (“A”, “B”, etc.) is a deviation from the CMOS; I find them unnecessary for relatively technical books.
I separate the entry and the page number by the string “/t” so that I can easily search and replace the “/t” to a tab, then convert the tabbed lines into a table in Microsoft Word.
The one instruction that the LLM had difficulty “understanding” was to express contiguous page numbers as ranges. A lot of the generated results were wrongly expressed as discrete entities, example “crew /t 1, 2, 3, 4, 5, 6” rather than “crew /t 1-6”. I experimented with a couple of fixes but did not find an effective solution in the time allotted.
I used Google’s AI Studio with Gemini 1.5 for this project. This pictorial book of 180 pages is only 36,000 tokens, so I did not need Gemini’s 1M token input maximum, and these two separate indexes each fit within its 7192 output maximum. When lengthy indexes are needed, the API can be used to implement a simple loop to accumulate entries in more than one pass. While I could have used OpenAI or Claude for this task, I have been happy with Gemini’s outputs for publishing tasks; OpenAI veers into “Golden Retriever” mode a bit too often, while Gemini stays businesslike.
Generative AI enables humans to be more creative indexers.
PT boats were powered by triple V-16 Packard marine engines: essentially, Ferraris. Like supercars, they were at the tip of a long and complex supply chain. As I reflected on this analogy, it occurred to me to ask Gemini to create an Index of Technologies. Let me emphasize that this is not a standard “part of the book”. I have seen similar topical indexes in books about the history of science and so on, but not often. And let me also emphasize that classic indexing software does not know what a “technology” is. A human must review every page of a manuscript, decide what are “technologies”, ensure terminology is consistent, and assemble a page by page list. Gemini did this for me in 45 seconds.
There are things I would quibble with in this list. Is a “Seabee” a technology? Probably not, unless you admit that human beings in general are often widgets to one another. But the main point is clear: at minimal cost, generative AI lets me give readers an interesting and useful new “part of the book.” Will topical indexes survive and multiply? I hope so. I certainly intend to experiment with more variations. Why not an Index of Sensitive Passages for a dated classic, or an Index of Personal Insults for a new memoir?
Now, instead of time and cost, the limiting factor on indexing is human imagination.
The same is true for other parts of the book. There is room for a Cambrian explosion of innovative book structures. I hope you find this as inspiring as I do. While this may seem a bit like “inside baseball” for book people, remember the magnitude of books’ impact on civilization. Refreshing the tools available to authors and publishers for making that impact can be profoundly consequential for all of us.
Cordially,
Fred Zimmerman
Publisher, Nimble Books LLC
The AI Lab for Book-Lovers
Interesting stuff