Running an LLM On-Prem: What a Community Bank Actually Needs

Every community bank has been asked the same question by its board in the last year: what are we doing about AI? And every CIO has given a version of the same answer: carefully.

The careful answer is right. A bank cannot paste a trust document into a public chatbot. It cannot sign a cloud AI vendor’s standard terms and call vendor management done. And it cannot wait five years while the credit union across town answers customer questions in seconds.

There is a third way, and it is the one we are building right now for Exchange Bank, a California community bank: run the model on the bank’s own hardware, inside the bank’s own building, so no customer data ever leaves the perimeter. This post is about what that actually involves, written for the CIO who has to put a number and a plan in front of the board. The engagement itself is under NDA, so what follows is the general shape of the work, not the bank’s specifics.

Start with the task, not the model

The vendor conversation starts with GPUs. The useful conversation starts with a question: which job, done by which people, takes the most time today and involves reading documents?

In a bank it is almost always document review. Trust administration, loan file completeness, compliance checks against a policy, answering the same twenty questions about a two-hundred-page instrument. A person reads, extracts, checks and writes it up. That is the work a language model is good at, and the work where “90% right, with a human checking” is still a large gain, because the human was going to read it anyway.

Pick one such process. Write down the twenty or thirty questions a person answers when they do it. That list is the specification for everything that follows: the model has to answer those questions, on those documents, well enough that the reviewer trusts it. If you cannot write the list, you are not ready to size hardware.

The thing nobody tells you: the model cannot read your documents

Large language models read text. A great deal of what a bank holds is not text: it is a scan of a signed instrument, a fax of a fax, a PDF that is secretly a picture. A model given that file sees nothing.

So the first component of an on-prem AI system is not the model. It is the pipeline that turns scanned documents into clean text, with the layout preserved well enough that a table stays a table and a signature block stays where it was. Optical character recognition has existed for decades; the recent generation of document-reading models is far better at it, and some of them can also run on your own hardware.

This matters for sizing because it is a second workload, with its own GPU demand, and it matters for the project because it is where the quality is decided. A superb language model fed bad text produces confident nonsense. Budget for the document pipeline as a deliverable in its own right, and test it on your ugliest scans first.

Hardware: size it after you have tested, not before

Here is the honest version of the GPU question.

A model’s size is measured in parameters, and the parameters have to fit in GPU memory, with room left over for the document it is reading. A small open-weight model fits on one professional card. A large one needs several cards, or a smaller version of itself that trades some quality for a fraction of the memory. Which one you need is not knowable from a spec sheet; it is knowable from putting two or three candidates in front of your question list and your documents and seeing which one the reviewers trust.

What that means in practice:

• Buy for the model you chose, not the biggest model you have heard of. The difference between “fits on one card” and “needs four” is tens of thousands of dollars and a different power and cooling conversation with facilities.
• Check the power supply and the slot layout before you buy a second card. Vendor reference configurations are sometimes built for one card and quietly undersized for two. We have caught exactly that on a proposed build.
• Start with two servers and room to add. One to run the model and the document pipeline, one to hold the data and the application in front of it, and a chassis that can take another GPU when the second process comes along.
• The bank buys the hardware. In every engagement of this kind we run, the bank purchases the servers and licences directly, in its own name, in its own data center. We design, install and operate; we never hold the data. That sentence is the one that makes vendor management’s job easy.

Models: three candidates, one bake-off, the reviewers decide

There are now several open-weight model families that can be downloaded and run with no connection back to the vendor. They differ in what they are good at. One reads long contracts well. Another is sharper at structured extraction. A third is small enough to run fast on modest hardware and is good enough for routine questions.

The selection method that works is boring and rigorous: take the question list, take twenty real documents, run every candidate against all of them, and have the people who do the job today score the answers blind. Keep the model that the reviewers trust most, not the one with the highest benchmark score on a leaderboard. Benchmarks are measured on other people’s documents.

One more decision the bank, not the engineer, has to make: provenance. Some of the strongest document-reading models come from research groups or companies whose origin will matter to your examiner or your board. Put that on the table early, with the trade-off written down, rather than discovering it in a vendor-management review after the hardware is bought.

The parts that make it a bank system, not a demo

A working model answering questions is a demo. A bank system has four more things.

Access control. Who can submit a document, who can see an answer, and whether the system can be reached from outside the bank’s network at all (it should not be). This is the same question you answer for every other system, and the answer should be the same.

A review step. The model’s output goes to a person before it goes anywhere else. That person’s acceptance or correction is logged. This is both the quality control and the thing the examiner will ask to see.

A log of everything. Every document submitted, every question asked, every answer given, who reviewed it and what they changed. Kept with the same retention as the records it concerns. On-prem, this is a database in your building; in the cloud, it is a request to a vendor.

A network allowlist for the build itself. Installing the software means downloading it, and a bank’s security team will, correctly, want to know every external address the build touches. Write that list first and hand it to the security team before anyone plugs in a server. It avoids the week where the engineer is on site and the firewall is not.

What the examiner will ask

From the regulated-industry work we have done for twenty-five years, the questions are predictable, and they are the reason on-prem is easier to defend, not harder:

• Where is customer data processed? In this building, on this hardware, which the bank owns.
• Who can access it? This list, reviewed on this date.
• How are outputs validated? A named reviewer, for every document, logged here.
• What third parties are involved? The engineering firm that built and runs it, under a consulting agreement that grants it no data custody. No model vendor has a contract because no model vendor receives data.

Those are one-sentence answers. The cloud versions of the same questions are each a paragraph and a link to someone else’s report.

What this costs, in shape rather than number

Hardware the bank buys once: two servers and the GPUs the chosen model needs, with power and cooling checked by facilities. Engineering, time-and-materials, for the design, the document pipeline, the model bake-off, the application in front of it and the first process in production. Then an ongoing arrangement for running it, patching it and adding the second process.

The number depends on the model that wins the bake-off and the volume of documents, which is why we do not quote it from a web page. What we can say is that the first process pays for the hardware by taking a task that occupied skilled staff for hours per document and reducing it to minutes of review, and that the second process runs on the same cluster.

Phoenix designs, installs and runs on-premises AI infrastructure for banks and other regulated businesses that cannot send customer data to a third-party model. Exchange Bank’s build is in progress now. The North Bay practice is run from Rohnert Park: IT consulting and on-prem AI in Sonoma, Marin and Napa. If your board has asked the question, talk to us about private AI: forty-five minutes, with the engineer who would do the work, no sales pitch.