"Can we use AI without our data leaving the building?" Owners often ask this when they plan a central company brain: one place where all company knowledge lives, kept current, and used by staff and AI agents alike. That system holds contracts, client files and hard-won know-how, so where it runs matters.
The honest answer is that "private" is not a single setting. It is a trade-off between control, capability and cost. Here are the three common setups, what each really protects, what you pay for, and when your own servers make sense.
What "private" actually means
Separate three questions that sales pitches often blur together:
- Where is the data processed? Not just stored. A model has to read your text to answer, on someone's hardware, somewhere.
- What is kept, and for how long? Prompts, answers, logs and files can each have their own retention rules.
- Who can access it? The vendor's staff, subcontractors, and the authorities of whichever country the vendor answers to.
The three common setups
1. Hosted AI APIs from major providers
You send requests to a model run by a provider such as OpenAI, Anthropic or Google, and pay for what you use. Nothing to install, and you get the most capable models as soon as they ship.
Business terms are often better than people assume. OpenAI, for example, states that API data has not been used to train its models since March 1, 2023 unless you opt in. But "not used for training" is not "not kept": by default, abuse-monitoring logs, which can include prompts and responses, are retained for up to 30 days. Zero retention exists, but only for customers OpenAI approves.
Residency is the other catch. OpenAI's Canadian region is listed as storage only: content is stored at rest in Canada, but may be processed elsewhere. For models released since March 5, 2026, residency also carries a 10% price uplift.
Costs: usage fees, usually priced per token (a fragment of a word), plus any premium for residency or retention controls. Low commitment, but the bill grows with every user and agent you add.
2. Your own cloud account with private model hosting
Here the model runs inside your company's own cloud account, in a region you choose, such as Canada. Either you use a managed model service (Amazon Bedrock, Microsoft Azure, Google Vertex AI) and still pay per use, or you rent GPU servers and run an open-weight model yourself.
This tightens control. On Amazon Bedrock, AWS states that model providers have no access to the accounts where their models run, so they cannot see your prompts. You can set retention per region, although some models require AWS to retain prompts and outputs for up to 30 days for safety review.
Check the routing, too. AWS's guide to cross-region inference in Canada explains that stored data stays in the Canada (Central) region, but requests can be processed in US or other regions, depending on the profile you choose. Confirm that setting in writing.
Costs: per-use fees for managed services, or GPU hours for rented servers, which bill while switched on, busy or idle. Add storage, networking and the staff time to configure and monitor it all.
3. On-premise hardware running open-weight models
Open-weight models can be downloaded and run on your own servers, and many are now practical for business tasks. OpenAI's gpt-oss-120b, under the Apache 2.0 licence, fits on a single 80 GB GPU; its smaller sibling runs within 16 GB of memory.
Data stays on your premises and retention is your decision. So is responsibility: patches, access control, backups and incident response become your job.
Costs: hardware purchase; power and cooling, which are not trivial when a single NVIDIA H100 SXM is rated at up to 700 W; rack space; maintenance and eventual replacement; and the staff to run it. You also accept that the largest frontier models are generally offered only as hosted services.
The Canadian picture
Federally, PIPEDA sets the rules for commercial activity, while Alberta, British Columbia and Quebec have their own substantially similar private-sector laws.
Outsourcing the processing does not outsource the responsibility. The Privacy Commissioner's guidelines for processing data across borders say you remain accountable, should tell customers their information may be processed in another jurisdiction, and that no contract can override the laws of the country where the data ends up. The Commissioner's principles for generative AI also recommend privacy impact assessments.
Quebec goes further. Under section 17 of its private-sector privacy act, as amended by Law 25, a business must complete a privacy impact assessment before communicating personal information outside Quebec, including to a service provider, and needs a written agreement. A data centre in Ontario counts as outside Quebec. Penal fines for businesses can reach $25 million or 4% of worldwide turnover, whichever is greater.
Residency is not sovereignty, either. The federal white paper on data sovereignty and public cloud notes that data residency does not protect against foreign laws: a provider subject to another country's laws may have to answer to them, wherever its servers sit.
And watch Ottawa: Bill C-36, introduced in June 2026, would enact a new federal privacy law. It is still at second reading, so plan around PIPEDA for now.
The cost that doesn't change
Prices move too fast to quote, but one cost is the same in every setup: connecting your documents, systems and permissions so the AI gives correct answers to the right people, and keeping that knowledge current. Budget for it whichever option you pick; it is easy to underestimate.
When on-premise makes sense, and when it doesn't
On-premise tends to earn its keep when most of these are true:
- Strict data rules. Regulation, client contracts or your own risk appetite say certain data must stay on your premises.
- Steady, high volume. Hardware is a fixed cost. It pays off when busy every day, not when idle between occasional requests.
- Existing IT capacity. You already run servers, and someone can patch, monitor and back them up.
It usually doesn't make sense when usage is low or unpredictable, when you need the strongest models, when nobody in-house can look after hardware, or when you are still experimenting. Then a hosted API or a Canadian cloud region with the right settings is cheaper and faster to start.
A mix is often the practical answer: sensitive records handled by a model on your own hardware, general drafting sent to a hosted model with strict retention terms. That is how we usually design a company brain at 2MDA, so the same system can run on your servers or in your cloud, and move later.
Questions to ask any vendor
- Where is our data processed, not just stored? Can requests leave Canada, and can we prevent that?
- What do you retain, for how long, and can we set it to zero?
- Is our data used to train or improve any model? Where does the contract say so?
- Who can access our data, and under which country's laws?
- Which subprocessors do you use, and where are they?
- Will you sign a written agreement that supports our privacy impact assessment?
- How is pricing calculated, and what happens if usage doubles?
- If we leave, can we export our data and configuration in a usable form?
Takeaway
"Private AI" is a spectrum, not a switch. Hosted APIs are the quickest way to start and can be private enough for many uses; your own cloud account adds control over region and retention; on-premise gives the most control at the highest fixed cost and effort. Start from your data rules and expected volume, put the questions above to every vendor, and choose the simplest setup that meets them.