On this page
- What does PHIPA say about generative AI?
- What the IPC's AI scribe guidance means for admin tools
- Who is the custodian, and is the AI vendor its agent?
- Do you need patient consent or another basis?
- Can patients see and correct what the AI wrote?
- Does PHIPA require AI to run in Canada?
- Can we use de-identified data instead?
- What happens if staff use an unapproved AI tool?
- Which other laws and rules can apply?
- What should a privacy lead check before turning on an LLM tool?
- FAQ
- Working with XY Space
Yes, PHIPA allows an Ontario health information custodian to use generative AI with personal health information (PHI), under the Act's usual duties. The Information and Privacy Commissioner of Ontario (IPC) adds expectations you should plan to meet: a written vendor agreement, a privacy impact assessment (PIA), logging with regular review, and a public statement that says whether PHI leaves Canada.
The IPC's 2026 guidance, though, is about AI scribes, the tools that record and summarize patient visits. Much of it carries over to admin tools such as referral sorting, intake and inbox triage, and this post says which parts do.
The same duties settle whether you can use ChatGPT with patient information in Ontario. A staff member pasting a referral into a personal ChatGPT account has no vendor agreement or log behind it, while a tool the custodian has contracted, configured and logged can meet the duties.
This guide is for privacy, operations and IT leads at Ontario custodians and their vendors, and it is general information, not legal advice.
What does PHIPA say about generative AI?
Nothing by name. The Personal Health Information Protection Act, 2004 is technology-neutral, and its duties map onto an LLM workflow like this:
| Obligation | What it means for an LLM workflow | PHIPA |
|---|---|---|
| The custodian stays accountable for its agents | The vendor acts only as you permit, shown by a written agreement | s.17(1), s.17(3)(b) |
| Collection, use and disclosure each need consent or a PHIPA authority | Every workflow needs a stated basis, and so does any new purpose | s.29 |
| Reasonable safeguards against theft, loss and unauthorized use or disclosure | Access control, encryption, vendor due diligence, and defenses against instructions hidden in inbound documents | s.12(1) |
| Reasonable steps to keep PHI accurate and complete for the purpose | A hallucinated medication in a summary can become a statutory accuracy problem if reasonable steps were not taken | s.11(1) |
| No PHI if other information will do, and no more than necessary | Use de-identified or aggregate data where it works, send only the fields the task needs, and scope retrieval | s.30(1)-(2) |
| Secure retention, transfer and disposal | Prompts, outputs, embeddings and caches are PHI too | s.13(1) |
| A written public statement of your information practices | Add AI tools to it | s.16(1) |
| Breach notice to individuals, and to the IPC in prescribed cases | Your breach protocol has to reach the vendor | s.12(2), s.12(3) |
What the IPC's AI scribe guidance means for admin tools
In January 2026 the IPC published AI Scribes: Key Considerations for the Health Sector (the AI scribe guidance) with a companion checklist. In our view, these parts transfer to a referral, intake or inbox tool:
- A PIA before go-live, reviewed and updated over the system's life (s.3.4).
- Confidentiality agreements signed before staff get access and re-signed every year (s.3.9).
- Monitoring, logging and auditing "on an ongoing, targeted, and random basis" (s.3.6).
- Human oversight of the system (s.3.10).
- Vendor agreements and reuse limits. Part III of the checklist lists the contract terms the IPC expects, and s.5.4 bars agreeing to vendor reuse that PHIPA does not authorize.
- Accuracy review. For AI scribes, the guidance says custodians must have policies ensuring every record of PHI the scribe creates or alters is "reviewed by the custodian for accuracy before they are used for any purpose or disclosed to others" (s.6.1). The same review makes sense for any LLM output.
The consent analysis (s.6.2) and the points about recording a patient visit do not transfer, because an admin tool does not listen to the visit and can rest on a different basis.
An IPC investigator is likely to look for these in any AI complaint. A 2020 amendment to PHIPA adds an electronic audit-log duty that takes effect when proclaimed, and the IPC expects logging under the general safeguards duty either way. A human approval step before a drafted message goes anywhere is the simplest way to meet the review expectation (see our piece on human-in-the-loop AI).
Who is the custodian, and is the AI vendor its agent?
Start with the custodian, because it is often not the organization signing the contract. In many family health teams, group practices and shared clinics, the individual physicians are the custodians, and a multi-physician team has two common ways to make that workable. They can designate one custodian, such as the physicians' family health organization or the family health team, with the other acting as its agent. Or they can act jointly through a written agreement that names who signs vendor agreements, who owns the PIA and who answers access requests. The Association of Family Health Teams of Ontario describes these options and recommends legal advice first.
If the physicians stay separate custodians and one platform lets them share PHI with one another electronically, such as a shared front-desk inbox serving all of them, the vendor may be a health information network provider. That role carries its own duties under PHIPA's general regulation, which include notifying every custodian of a breach and keeping logs of access and transfers (General Regulation s.6(3), paras 1 and 4).
An agent acts for the custodian with PHI, "whether or not the agent is being remunerated" (s.2), and may handle it only as the custodian permits. A workflow vendor that reads referrals and drafts actions on your instructions usually fits, and you remain accountable for what it does.
An electronic service provider supplies technology without acting for the custodian, like a cloud account or a zero-retention model API, and may use PHI only as needed to deliver the service (General Regulation s.6(1), para 1). Many AI products sit in both roles, and the agreement should say which applies where, but the facts decide: a vendor that calls itself an electronic service provider and reads and acts on PHI for you is an agent.
Do you need patient consent or another basis?
Handing PHI to your own agent counts as "a use by the custodian, and not a disclosure" (s.6(1)), so you do not need a disclosure authority for the hand-off. For the same reason, the s.50(1) rule on disclosure "to a person outside Ontario" is not engaged when your agent processes data elsewhere for you. The use still needs a basis under s.29, which depends on the workflow.
AI scribes. The IPC's position is plain: "Therefore, consent of individuals would generally be required." It treats express consent recorded in the health record as best practice.
Admin tools. Record which of two bases each workflow relies on.
- Providing care. Implied consent within the circle of care may cover work that helps deliver care to the patient, such as routing a referral to the right clinician or booking the appointment it asks for (s.20(2), which applies only to the custodians listed in paragraphs 1, 3 and 4 of s.3(1)). Whether a workflow fits is a judgment for your counsel.
- The custodian's own permitted uses. PHIPA lets a custodian use PHI without consent for purposes such as planning and managing its services, quality improvement and de-identifying information (s.37(1)(c) to (f)). Inbox work on billing, complaints or third-party requests usually rests here, and s.37(2) lets your agent carry these uses out for you.
Where neither fits, you need consent, and a new purpose such as fine-tuning needs its own basis.
A vendor that uses the data for its own ends, such as training its model, steps outside the agency, and the hand-off becomes a disclosure. Section 5.4 of the AI scribe guidance is direct: "Custodians must not agree to disclose any personal health information to a third party vendor" for purposes like that "unless they have the legal authority to do so under PHIPA."
Patients can also withhold or withdraw consent for particular information, the so-called lockbox. An LLM that summarizes a whole chart can override it without anyone noticing, so retrieval has to honor the consent directives and masked records in your EMR.
Can patients see and correct what the AI wrote?
Yes. PHIPA gives individuals a right of access to their records of PHI and a right to ask for corrections, and AI summaries, triage notes and drafted letters filed to the chart are records of PHI. Stored prompts and outputs about a patient may be too. The IPC expects custodians to handle access and correction requests, including complaints "contesting the accuracy of notes enabled by the use of AI scribes" (AI scribe guidance s.3.13). Mark what the AI created or changed and route corrections through your normal process.
Does PHIPA require AI to run in Canada?
No, PHIPA has no data localization rule. In the developer section (s.4.6) of AI Scribes: Key Considerations for the Health Sector, the IPC writes that "PHIPA does not prevent personal health information from being retained or stored outside Ontario or Canada," while custodians remain responsible for protecting it.
You still owe s.12 safeguards, and your PIA should cover foreign-law access. The checklist (Part IV) also expects your public statement to say whether PHI is processed or stored outside Canada, which goes beyond what s.16 itself requires.
The Canadian-residency rule people attribute to PHIPA comes from procurement. Supply Ontario's AI Scribe vendor of record arrangement requires PHI to be stored and processed in Canada with narrow exceptions. It also bars vendors, with one narrow exception, from training on patient data even when de-identified. Those terms bind vendors under that arrangement, not a clinic buying a different tool, but they are becoming a market baseline for negotiation. Our AI inference data residency guide shows how to find where a model call runs.
Can we use de-identified data instead?
Only for side uses such as analytics or training, since sorting live referrals needs the patient's identity. The IPC says removing direct identifiers such as names and health numbers is not enough, and, citing PHIPA Decision 175, treats de-identification itself as a use of PHI. The risk of re-identification must be very low, which is hard with free-text referrals, and the vendor contract should prohibit re-identification.
What happens if staff use an unapproved AI tool?
In a letter of 27 October 2025 on a reported breach (HR24-00691), the IPC dealt with an unapproved Otter.ai tool that recorded a hospital's hepatology rounds and emailed the PHI of 7 patients to 65 recipients. It told the hospital to ask the vendor for deletion, to have its privacy office contact third parties directly in future, and to make clear that staff may use only hospital-approved devices.
PHIPA's breach rules have no risk-of-harm threshold. Any theft, loss, or unauthorized use or disclosure means notifying the affected individuals at the first reasonable opportunity, including of their right to complain to the IPC. An agent must tell the custodian of a breach "at the first reasonable opportunity" (s.17(4)(b)), and the custodian must also tell the IPC in the circumstances listed in General Regulation s.6.3(1). These include theft, further use or disclosure, a pattern of similar breaches, events reportable to a College, and a significant breach, as well as use or disclosure by someone who knew or ought to have known they had no authority (para 1). Our reading, which your counsel should confirm for your facts, is that a staff member who knowingly pastes PHI into an unapproved AI tool is likely in that category.
The IPC can make orders and, since 1 January 2024, impose administrative monetary penalties of up to $50,000 for an individual and $500,000 for an organization. The main offences, such as wilfully collecting, using or disclosing PHI in contravention of the Act (s.72(1)(a)), require wilful conduct and carry fines of up to $200,000, up to 1 year in prison, or both, for individuals, and up to $1,000,000 for organizations. Training should tell staff they can be prosecuted personally, and the custodian still answers for its safeguards.
Which other laws and rules can apply?
- Provincial systems. Workflows that pull from ConnectingOntario, OLIS or HRM fall under Ontario Health's access agreements, and OntarioMD's EMR integration terms can bind clinics. These generally limit use to providing care, so ask before an admin tool reads those feeds.
- PIPEDA. The federal exemption order SOR/2005-399 exempts Ontario custodians, not their vendors, from Part 1 of PIPEDA, so PIPEDA can apply to a commercial vendor's own handling of the data.
- Bill 194. The Enhancing Digital Security and Trust Act sets AI rules through regulations that have not yet been made. Public hospitals are public sector entities under it, so AI regulations could reach them once made; private clinics and family health teams are generally outside it.
- Medical devices. Sorting referral faxes by administrative category, such as specialty or document type, is admin work. Ranking them by clinical urgency moves toward clinical decision support, which may raise questions under Health Canada's software as a medical device guidance.
What should a privacy lead check before turning on an LLM tool?
Before you sign
- Identify the custodian or custodians, agree who signs and who owns the PIA, and decide whether the vendor is an agent, an electronic service provider or a health information network provider.
- Complete a PIA covering where inference runs and foreign-law access, and update it when anything material changes.
- Record the basis for each workflow, and for any fine-tuning or index built from your PHI, noting what went in and how you will destroy it.
In the agreement
- Limit purpose and bar training and secondary use, set a breach-notice timeline, keep deletion and audit rights, and list every sub-processor and model provider with locations and notice of changes.
In the workflow
- Respect consent directives so retrieval honors lockbox instructions and masked records.
- Minimize each prompt to the fields the task needs and limit retrieval to the record in hand.
- Treat inbound documents as untrusted. Faxes and emails can carry hidden instructions to the model (prompt injection), so limit what it can retrieve and send, and test for it.
- Put a human review step in front of anything the model drafts before it is used or disclosed.
- Log every model call that touches PHI (user, record, input reference, output, approver, time) in a log built so that edits or deletions show, and review it on a targeted and random basis. PHIPA's own electronic audit-log duty (s.10.1) is not yet in force, since it starts on a day to be named by proclamation, but the IPC expects logging now.
Ongoing
- Split retention in two. Prompts, caches and vendor-side logs can be deleted quickly, while AI output filed to the chart follows your clinical-record retention rules. Any deletion schedule must pause for an access request or legal hold.
- Extend your breach protocol to the vendor, and count AI incidents in your annual statistics to the IPC.
- Keep an approved-tool list and block consumer AI tools.
- Update your public statement to cover the AI tool and say whether PHI is processed or stored outside Canada.
FAQ
Is Microsoft Copilot or ChatGPT Enterprise PHIPA compliant?
No product is compliant on its own, because compliance depends on how the custodian contracts for, configures and monitors it. For ChatGPT Enterprise, the test is whether your agreement makes the vendor act only on your instructions, bars training on your data and says where it is processed. Copilot surfaces whatever a user already has permission to see, so loose SharePoint or Teams permissions become a safeguards problem the day you switch it on.
Do we need a PIA before using AI with PHI?
The IPC expects one before you introduce any AI system that handles PHI. It is an IPC expectation rather than a general PHIPA duty, but plan on one.
Working with XY Space
XY Space builds AI workflows for admin work such as referral triage. We run the workflows we build inside the client's own cloud account, with human approval steps and an audit log, which narrows what we can reach and makes our role easier to describe in a PIA. A written agreement is still needed, and our healthcare page describes how we deploy for custodians.
*Mahmoud Halat was product lead for software in more than 100 healthcare deployments. To talk through a specific workflow, bring one job to a free 30-minute call.*
Written by
Mahmoud Halat
Principal AI Solutions Architect, XY Space
Principal AI Solutions Architect at XY Space. A decade of regulated health-tech deployments across 100+ hospitals and 10M+ patient journeys, with a co-authored patent on patient journey reconstruction.
More from Mahmoud HalatShare this article
Work with us
We build the systems these posts describe, and we'll tell you in the first call whether yours is worth building.
Start a project