AI and Privacy
AI poses serious challenges to privacy. The threats are numerous and do not have obvious solutions or countermeasures. Here are my thoughts on a few of them.
agentic systems
I think the most obvious privacy issues involve AI agents rummaging through filesystems, email accounts, cloud storage, etc. The trend is to give more agency to the AI agents, so they can manage more complicated tasks. Unfortunately this means giving more access to more data and more accounts, with more ability to make decisions and take actions on your behalf.
They might correspond with your contacts and learn things you don't want them to. They can infer things based on your photos, activities and correspondence that you wouldn't want to tell them outright.
stylometry
A less obvious privacy concern with AI is stylomtery. Stylometry is the technique of doxxing you based on your writing style. Human experts used to do this but their bandwidth was limited, so only people of interest got this attention. AI has democratized the capability to treat writing as "a fingerprint". The costs have changed, the scope has changed, and now anyone can do this.
One subtle change brought about by AI is the nature of how stylometry is used. Back in the day, it was used to confirm an identity. J.K. Rowling was outed as the author behind the "Robert Galbraith" pen name using stylometry, for example. A group of suspect personas was established and compared to samples. Now it can easily be used in an exploratory way, to try to doxx people. These fishing expeditions are far from reliable in some cases because stylometry proves nothing absolutely - it is probablistic.
AI-based stylometery is more robust than the classic version too. Stylometry used to be all about word frequencies, sentence lengths, punctuation, slang, vocabulary. Those are all things an author can manipulate, and they are crude measures at best.
AI's do stylometry better. They capture all the same things that human experts did, but notice so much more. They discern topic-selection tendencies if the sample size allows it. They notice not just typos but idiosyncratic errors, and formatting quirks. As it turns out, there is evidence that they can often fingerprint an author even when said author uses AI to try to obfuscate the source of the writing!
Countermeasures are less effective than ever now. You might think that more capability exists now, using AI tooling, but so far that's not the case. Homoglyphic substitution, for example, is a technique where you swap characters for visually identical Unicode characters. Studies suggest that attribution accuracy is reduced but is far less effective than previously thought.
AI-based stylometry also has an advantage of scope. Multiple accounts can be linked together and analyzed as a group. This defeats the strategy of trying to post all of one content type in one style, and another type of content in another, under two separate accounts. AI's can literally build composite profiles and get equally strong results.
right to be forgotten
A growing concern is what I'll refer to using the EU terminology, your "right to be forgotten", which used to mean deletion from databases. AI systems that continuously learn create problems with this due to their rapid incorporation of data.
Once your data turns into model weights, deletion becomes technically impossible. When your data, that used to be private, becomes an integrated part of the model it no longer exists in the same form, and cannot easily be exorcised. Gone are the days when you could simply select data from a database and delete it.
open-weight models
Governments want to impose restrictions on open-weight AI models in the hopes of preventing malevolent actors from using them to do bad things. Their fears are well-founded.
Our freedoms and privacy ultimately depend on using open models running on our local devices, for the same reasons we need FOSS instead of depending on the goodwill of big corporations and their closed-source software. If we run open-weight models locally, we avoid sending our data to some big company to be used for model training, or sold to data brokers. With AI being integrated into most commercial software it's difficult to assess, let alone audit where the data goes and what it's used for.
When we use open-weight models, we have choices. Not just the choice of which models to use, a choice we can change on a whim. In doing so, we avoid being locked into monolithic privacy standards set by AI vendors. One size does not fit all.
It all comes down to agency. When we use big tech's AI models, they get to choose our level of privacy. They can change that as circumstances change. For example, when governments put presure on them, they can provide different models with different privacy profiles.
Big companies will end up offering limited privacy options, privacy-for-pay options, or maybe none at all. Offering the end users choices is only appealing to them when there is profit to be made.