Responsible AI: Ensure you're protecting confidential information

When using search, there is a limit to how much information we are giving away.

---

If you are using LLMs such as ChatGPT, Meta AI, Claude, DeepSeek or the plethora of other options now out there, it is important that you give careful thought to what information you are sharing with them, particularly if you are using these tools for work tasks. This is especially the case if you, or your software development team, are using AI-enhanced coding tools.

You may have heard an anecdote of how someone was chatting with a friend about something random, say, pool cleaning, and then they started seeing adverts on their phone to do with pool cleaning. Sometimes this may be because they've done a Google search, providing Google with information that they are now interested in pool cleaning. Netflix's The Social Dilemma (trailer on YouTube) opened many people's eyes to how our use of social media and search was providing information to the provider, which information is then used to target advertising at us.

LLM-based AI agents are able to get a lot more information out of us because we're not just searching, we're chatting with them, and our chats can in turn be used in ways we never intended. In our personal lives, this can be concerning if we are not wanting to be constantly profiled and tracked – but it affects only us. In business, however, this can be a liability concern as oftentimes the information we deal with is private (protected by law) or subject to non-disclosure agreements.

When using search, there is a limit to how much information we are giving away. For example, in my job, I may be needing to read up on occupational health and safety (OHS), and I might use Google to search for that, and find the various government documents related to OHS. If I do that, then Google "knows" I am interested in OHS, but doesn't necessarily know why.

AI assistants are encouraging far more than this: for example, NotebookLM, again from Google, encourages the user to upload multiple documents to be able to ask the assistant questions about them. In my OHS example, if I were to use NotebookLM, perhaps I might upload South Africa's health and safety laws and policies (no problem, these are public), my company's own health and safety manual (problematic, probably confidential) and a description of the incident that has taken place (very problematic, definitely private), and ask for its assessment. Now, instead of just knowing that I have some interest in health and safety, Google knows exactly why I am suddenly researching it, and I have shared confidential and private information.

Sharing documents like this with someone you have a trust relationship with, such as a lawyer, or a professional consultant, would not be a problem at all – the issue is not the sharing of information, it is the trust relationship with the provider. Like social media companies and Google in The Social Dilemma, unless we are very sure about the policies of the provider we are using, we can consider it likely that they are using this information in ways we don't intend.

In order to train their models, AI providers need content, and given that the current models are trained on the entire public internet, model trainers are trying to find ways to get more content, whether from publishers, digitising public domain texts or from you, the user of their products. When you upload a document, or a meeting transcript, or invite an AI meeting attendee, that is a chance for the AI providers to get more training data, which they are in sore need of, so they are incentivised to get that data from you, because they need human-created data to train future models.

One counter-argument is that our lawyer or consultant is doing exactly the same thing – they are also adding this new health and safety case/incident to their knowledge and experience, updating their model, so to speak. However, humans are known to be inordinately better at keeping confidential information confidential. An AI has no understanding of the power of professional bodies and consequences for breaking their standards or ruining one's reputation. Just see for yourself how easy it is to get an AI to tell you things it shouldn't with a little game called Gandalf. Other examples abound of specific prompts breaking the carefully built in safeguards that model providers attempt to add.

AI providers are mixed in their policies about how they use your prompts (ie, your chat messages or your uploads). At the time of writing, OpenAI says that "we may use content you provide us to improve our services, for example, to train the models that power ChatGPT". It also says you can opt out of this and that this will stop future prompts from being used, but, as far as I can tell, it won't remove what you've already sent it. It also has a helpful article explaining how your data are used to improve model performance, which indicates that enterprise subscribers (starting at around R500 a month) are automatically opted out of its prompts being used for training, implying that the prompts of non-paying users are definitely being used.

A "responsible AI" movement has recently grown because of these precise concerns. For example, the ISO has published ISO/IEC 42001:2023 standard, which touts "responsible use of AI" and traceability and transparency, so that you can better see exactly what compliant companies' policies are. Anthropic, which has implemented this ISO standard in January 2025, insist that its model, Claude 3.7, was not trained using user prompts. Anthropic also says it collects prompts if feedback is provided (ie, a user indicates an issue with a response they've received), or if a response is flagged for trust/safety, and in this case, your entire prompt chain could be kept and used for training the model with "legitimate interest" as the justification. This is understandable as it wishes to remove any harmful tendencies with more training, but it is potentially a loophole for using your prompts for training in the future. In addition, while providers previously made their training data public, much of it is now licensed private data, so they are doing this less and less, which means the general public is not able to confirm whether they are adhering to their own policies.

If you do purchase an enterprise subscription to one of these services (at upwards of R500 per user per month), it often includes reassuring clauses that indicate your usage data are kept separate from others, and are not stored and not used to train new models. But if you are using a free or cheap/personal service, these assurances are not provided, or you need to manually opt out (as with ChatGPT), and if you don't opt out, it is likely your prompts will be used to train future AI models.

This is of particular concern in software development tooling, where AI is becoming de rigueur. It's one thing if you can control exactly what is sent to the LLM – you can, for example, remove anything confidential before sending your chat message to ChatGPT by adjusting the snippet of code before it is sent. But with LLM-enabled code-completion tools, the tool is deciding what context is sent, and it may send something you don't want it to, such as secret keys, passwords, certificates and the like that may be on the machine the tooling is running on. If it does that, and the AI provider uses that information to train future models, then a secret could end up trained into a model and pop out somewhere else. There are ways to carefully keep these secrets away from the tooling, but other forms of confidential knowledge embedded in the code could slip out in this way – trade-secret algorithms, for example.

As a software development consultancy, Chillisoft has instructed its developers not to use free or personal-level AI tooling for work-related tasks in order to ensure it does not inadvertently leak information that should not be leaked. Meanwhile, Chillisoft is trialling ways to use LLMs that do not risk confidential information being leaked, while also being affordable.

So far, the company has identified three ways this can be done:

Local model (on-machine): If you talk to an offline model running on your local machine, there is no risk of data leakage. For this you need a powerful computer with an advanced modern graphics card with enough memory to fit the model. And even then, models that can fit on these machines are not nearly as powerful as the big models, but they are good enough for some coding or work tasks. Chillisoft has found, for example, that they can help if you have any repetitive, recipe-based tasks, such as basic code generation/transformation (eg, generating a class from an existing database table), for generating test data in a specific format, or for re-formatting test data, none which we would want to get a free online model to do, for the aforementioned confidentiality and privacy concerns. Even doing simple tasks like this, the developer needs to always check what is output as it frequently has errors in it – however, it can save a bit of time with certain specific tasks.

Shared local model (on-network): Instead of running models on each machine, you could purchase some specialised hardware and run a larger model on your company network, which network and VPN-connected users can access. In this case, it is plausible to run fairly large models, but to do so you would need to spend in the region of hundreds of thousands of rands, or even some millions. This may, however, turn out cheaper than paying per token or per user for Anthropic or OpenAI's enterprise services, depending on usage. The risk is that the models are evolving so fast that you won't be able to keep up with the new models that come out next month or next year without updating your hardware.

Cloud-based model: A third option is to configure your own ring-fenced cloud-based model. Using this approach, you can have a model of your choice running in a cloud service (such as Azure), which is under your control, and which you can interact directly with using tooling for the purpose. In this case, you are controlling the data transfer – it is going from your network to the ring-fenced cloud-based model, with transport layer security, and definitely not being used for training and not stored by default. To get this even more secure, you could establish a VPN between the cloud and your network. For this approach you are paying per token, so it may turn out expensive, but it might also be a good way to trial the benefits in a safe manner.

The thing in common between these three approaches is that they have little to no risk of confidential or private information being shared inadvertently. However, they are more effort to set up than using a public provider, and could require building new organisational capabilities. For most enterprises with a concern for their confidential information and their customers' and employees' private information, it is going to be easiest (and most expensive) to use a public provider and pay for the enterprise options which promise not to use your data, for now.

As an aside, none of these options obviate the output risk – the risk that what the AI produces could be harmful – that is a whole different risk that needs to be considered, but it is outside the scope of this press release.

Like in your use of search and social media, if you're not paying fair value for a service (and fair value for AI is very expensive because of the compute power needed to do it), you are being harvested in some way, or providing a service yourself. In the case of AI providers, you are potentially providing new training content for their future models, so beware what you send them. There are ways to harness the power of LLMs without this risk, but you will need to pay a reasonable amount for the service, or do the work to do it yourself. The ecosystem for doing it yourself is getting better quickly (eg, Ollama for local models), and it's worth keeping an eye on it.

With the new capabilities LLMs potentially unlock come new opportunities but also new risks. It's important to use them in a risk-assessed manner – choosing a software development partner that understands these risks and has developed mechanisms and strategies to responsibly make use of AI will help you reap the potential benefits of LLMs while navigating the many potential dangers.

Please visit www.chillisoft.co.za.