LLMs & Malicious Code Injections: We Have to Assume Its Coming

Large language models promise to enhance secure software development life cycles, but there are unintended risks as well, CISO warns at RSAC.

A rise in prompt injection engineering into
large language models (LLMs)
could emerge as a significant risk to organizations, an unintended consequence of AI discussed during a CISO roundtable discussion on Monday. The panel was held during
Purple Book Community Connect–RSAC
, an event at this weeks RSA Conference in San Francisco.
‍One of the three panelists, Karthik Swarnam, CISO at ArmorCode, an application security operations platform provider, believes incidents arising from prompt injections in code are inevitable. We havent seen it yet, but we have to assume that it is coming, Swarnam tells Dark Reading.
LLMs trained with malicious prompting can trigger code that pushes continuous text alerts with socially engineered messages that are typically less adversarial techniques. When a user unwittingly responds to the alert, the LLM could trigger nefarious actions such as unauthorized data sharing.
Prompt engineering will be an area that companies should start to think about more and invest in, Swarnam says. They should train people in the very basics of it so that they know how to use it appropriately, which would yield positive results.
Swarnam, who has served as CISO of several large enterprises including Kroger and AT&T, says despite concerns about the
risks of using AI
, most large organizations have begun embracing it for operations such as customer service and marketing. Even those that either prohibit AI or claim theyre not using it are probably unaware of down-low usage, also known as
shadow AI
.
All you have to do is go through your network logs and firewall logs, and youll find somebody is going to a third-party LLM or public LLM and doing all kinds of searches, Swarnam says. That reveals a lot of information. Companies and security teams are not naive, so they have realized that instead of saying No [to AI usage] theyre saying Yes, but establishing boundaries.
One area in which many companies have embraced AI is incident response and threat analytics. Security information and event management is definitely getting disrupted with the use of this stuff, Swarnam says. It actually eliminates triaging at level one, and in a lot of cases at level two as well.
When using AI in application development tools, CISOs and CIOs should establish what type of coding assistance is practical for their organizations based on their capabilities and risk tolerance, Swarnam warns. And dont ignore the testing aspects, he adds.
It is also important for leaders to consistently track where their organizations are failing and reinforce it with training. They should focus on things that they need, where they are making mistakes — they are making constant challenges as they do development work or software development, Swarnam says.

Last News
▸ ArcSight prepares for future at user conference post HP acquisition. ◂ Discovered: 07/01/2025 Category: security	▸ Samsung Epic 4G: First To Use Media Hub ◂ Discovered: 07/01/2025 Category: security	▸ Many third-party software fails security tests ◂ Discovered: 07/01/2025 Category: security

**Cyber Security Categories**
Google Dorks Database	Exploits Vulnerability	Exploit Shellcodes

CVE List

Tools/Apps

News/Aarticles

Phishing Database

Deepfake Detection

Trends/Statistics & Live Infos

Tags:
LLMs & Malicious Code Injections: We Have to Assume Its Coming