David Chess
IBM Thomas J. Watson Research Center
1998
Is it safe to click on this hyperlink? Do I trust the person who sent me this Word document? If I send this program out into the Web to find me some bargain CD's, will it get cheated? "Mobile" and "intelligent" agents are hot topics, and both have security implications. This talk will address some of these implications, and try to answer, or at least ask, some of the more obvious questions.
What is an agent, and should you be worried about them? Because agents are a hot concept, the label is used, and overused, in all sorts of contexts. But one common theme is that an agent is a computer program that represents, in one sense or another, a human user. This may mean that the program makes decisions, or gives advice, that would normally have been made or given only by a human. It may mean that the program travels over a network, to carry out operations on remote systems on behalf of a less mobile human. It may mean both. Agents that travel from system to system are called "mobile" or "itinerant" [1].
A sample itinerant agent is a program that I dispatch into a network to find some information for me. It moves from system to system, finding pieces of the desired information, and clues as to where more of it, or more clues, may be found. A sample "intelligent" agent is a sophisticated mail-sorting program, that processes incoming mail for me, making decisions about what mail to discard unread, what to mark as urgent, what to send a canned reply to, and so on. An agent that falls into both categories is a shopping agent, that I dispatch into the network to find goods for sale, and to decide which, if any, to purchase on my behalf. This paper will concentrate primarily on security concerns raised by itinerant agents, although I will also speculate briefly on some of the risks of intelligent agents.
Agent systems face all the security challenges involved with any system that sends messages (see for instance [2]). Delegating tasks to mobile programs raises another set of concerns as well.
Whenever programs travel rapidly from system to system, we need to be wary of openings for viruses. In the aftermath of the Internet Worm in 1988 [3], security experts generally concluded that the Worm had succeeded partly by arranging to begin executing as soon as it arrived at a new host. The obvious lesson was that this is a bad idea: systems should be carefully checked to ensure that there is no way for a program to send itself to the system, and begin executing upon arrival. Itinerant agent systems, of course, break this rule purely and directly! Given that we now want to allow programs to arrive and begin running at once, we need to develop ways to allow them to do this safely.
Agent-based systems also increase the connectedness and the complexity of the computing universe. As connectedness and complexity increase, new failure modes arise, some of which have security implications. These emergent security concerns are more speculative than the more traditional ones inherited from previous computing systems, but I will mention a few of the possibilities.
"Please run me. I'll be good. Really!" You probably wouldn't run a program sent to you by a stranger on the Net. But would you hesitate to open a document sent by a stranger as e-mail? Depending on what you use to read that document, you may be running a program simply by opening it; your document-reading program may be obeying hidden orders embedded by the document author. Hundreds, and probably thousands, of users of Microsoft Word have recently discovered this first-hand.
The Concept virus [4] is a piece of code written in WordBasic, the macro language used by the word processor Microsoft Word. If I open (i.e. use Word to read) a document infected with this virus, part of the virus immediately runs, and copies the virus into my installed copy of Word, in such a way that future documents that I save may become infected with the virus.
Concept itself has no intentionally destructive "payload", but it does sometimes cause problems because of some of the side-effects it has when it embeds itself in documents. The virus spreads well enough that it has infected at least hundreds of different computers, and it seems to have become sufficiently widespread that it is unlikely to become extinct in the near future.
So whether we like it or not, agent systems are here, are in wide use, and are already troubled by viruses. We may be using them unintentionally now; as we begin to use them on purpose, we have a chance to decide how they should be controlled.
In a typical corporate environment today (February, 1996), users are forbidden by policy from using any software that did not come into the company through the usual channels (the Purchasing Department or equivalent). Many individual users, especially virus-conscious ones, follow similar policies, and use only software from shrink-wrapped boxes and the most reliable shareware and BBS sources.
In some ambitious visions of the future, agents are programs that flit from machine to machine, accessing services, performing searches, carrying messages, and generally constituting the entire computing infrastructure. This vision is not entirely compatible with the ability to control which programs my system executes. If all mail is in the form of nimble message-carrying programs, I have to be willing to execute programs sent by anyone I want to accept mail from. If the standard way of interacting with information providers is via small downloaded programs, I have to be willing to execute programs sent by anyone whose data I want to browse. This programs-from-anywhere model of computing is not required by the agent systems that enable it. All of them provide, or plan to provide, optional facilities to restrict the set of users that programs will be accepted from, and in some cases to restrict execution to programs that are digitally certified as benign by some trusted agency. Users or system administrators are free to use these facilities. On the other hand, if the general consensus is that these facilities are not needed, it is likely that popular and widely-used resources will appear that require them to be shut off.
And in that case, users will shut them off: security features that get in the way of getting work done always get shut off.
The designers of agent systems, of course, have thought of these things. As well as restricting the sources from which one will accept programs, well-designed agent systems also provide facilities to control what an incoming program is allowed to do once it has arrived. This is generally done either by having the programs run in a virtual or interpreted environment, or (on multi-user systems) having the programs run with only certain privileges, or both. Safe-tcl [5], Java [6] and Telescript [7] programs are all interpreted, to give the system detailed control over the actions that received programs can take (more detailed than that provided by the access controls, if any, in the native operating system). Exactly how to use this control is not always clear. Java applets, for instance, can be forbidden to access any part of the filesystem except those parts set aside for use by applets. In the present environment, this is good protection. But if Java applets begin to form an important part of the computing infrastructure, the applet-usable part of the filesystem will become more important, and need to be protected itself.
In some cases, the privileges that an incoming program has should depend on where the program came from, as well as some function of where it has been. There are complexities involved in authentication and authorization when a program has been through multiple hops since leaving its source; these technologies are still evolving rapidly.
Controlled execution of received agents is naturally not as secure as not running them at all. But if the security of the systems, both real and perceived, is sufficient, and if they let us do useful new things, we will eventually accept them, and profit by them.
If I've written an agent that embodies a clever stock-trading algorithm that I just invented, how willing should I be to send it off into the net to execute?
System administrators need to protect their systems against malicious agents; can we also offer the writers and users of itinerant agents some protection against malicious servers? The obvious encryption techniques can prevent anyone but the intended recipient server from reading the agent. But what if the intended recipient itself is not completely trusted? What if I want to have my agent run on some server that I saw advertised in the personals column?
This is a difficult problem for a number of reasons. The server must be able to read the agent's code, and any data that it uses, in order for the agent to execute; so the agent will have a hard time guarding any secret data or algorithms. It can use public-key encryption to encrypt information found on one server, to protect that information from servers that it may run on in the future, but only at the cost of no longer being able to read that information itself. It can deliver it, and it can read it once it arrives at a place where the corresponding private key is available to it, but while it is travelling in untrusted places, all the secret information that it carries must be hidden from it.
If the agent is to be able to learn from experience, the server must also be able to change the agent's data before sending it on. While the author of a piece of agent code can digitally sign the code itself, the agent's data and process-state information cannot be signed in advance; if we knew what it was going to do in advance, there'd be no reason to send it out! The main entity that can sign an agent's state information is the server that the agent was on when it reached that state. So an agent that has been roaming around on a network through a number of hops may have a program signed by one principal, and a record of state-changes signed by a number of intermediate servers. It may be a complex task to determine how much such an agent should be trusted, especially if some of the entities that have signed it are unknown.
To make matters worse, since the agent itself cannot carry any secrets, such as private cryptographic keys, it cannot itself sign any messages it might want to send, or changes it might want to make in itself, since that would give any server who watched it execute the ability to forge that signature on other objects. Research in this area of agent security is just starting [8], and it is likely to remain an interesting field for some time.
It's not called the Net of a Million Lies for nothing. So, you're totally wired. Your company has a T1 line to the Net, you have accounts with three different ISP's, and you have five Web-search engines in your hotlist.
And the more information that you can get to, the more uncertain you are which of it to trust. Now consider the plight of an "intelligent" agent, sent out into this morass to find out something for you. Not only is it deluged and overloaded with information, but it's also stupid.
Even the most "intelligent" programs today are orders of magnitude less capable than humans, especially in their ability to apply broad world knowledge to particular problems, to grasp wholes, to appreciate subtleties, to evaluate plausibility. If we place too much trust in the intelligence of "intelligent agent" systems, we find ourselves indirectly subject to various new kinds of fraud.
If you search on "sex" or its obvious variants on one of the public-access web-search pages, you will find, among other things, pages with titles of the form: John(Sex) Smith's(Sex) Home(Nude) Page(GIF) On the page itself, John will explain that there's really nothing here about sex, he just wanted to get lots of visitors.
If you search on the names of any of the candidates in the 1996 U.S. Presidential elections, you are likely to find a number of pages. Some are provided by the candidate's organization, others by independent and unofficial supporters, and others by opponents. Especially in the case of "parody" pages, it can be difficult to tell which pages are which. A human, let alone an automatic web-searching agent, might have a hard time getting a reliable and consistent impression of the candidate's stands from this set of information.
At present, this sort of thing is not a serious problem. People use web-searching agents only to retrieve a list of pointers to possibly-interesting sites; the human user then (often slowly and painfully) culls through the pointers, ignoring some, following others, adding the best hits to a hotlist, or just recording the data found there. A human will not try to use John (Sex) Smith's home page as a source for naughty pictures, nor form an impression of Robert Dole's opinions based on information from a page whose background is a lattice of Dole brand pineapple cans.
But a program might not be so acute. I cannot count on an agent program to venture off into the network to make purchases on my behalf unless it can reliably determine just what is for sale, and how likely it is to satisfy my needs. Especially in the case of information goods, whose content may be complex, subjective, and hard to summarize, agent programs will be (even) easier to mislead than humans.
This is a security problem itself, if not one that computer security professionals are used to considering. It also leads indirectly to more familiar concerns: if agents are used to negotiate deals for security-significant resources (which includes everything from time and directory services to routing information), they will need to have some assurance that they are not getting an especially good deal because the seller intends to attack the system by selling it corrupted information. There are a number of plausible approaches to solving this problem:
Whatever approaches turn out to be most effective, the danger of agent-targeted fraud will have to be addressed in any mature agent-based information system.
"Mr. Chess is busy browsing alt.sex right now; can I take a message?" Sophisticated intelligent agents can form models of the people and organizations that they serve. These models are useful to guide the operation of the agent; but some aspects of the model may include facts that should not be revealed to outsiders.
My secretary-agent may know that I am unavailable because I am in an urgent meeting with Rob and Maureen about a deadline we're about to miss.
Will it be smart enough to know that it can tell callers that I am in a meeting, but that the subject of the meeting must not be mentioned?
If it has learned that every Thursday at this time I am unavailable, and in an emergency I can be contacted at the Fair Hills Golf Course, will it know exactly who it can reveal which parts of that knowledge to?
To the extent that I have explicitly told the agent these things, I can also tell it how closely to guard them. But to the extent that it has figured them out itself, it will also have to figure out the confidentiality itself, and programs are generally not very good at that sort of task.
Agents may also reveal secret information by the patterns of queries they send out, the bids they solicit, or the goods they purchase.
People can and do make this same sort of mistake, of course, but we can expect agent programs to do it even more often, and some clever design work will be required to avoid it.
"Here be dragons" As a system becomes more complex and more interconnected, it is more likely to exhibit large-scale behaviors that we would not predict from knowledge of the individual elements.
The worldwide computer-virus problem, for instance, turns out to depend more on various characteristics of the computing infrastructure than on the bits and bytes of any particular virus [10]. Telephone networks have been brought down by obscure bugs in computerized switches, where the chances of any one switch failing was small, but once one had failed, the effect spread to its neighbors in unanticipated ways [11]. What sorts of emergent failures and security weaknesses might we expect to see in highly-connected agent systems? [12], for instance, shows that increasing the intelligence of the elements of a system does not necessarily improve the behavior of the system as a whole; seemingly-intelligent strategies can interact perversely.
As agents begin to exchange, process, retransmit, and otherwise shuffle information, we can expect to see more sophisticated versions of the message-routing loops that occur in current networks.
Relatively simple message tracking solutions suffice to prevent simple loops, but loops that involve more complex transformations and combinations of data are much harder to prevent [13].
Traffic floods caused by such loops can be accidental annoyances, or intentional attacks. Viruses of various kinds may also be a problem in some kinds of agent systems.
For instance, any system that allows one agent to modify or replace another may be vulnerable to a spreading attack, where a piece of malicious code modifies other agents to include a copy of itself.
Eventually, the code might be widely distributed around the system, and able to do various sorts of damage. A bit farther afield, one can imagine an information virus, that spreads between knowledge-based intelligent agents that exchange production rules. An infected agent would contain a set of rules that would make the agent more likely to send those rules to other agents.
See [14] for further speculation on more or less exotic phenomena that may grow in future networks.
We will live in interesting times. Agent-based systems are still new enough that any attempt to predict the problems they will actually face is risky; the field changes every day.
By the time you read this, there will no doubt have been developments that I would have included in this paper if the deadline had been later.
But I hope that the considerations given here provide some flavor of the kinds of thinking needed to make agent systems secure, and the kinds of things that can go wrong if they are not.