Texas A&M Network Security Package Overview 7/1/93 Dave Safford Doug Schales Dave Hess ABSTRACT: Last August, Texas A&M University UNIX computers came under extensive attack from a coordinated group of internet crackers. This package of security tools represents the results of over seven months of development and testing of the software we have been using to protect our estimated twelve thousand internet connected devices. This package includes three coordinated sets of tools: "drawbridge", an exceptionally powerful bridging filter package; "tiger", a set of convenient yet thorough machine checking programs; and "netlog", a set of intrusion detection network monitoring programs. While these programs have undergone extensive testing and modification in use here, we consider this to be a beta test release, as they have not had external review, and the documentation is still very preliminary. A BRIEF HISTORY OF THE INCIDENTS: On Tuesday 25 August 1992, we were notified by Ohio State that a specific TAMU machine was being used to attack one of their computers over internet. The local machine turned out to be a Sun workstation in a faculty member's office. Unfortunately, this faculty member was out of town for a week, so rather than trying to convince the department head to let us in without the faculty present, we decided to monitor network connections to the workstation, and if necessary, disconnect the machine from the net electronically. This decision to monitor the machine's sessions rather than immediately securing it turned out to be very fortunate, as this monitoring gave us a wealth of information about the intruders and their methods. Our initial monitoring tools were very simple, but as the significance of what we were seeing became apparent, we rapidly improved the tools to the point that we were able to watch the intruder's entire session in real time, keystroke by keystroke. This monitoring led to the discovery that several outside intruders were involved, and that many other local machines had been compromised. One local machine had even been set up as a cracker bulletin board machine, that the crackers would use to contact each other and discuss techniques and progress! By Thursday 27 August we felt we had enough information about which machines had been compromised, and how they had been broken into, that we could effectively clean them up. In addition, the severity of the modifications the intruders were making, particularly on the bulletin board machine, made it imperative to stop the intrusions, so we contacted the respective system managers, and arranged to shut down all machines, and scheduled the system cleanup for the next day. On Friday 28 August, we worked on the known affected machines, closing the security holes that had been used to break in, and brought them all back up on the network. On Saturday 29 August, we received an emergency call from one of the system managers, saying that the intruders had broken back into the cracker bulletin board machine. Concerned about the integrity of their research data, they asked for their machines to be physically disconnected from the rest of the network. On Monday 31 August, we analyzed the logs of the new break-in, and determined that 1) the crackers were much more sophisticated than we had been led to believe, and 2) many more local machines and user accounts had been compromised than we initially knew. We even found a file containing hundreds of captured passwords. It appeared that there were actually two levels of crackers: the high level was the more sophisticated, and had a thorough knowledge of the technology; the low level were the "foot soldiers" that merely used the supplied cracking programs with little understanding of how they worked. Our initial response had been based on watching the later, less capable crackers, and was insufficient to handle the more sophisticated ones. After much deliberation, we decided that the only way to protect the computers on campus was to block certain key incoming network protocols, reenabling them to local machines on a case by case basis, as each machine had been cleaned up and secured. The problem was that if the crackers had access to even one unsecure local machine, it could be used as a base for further attacks, so we had to assume all machines had been compromised, unless proven otherwise. The recommendation to filter incoming traffic was presented to the Associate Provost for Computing on Monday afternoon, and approved. We bought or borrowed the necessary equipment for the filter and monitor machines late that afternoon, and worked all night on the design and coding of the filter. Particular effort was made in the design to achieve the necessary security with the minimum of impact to local users. The filter was completed and installed by 5PM Tuesday 1 September. On Thursday 3 September, our monitor logs showed an obviously automated attack by ftp which was sequentially probing every machine on campus. Here again we decided to monitor this attack, as we were unsure what it could accomplish. This decision to observe, rather than immediately block, turned out to be very fortunate. Shortly after midnight on friday we were notified by CERT that another site was monitoring their machines, and had noticed that the crackers had broken back into our machines. We immediately went in and analyzed our logs, and determined that the crackers had used ftp to install a program that allowed them to tunnel past our filter's blocks. In addition, the crackers were now installing some very sophisticated trap doors and trojan programs throughout a large number of machines, apparently to ensure that they were never locked out again. These system modifications were particularly nasty in that they went to great pains to conceal them, including patching the modified binaries to match the original timestamps and checksums. At this point we completely redesigned the filter to keep the crackers out, and installed the new version by 5AM Saturday. The new version, while providing much greater security, also was unfortunately more visible to valid users. The good news was that the new filter interrupted the new sophisticated crackers while they were working (apparently they did not anticipate so rapid a response), and we discovered one machine still had over 40 megabytes of the cracker's tools. This captured information provided us the most detailed information yet as to the cracker's methods. Since the new filter was installed, we have observed no successful intrusion attacks against our machines, despite continued logging of probes and continued attempts. (We did log one incident of a local student trying to attack an off campus machine, but that is another topic.) Our efforts since then have centered in three areas: improving the filter (largely for ease of use, and throughput), improving the monitoring tools (to reduce manpower requirements), and developing a program to help local system managers check their machines for proper security configuration. RESPONSE OVERVIEW: Our response to the intrusion incidents has three major thrusts: filtering, monitoring, and cleaning. Our first line of defense is the bridging filter package drawbridge, which is used to filter all packets to or from the internet. Drawbridge allows us to control internet access on machine by machine and port by port basis on a full ethernet bandwidth basis. While other firewall configurations are known to be stronger, drawbridge provides a level of compromise between security and availability more acceptable to the university environment, and provides much needed flexibility and throughput for our large scale network. Realizing that drawbridge was a compromise between convenience and security, we developed a set of monitoring tools to look for intrusions that might be attempting to circumvent the filter. These tools monitor our internet link continuously, checking for unusual connections, or patterns of connections, and for a wide range of specific intrusion signatures. Finally, we developed tiger scripts, a tool for helping system administrators check the configuration and security status of their individual machines. The following diagram shows an overview of the filter and monitor implementation. In traditional secure gateways, a filter and secure bastion host are used, and all traffic to or from internet is forced through them. This typically means that users need proxy clients for external access, such as for telnet and ftp, so that they all do not have to log on to the bastion host for external access. In our case, the filter allows arbitrary protocol filtering on a host by host basis, so that each department can set up its own authorized hosts, with their own service configurations, (subject to the campus wide minimum standards.) This provides a reasonable level of both security and flexibility for educational and research requirements. For a host to be enabled at all beyond the default incoming permissions for mail, it must pass the very thorough tiger scripts, as described later. The monitor node is placed outside the filter so that it can record connection attempts which are blocked by the filter. This placement has been crucial to recognizing intrusion attacks, but does place the monitor itself at risk. To minimize this risk, we placed both the filter and monitor in a controlled access machine room, and configured the monitor to accept no other network requests other than validated monitor requests. The filter is similarly programmed only to respond to secure filter update requests. \ \/\ T1 to internet \ --------- |WAN | |Router | |(cisco)| --------- | ----------- | |Secure | ethernet |-----|Monitor | "netlog" | |(Sun4/60)| | ----------- | --------- |Filter | |Bridge | "drawbridge" |(486PC)| --------- | | ethernet to campus \/ ---------------------------------------------- | | --------- --------- | | secured | | | | ... ... .... | | (checked with "tiger scripts") | | hosts | | --------- --------- FILTER: (drawbridge) Our first line of defense is the bridging filter running our drawbridge program. We initially looked at using the filtering built into our WAN routers (cisco), but determined that our requirements, particularly in the need for supporting potentially different filtering to each of our roughly 12,000 machines in our class B network, were too complex for the router. In addition we needed something that could handle full ethernet bandwidth, was itself very secure, and which could be implemented VERY rapidly. D. Brent Chapman presented an interesting analysis of the limitations of current filter implementations in his "Network (In)Security Through IP Packet Filtering", Proceedings of the Third UNIX Security Symposium, September 1992. Our drawbridge program, along with its support filter specification language and compiler, address his critical recommendations with respect to both functionality and ease of specification. We decided on a PC with two SMC 8013 (AKA Western Digital) ethernet cards, and based our first software implementation on pcbridge, by Vance Morrison. This initial version was soon rewritten from scratch in turbo C++, to make the addition of needed features somewhat easier. The current filter design provides "allow" based filtering per host with separate incoming and outgoing permissions. (Allow based filtering prohibits all connections except those that are explicitly allowed.) For both performance and configuration management, the filter tables are created on a support workstation, based on a powerful filter configuration language, and then securely transferred to the filter machine, either at boot time, or dynamically during operation. The support machine does all the hard work of parsing the configuration file, looking up addresses, and building the tables, so that the filter itself need only perform simple O(1) table lookups at run time. Updating the tables dynamically is made secure with a DES authentication. Our current default configuration allows any outgoing connection, but allows in only smtp (mail). Several campus and departmental servers have been checked and set up as hosts which are able to receive incoming telnet, ftp, nntp, and gopher requests. MONITOR: The goal of monitoring is to record security related network events, by which intrusion attempts can be detected and tracked. This is a very difficult problem in general. The communication data rates make this problem somewhat like trying to take a sip of water from a fire hose; our campus has some 30 terabytes of internal data transfer per day, and our internet connection is on the order of ten gigabytes per day, with an average of 50,000 individual connections during that period. Clearly we have to be both very selective and flexible in what we monitor, and we need automated tools for reviewing even these resultant logs. Another problem is that of monitor placement. It is important that monitors be placed so that critical segments can be observed, and so that the monitors themselves are secure. Our solution includes the programs "tcplogger", "udplogger", "etherscan", and some associated support programs. The tcp and udp loggers basically log a one line summary for all connection attempts. The associated analysis programs report on suspicious connections or patterns of connections. In addition, these logs have been very useful in analyzing details of security events after the fact. The etherscan program goes much further, actually scanning all packets and their contents, looking for a specific set of intrusion signatures, such as root login attempts from off campus. The details of the intrusion signatures, as well as the etherscan program itself will not be discussed in detail, as it contains sensitive information and techniques that we do not want widely distributed. MACHINE CLEANUP: (tiger scripts) Our third major thrust has been the development of an automated tool for checking a given machine for signs of intrusion, and for network security related configuration items. The resultant tool is actually a set of bourne shell scripts (for high portability). The major goal of the tiger scripts is to provide a simple way to check a machine prior to allowing it any level of greater internet connectivity. For ease of use, the tiger scripts label all outputs with an error classificatio n: --FAIL-- The problem that was found was extremely serious. --WARN-- The problem that was found may be serious, but will require human inspection. --INFO-- A possible problem was found, or a change in configuration is being suggested. --ERROR-- A test was not able to be performed for some reason. The checking performed covers a wide range of items, including items identified in CERT announcements, and items observed in the recent intrusions. The scripts use Xerox's cryptographic checksum programs to check for both modified system binaries (possible trap doors/ trojans), as well as for the presence of required security related patches. Currently the tiger scripts have been configured for SunOS 4.1.x, Solaris 2.0, SVR4, Nextstep 3.0, and UNICOS 7.0 releases. The programs are largely table driven for ease of porting, and we are working on ports to Ultrix, AIX, SGI, etc. POLICIES: The policies and procedures need to provide both security and flexibility. Our resultant decision was to filter incoming traffic other than mail to all machines, and then allow case by case requests for authorized hosts status, based on successful demonstration of basic security configuration with the tiger scripts. In special cases, we have had requests for allowing incoming requests to special servers which are not easily checked, such as for embedded robot controllers. In these cases, we have allowed the connections, but implemented special monitors on these services. Long term policy questions which remain unanswered include how to handle updates in response to critical CERT announcements, and how to handle OS updates. Obviously we need some way to coordinate both periodic and quick response host reviews. Our filter configuration language does support machine classes, so we could do something such as "disable ftp to all SunOS 4.1.1 machines" in response to a CERT announcement of a respective problem, but it would be nice to have a mechanism to communicate such announcements to the respective managers BEFORE cutting off access. The problem on a large campus is maintaining a contact list for a large number of machines, given the high rate of turnover in student managers. In addition, the information in our filter configuration file may rapidly become outdated, as managers update their machines hardware and software. Our current plan is to require annual security checks with the current tiger scripts, enforced with the warning of loss of authorization status. In the case of aperiodic security events or announcements, we will attempt to evaluate the time criticality of responding, and require appropriate event specific checking. As the tiger scripts are so easy to run, we anticipate that this requirement will not be a significant burden to system managers. A recent case in point was the announced security problem with the wuarchive anonymous ftp code. In this case, we knew exactly which machines had ftp authorized, and contacted the respective managers immediately. They all updated their software so rapidly that was not necessary to block access, and the limited number of authorized machines avoided the need for an immediate tiger update. AVAILABILITY: Drawbridge, tiger, and all monitoring tools other than etherscan are now available via anonymous ftp in sc.tamu.edu:pub/security/TAMU. Due to export restrictions, the DES routines used in drawbridge have been put in a separate tar file, and are readable only by US sites. Other sites should have no problem either running the filter without encryption, or dropping in their own favorite encryption package. The distribution of etherscan has been hotly debated within our group. One argument is that etherscan should be freely released, as the crackers already have equivalent knowledge and tools (they do), and restrictions would only hurt valid administrators. The counter argument is that free availability of the intrusion signatures would enable the crackers to design better intrusions, and the availability of sources would provide novice crackers a significant help. Our resultant compromise will be to provide copies to NIC registered site contacts, given an official request on respective letterhead. Requests should be sent to: Dr. Dave Safford Director, Supercomputer Center Texas A&M University MS 3363 College Station, TX 77843-3363 CONCLUSIONS: In response to a significant series of intrusions from internet, we have developed a set of policies and tools for filtering, monitoring and checking. Each of these three areas has proved critical; the filtering for its ability to protect machines from attack, the monitoring because it has yielded significant information about the intruders and their methods, and augments the filter, and the checking tools for their ability to automate the task of checking and cleaning a large number of machines.