What is Intrusion Detection?

Intrusion  detection, is the attempt to monitor and possibly  prevent attempts to intrude into or otherwise compromise your system and network  resources. Simply put,  it works like this: You have a computer system.  It is attached to a network, and perhaps even to the  internet. You are willing to allow access to that  computer system from the  network, by authorised  people, for acceptable  reasons. For example, you have a web server, attached  to the internet, and you  are willing to allow your  clients, staff, and potential clients, to access the web pages stored on that web server.

You  are not, however, willing to allow unauthorised access to that system by anyone,  be that staff, customers, or unknown third parties. For example, you do not  want people (other than the web designers that your company has employed)  to be able to change the  web pages on that computer. Typically, a firewall or authentication system of some kind will be employed to prevent unauthorised  access.

Sometimes, however, simple firewalling  or authentication systems can be broken. Intrusion detection is the set of  mechanisms that you put  in place to warn of attempted unauthorised access to the computer. Intrusion  detection systems can also take some steps to deny  access to would-be intruders.

Why  use Intrusion Detection?

The  underlying reasons why  you might use intrusion detection systems are relatively straightforward: You want  to protect your data and  systems integrity. The  fact that you cannot always protect that data integrity  from outside intruders  in today's internet environment using mechanisms such as ordinary password and file security, leads to a range  of issues. Adequate system  security is of course the first step in ensuring data protection. For example,  it is pointless to attach  a system directly to the  internet and hope that  nobody breaks into it,  if it has no administrator  password! Similarly, it  is important that the system  prevents access to critical files or authentication  databases (such as the NT SAM or the Unix /etc/passwd or /etc/shadow files) except by authorised systems administrators.

Further  measures beyond those normally  expected of an intranet system should always be  made on any system connected  to the internet. Firewalling  and other access prevention mechanisms should always be put in place. While it may be acceptable to allow NT logon, file sharing,  or telnet access to a system  that is entirely internal,  an internet server should  always use more secure  mechanisms, such as firewalling off the NT file sharing  (SMB protocol) ports such as TCP/UDP ports 137 - 139, and using secure shell  (SSH) instead of telnet for access to Unix systems.

Intrusion  detection takes that one  step further. Placed between  the firewall and the system being secured, a network based intrusion detection  system can provide an extra layer of protection to that system. For example,  monitoring access from  the internet to the sensitive data ports of the secured  system can determine whether  the firewall has perhaps  been compromised, or whether  an unknown mechanism has  been used to bypass the security mechanisms of the firewall to access the network being protected.

What types of Intrusion Detection systems are there?

Intrusion  Detection systems fall  into two broad categories.  These are:

• Network  based systems. These  types of systems are  placed on the network,  nearby the system or  systems being monitored.  They examine the network  traffic and determine whether it falls within  acceptable boundaries.
• Host based systems. These types of systems actually  run on the system being monitored. These examine the system to determine  whether the activity  on the system is acceptable.

A  more recent type of intrusion detection system are those that reside in the operating system kernel and monitor  activity at the lowest level of the system. These systems have recently started  becoming available for  a few platforms, and are  relatively platform specific.

Network  Based Intrusion Detection

Network  based intrusion detection systems are those that monitor traffic on the entire network segment.  A network interface card  (NIC) can operate in one  of two modes, these being:

• Normal mode, where packets which  are destined for the  computer (as determined by the ethernet or MAC address of the packet) are relayed through to the host system.
• Promiscuous  mode, where all packets that are seen on the ethernet are relayed to the host system.

A  network card can normally be switched from normal  mode to promiscuous mode, and vice-versa, by using a low-level function of  the operating system to talk directly to the network card to make that change.  Network based intrusion detection systems normally require that a network interface card is in promiscuous mode.

Packet Sniffers and Network Monitors

Packet Sniffers and Network Monitors  were originally designed  to aid in the process of  monitoring the traffic  on an Ethernet network. The first of these were  two products; Novell LANalyser  and Microsoft Network Monitor.  These products basically capture all packets that they see on the network. Once the packets are captured, a number of possibilities  arise:

• Packets  can be counted. Counting  the packets that come past, and adding together  their total size over a period of time (including  overheads such as packet  headers) gives a pretty good indication of how heavily loaded the network  is. Both LANalyser and  Microsoft Network Monitor provide load graphs or meters to show the relative load of the network.
• Packets  can be examined in detail.  For example, you might  want to capture a set of packets arriving at a web server to diagnose some problem with the  server.

Packet sniffing products have become more sophisticated  in recent years. These  products can disassemble  the insides of various types of packets to show what type of communication is happening inside that packet. These tools can  be used to do evil as well  as good. For example, packet  sniffing can be used to find out someone's Unix  password by sniffing telnet packets to the machine that they connect to. Once an attacker has compromised  your network, one of the first things they might  install is a packet sniffer of some kind.

Packet Sniffing and Promiscuous Mode

All  packet sniffers will require  that a network interface  is in promiscuous mode. Only in promiscuous mode will every packet received by the NIC be passed up  to the packet sniffer itself. The packet sniffer normally  requires administrative privileges on the machine  being used as a packet  sniffer, so that the hardware of the network card can  be manipulated to be in promiscuous mode. Another  point to consider is the  use of switches, rather than hubs, in a network. Note that packets received on one interface of a switch are not always sent to other interfaces of the  switch. For this reason,  a heavily switched environment,  rather than an all-hubs (single segment) environment,  will often defeat the use of packet sniffers.

Network  Based Intrusion Detection:  The Evolution of the Packet Sniffer

Unfortunately, from a security point of view, a packet sniffer is of limited benefit. The task of capturing every  packet on the network,  disassembling it, and manually  taking action based on  the contents of the packet  is far too time-consuming,  even for a horde of specially trained network gnomes.  What if we were to have some software that automated  the process for us (after all, that is what computers  are for in the first place, is it not?)

This is basically exactly what  a network based intrusion detection package does.

Examples of some of the types of  intrusion detection that  an IDS engine can perform are:

• Examine  packets that pass through the network.
• For  legitimate packets, allow them to pass (perhaps  recording them for future analysis).
• Where  a packet endangers the  security or integrity of a target system, stop transmission of the packet by sending TCP "connection  closed" or ICMP "port unreachable" messages  to both the target system and the system sending the packet.

In this manner, an IDS can  perform an effective second layer defence for a target system when hosted behind  a firewall. Some implementations have also used an IDS in place of a firewall, which is, however, something that Oxtrad Ltd. do not  recommend. Network based  intrusion detection can perform a few other tasks, for example:

• Monitoring the network for obvious  port scans. Before compromising a system, a cracker will often port scan the system to determine what vulnerabilities  might exist. Port scan  attempts from a host  on the internet can often be a signal that a person  on such a host intends  to damage your network.
• Monitor  valid connections for well known attacks. Accessing  a web server host on  the web server port (80)  might be seen as a relatively  harmless activity, but  some access attempts  are in fact deliberate  attacks, or attempts at attacks. For example, an access that looks like "GET /../../../etc/passwd HTTP/1.0" is probably  a bad sign, and should be blocked.
• Identify IP spoofing attempts of various sorts. The  ARP protocol that is  used to convert IP addresses  to MAC addresses is often a target for attack. By sending forged ARP  packets over an ethernet, an intruder who has obtained access to one system can also "pretend" to  be operating as a different system. This can lead  to denial of service  attacks of various sorts, as well as system hijacking, whereby an important server (such as a DNS  server or authentication  server) is "spoofed". Crackers can use this  "spoofing" to redirect  packets to their own  system, and perform "man  in the middle" type attacks on what would otherwise  be a secure network.  By keeping a register of ARP packets, a network  based intrusion detection system can identify the  source (ethernet address) of a compromised system  and flush out would-be  crackers.

When unwanted activity is detected, network based intrusion  detection can take action,  including interfering with  future traffic from the intruder, or reconfiguring a nearby firewall to block all traffic coming from  the intruder's computer or network.

Host Based Intrusion Detection

Once a network packet has arrived at the host that it was  intended for, there is  still available a third line of defence behind the firewall and network monitor. This is called  "host based intrusion detection", and comes in several flavours.

The  two main types of host  based intrusion detection are:

• Network  monitors. These monitor incoming network connections to the host, and attempt to determine whether any of these connections represent a threat. Network  connections that represent  some kind of intrusion  attempt are acted on.  Note that this is different to network based intrusion detection, as it only  looks at network traffic  coming to the host it  is running on, and not  all traffic passing the network. For this reason it does not require promiscuous  mode on the network interface.
• Host monitors. These monitor  files, file systems,  logs, or other parts  of the host itself to look for particular types  of suspicious activity that might represent an intrusion attempt (or a successful intrusion). Systems administration staff can then be notified about any problems that  are found.

Monitoring Incoming Connections

It is possible on most hosts  to monitor packets that attempt to access the host before those packets are passed onto the networking layer of the host itself.  This mechanism attempts to protect a host by intercepting  packets that arrive for the host before they can do any damage.

Some of the actions that can  be taken include:

• Detect incoming connection attempts to TCP or UDP ports that are unauthorised, such as attempts to connect to ports where there are no services. This  is often indicative of  a possible cracker having a "poke around" to find  weaknesses.
• Detect incoming portscans. Again, this is a definite issue that should be addressed,  and alerting a firewall or modifying the local  IP configuration to deny  access from a possible  intruder host (eg: by using ipchains on Linux) is one action to take.

Monitoring Login Activity

Despite  the best efforts of the network administrator, and the most recently deployed and monitored intrusion  detection software, occasionally  an intruder will slip by  and manage to log on to a system using a unknown type of attack. Perhaps  an attacker will have obtained  a network password by some  means (packet sniffing  or otherwise) and now has the ability to log on to the system remotely.

Products are available that will  look for unusual activity on a system. This type of package monitors log-in and log-out attempts, and  alerts the system administrator to activity that is unusual  or unexpected.

Monitoring Root Activity

The  aim of all intruders is to obtain super-user (root)  or administrator access on the system that they  have compromised. Well-maintained and reliable systems that  are used as web servers and databases will usually have little or no activity by the super-user, except  at particular times of  the day or night for scheduled  system maintenance. Fortunately, crackers do not believe  in system maintenance.  They rarely stick to scheduled  downtime windows and often  work at odd hours of the  day. They perform activities  on the system that are  unusual for even the most propeller-headed system  administrator.

There  is one more line of defence:  Monitoring any actions  performed by the root user  or system administrator.  Many unix systems allow logging or other monitoring  of all activity by the  root user, packages are available that can then  scan these logs for unusual  activity and notify others  about it.

Administrators of open-source operating systems have one final option: alter the kernel to log specific types of activity. How to do this is outside of the scope  of this technology overview,  however speak to your account manager who can arrange  an for an Whitehelm consultant  who will be able to perform this task.

Monitoring the File Systems

Once an intruder has compromised  a system (and despite your  best hopes, and the best  efforts of the intrusion  detection systems that  you have laid down, one cannot discount entirely the possibility that one day an intruder will compromise a system), then they will  start to change files on  the system. For example,  a successful intruder might want to install a packet sniffer or portscan detector,  or modify some of the system  files or programs to disable  some of the intrusion detection  methods that they have  worked around.

Installing software on a system usually involves modifying some  part of that system. These  modifications will usually  take the form of modifying files or libraries on the  system.

There  are programs available  designed to detect when files change on the system,  and alert the system administrator  to any changes.

For  example, the following  approaches can be taken:

• Create MD5 or other cryptographic checksums of all of the  system files, and store these in a database. When a file changes,  its checksum will change.
• Store  the creation or modification  date and time of every  system file. Look for any changes in these timestamps.
• Keep a record of any suid (run-as-root) program  on the system. If any of these change, or if new ones are installed, or any are deleted, then there is a problem.

The  methods used by the various products vary somewhat,  but they are all based  on the above mechanisms.  These programs also take care to make sure that the database of known cryptographic  checksums itself isn't  compromised in any way. For example, it would be possible for an intruder with a knowledge of the  operating system and the  intrusion detection software to modify the system files and also modify the cryptographic  checksum database so that it would appear that nothing  had changed.

Kernel Based Intrusion Detection

Kernel based intrusion detection  is a relatively new art form, and one that is starting to become prevalent, especially  within Linux. There are  two main kernel based intrusion detection systems currently  available for Linux. These  systems take the approach  of preventing buffer overflows, increasing file system protection, blocking signals,  and generally making it  difficult for an attacker to compromise a system.  One of them also takes  steps to prevent certain  actions by the root user, such as installing a packet  sniffer or changing firewall  rules.

Kernel Protection vs File System  Monitoring

Obviously, systems like LIDS and systems  such as Tripwire take a rather different approach  to attempting to achieve  the same thing. Both of these packages attempt to prevent a cracker from  using the system for unauthorised purposes.

At first glance, one might  think that a system such  as Tripwire is less than  perfectly useful. While it is a Good Thing to monitor  file systems for signs  of abuse, it is readily accepted that once your  system has been compromised by an external intruder, it is time to shut down  and rebuild. The damage has been done, and the system integrity cannot  be guaranteed so it is  best to re-build your operating system from the pristine version supplied on CD from your vendor. The approach offered by LIDS seems much more attractive - to protect the system from damage rather than to note that the gate was left open after the horse has bolted,  so to speak.

Conclusions

It is possible, using the most up to date tools that are available, to protect  against virtually every type of threat that is currently known about. Unfortunately, new threats and security holes in some software package or another  are being discovered on  a daily basis.

It is important in any environment  to know what types of threats you might be facing. Be  aware of any potential  security holes in your system, and take care to prevent attacks against  these. For example, a web server that is connected  to the internet and placed  behind a firewall may be  reasonably secure against most packet based attacks, but a CGI program on the server might expose a vulnerability. Pay special attention to ensuring that CGI programs correctly bounds check all arrays and validates input data before processing.  An intrusion detection  program between the firewall and the web server could be configured to throw out any accesses that are  suspicious.

Click  on image for larger view

We have discussed a number  of different types of tools in this technology overview, all with different functions.  To keep your environments as secure as possible, it is important to choose  tools from across the range of functions. Each of the  tools forms an extra layer  of protection in case the others are defeated. So, your first layer of protection should be a firewall. Behind that, a network  based intrusion detection system will catch any breaches of the firewall. Behind  that again, a set of tools  that monitor connection attempts, which will form  an extra layer of protection. Finally, if all else fails, tools to catch an actual break-in form the final  layer of defense.

Please call 0870 421 4023 to find out how developing  an intrusion detection  strategy will benefit your  company, or e-mail ids.team@Whitehelm.com requesting more information.