Intrusion detection system for internet

Intrusion detection system for internet ABSTRACT The visibility to detect the rapid growth of Internet attacks becomes an important issue in network security. Intrusion detection system (IDS) acts as necessary complement to firewall for monitoring packets on the computer network, performing analysis and incident-responses to the suspicious traffic. This report presents the design, implementation and experimentation of Network Intrusion Detection System (NIDS), which aims at providing effective network and anomaly based intrusion detection using ANOVA (Analysis of Variance) statistic. A generic system modelling approach and architecture are design for building the NIDS with useful functionalities. Solving the shortcomings of current statistical methods in anomaly based network intrusion detection system is one of the design objectives in this project as all of them reflect the necessary improvements in the network-based IDS industry. Throughout the system development of NIDS, several aspects for building an affective network-based IDS are emphasized, such as the statistical method implementation, packet analysis and detection capabilities. A step by step anomaly detection using ANOVA (Analysis of Variance) test has been calculated in the report. Chapter 1 Introduction This chapter is introduction to the whole project. This chapter introduce the project, its motivation, main objective and advance objectives. The chapter also give brief methodology of the research. Introduction The Though with the rapid growth of computer networks make life faster and easier, while on the other side it makes life insecure as well. Internet banking, on line buying, selling, on internet, is now part of our daily life, along with that, if we look at growing incidents of cyber attacks, security become a problem of great significance. Firewalls are no longer considered sufficient for reliable security, especially against zero error attacks. The security concern companies are now moving towards an additional layer of protection in the form of Intrusion Detection System. D.Yang, A.Usynin W.Hines (2006) explain intrusion and intrusion detection as: Any action that is not legally allowed for a user to take towards an information system is called intrusion and intrusion detection is a process of detecting and tracing inappropriate, and incorrect, or anomalous activity targeted at computing and networking resources [16]. Idea of intrusion detection was first introduced in 1980 (J.P Anderson) and first intrusion detection model was suggested in 1987 (D.E.Denning). Intrusion Prevention System (IPS) is considered as first line of defence and Intrusion Detection Systems are considered as second line defence [16]. IDS are useful once an intrusion has occurred to contain the resulting damage. Snot is best example of working Intrusion Detection System and Intrusion Prevention Systems (IDS/IPS) developed by Sourcefire. Which combine the benefits of signature, protocol and anomaly based inspection. IDS can be classified in to misuse detection and anomaly detection. Misuse detection or signature based IDS can detect intrusion based on known attack patterns or known system vulnerabilities or known intrusive scenarios where as anomaly intrusion detection or not-use detection systems are useful against zero -day attacks, pseudo zero-day attack. Anomaly based IDS based on assumption that behaviour of intruder is different from normal user. Anomaly detection systems can be divided into static and dynamic, S.Chebrolu, et al A.Abraham J.P.Thomas (2004). Static anomaly detectors assume that the portion of system being monitored will not change and they mostly address the software area of the system [17]. Protocol anomaly detection could be the best example of static anomaly detection [17]. Dynamic anomaly detection systems operate on network traffic data or audit records and that will be the main area of my interest in research. Anomaly IDS has become a popular research area due to strength of tracing zero-day threats, B.Schneier (2002). It examines user profiles and audit records etc, and targets the intruder by identifying the deviation from normal user behaviour and alert from potential unseen attacks [18]. Active attacks have more tendencies to be traced as compared to passive attacks, but in ideal IDS we try to traces both. Anomaly based Intrusion detection system are the next generation IDS and in system defence they are considered as second line of defence. In that research my main concentration will be Denial of service attacks their types and how to trace them. Motivations Though Internet is the well knowing technology of the day but still there are security concerns such as internet security and availability. The big threat to information security and availability is intrusion and denial-of-service attacks. Since the existing internet was developed about 40 year ago, at that time the priorities were different. Then unexpected growth of internet result exhaustion IPV4 address along with that it brings lots of security issues as well. According to the CERT statistical data 44,074 vulnerabilities had been reported till 2008. Intrusion is the main issue in computer networks. There are too many signature based intrusion detection are used within information systems. But these intrusion detection systems can only detect known intrusion. Another approach called anomaly based intrusion detection is the dominant technology now. Many organizations are working on anomaly based intrusion detection systems. Many organizations such as Massachusetts Institute of Technology are providing data set for this purpose. Motivated by the observation that there is lots of work is done using the Massachusetts Institute of Technology (MIT) data sets. Another aspect of the anomaly based intrusion detection system is statistical method. There are too many good multivariate statistical techniques e,g Multivariate Cumulative Sum (MCUSUM) and Multivariate Exponentially Weighted Moving Average (MEWMA) are used for anomaly detection in the wild of manufacturing systems [3]. Theoretically, these multivariate statistical methods can be used to intrusion detection for examining and detecting anomaly of a subject in the wild of information science. Practically it is not possible because of the computationally intensive procedures of these statistical techniques cannot meet the requirements of intrusion detection systems for several reasons. First, intrusion detection systems deal with huge amount of high-dimensional process data because of large number of behaviours and a high frequency of events occurrence [3]. Second, intrusion detection systems demand a minimum delay of processing of each event in computer systems to make sure an early d etection and signals of intrusions. Therefore, a method which study the variation is called ANOVA statistic would be used in this research. But there is no research available that have implemented ANOVA and F statistic on data sets collected by The Cooperative Association for Internet Data Analysis (CAIDA). The data sets provided by CAIDA are unique in their nature as it does not contain any session flow, any traffic between the attacker and the attack victim. It contains only reflections from the attack victim that went back to other real or spoof IP addresses. It creates trouble in estimating the attack. I will take that trouble as challenge. Research Question In this section I will explore the core objective of the research and a road map to achieve those objectives. During that research I will study data sets called backscatter-2008, collected by CAIDA for denial of services attacks. I will use statistical technique ANOVA to detect anomaly activities in computer networks. My research is guided by five questions. What is an intrusion and intrusion detection system? How can we classify intrusion detection system? What are different methodologies proposed for intrusion detection systems? How to analyse the CAIDA Backscatter-2008 data sets and make them ready for future study and analysis. How to figure out the different types of DOS attacks. How to implement ANOVA statistical techniques to detect anomaly in networks traffics Aims and Objectives Dos attacks are too many in numbers and it is not possible to discuss all the dos attacks in one paper. In this paper I will look to detect anomaly in network traffic using number of packets. Main/Core objectives of the research Review literature of recent intrusion detection approaches and techniques. Discuss current intrusion detection system used in computer networks Obtaining a data set from CAIDA organization for analysis and future study. Pre-process the trace collected by CAIDA, make it ready for future analysis. Recognizing the normal and anomaly network traffic in CAIDA dataset called backscatter-2008. Investigate Analyse deviated network traffic using MATLAB for different variants of denial of services attacks. Review of existing statistical techniques for anomaly detection Evaluation of the proposed system model Advance Objectives of the research Extend the system model to detect new security attacks. Investigating and analysing the ANOVA statistical techniques over other statistics for anomaly detection in computer networks. Nature and Methodology The area of research is related with detecting anomaly traffic in computer networks. The revolution in processing and storage capabilities in the computing made it possible to capture, store computer network traffic and then different kind of data patterns are derived from the captured data traffic. These data patterns are analysed to build profile for the network traffic. Deviations from these normal profiles will be considered anomaly in the computer network traffic. This research presents a study of vulnerability in TCP/IP and attacks that can be initiated. Also the purpose of research is to study TCP flags, find distribution for the network traffic and then apply ANOVA statistical techniques to identify potential anomaly traffic on the network. Report Structure Chapter 1: Introduction This chapter is about the general overview of the project .First of all introduction about the topic is given then motivation of the research is discussed. Core objectives and general road map of the project is discussed under the heading of research question. Aims and objectives are described to enable readers to understand the code and advance objectives of the research and general overview of the research. Nature and Methodology includes the nature of research and what methods will be used during that research to answer the research question and to achieve core and advance objectives. Lastly at the end all chapters in the report are introduced. Chapter 2: Research Background The main focus of this chapter to explain what is Intrusion and Detection why we need Intrusion Detection Systems, types and techniques being used for Intrusion Detection Systems, Challenges and problems of Intrusion Detection System. Chapter 3: Security Vulnerabilities and Threats in Computer Networks This area of report is dedicated to the Network Security in general and issues with computer networks. Then types of Denial of services attacks are described in general. This chapter also include Types of DOS attacks and brief description of each attack. Chapter 4: Data Source Data sets collected and uploaded by CAIDA on their web site are not in a format to be processed straight away. This chapter described in detail how to obtain those data sets. Then all the necessary steps that are carried out on the data sets to convert that trace into format that is understood by MATLAB for final analysis. It also includes the problems faced during the pre-processing of data sets as there not enough material available on internet for pre-processing of datasets and the application used during that phase. Chapter 5: System Model As the research is based on TCP/IP protocol So it is vital to discuss the TCP and the weak points that allow that attacker to take advantage and use them for malicious purpose. What measures could be taken to recognize the attacks well before they happen and how to stop them. In this chapter I will discuss the Intrusion detection Model and features of proposed IDS and finally the steps in proposed model. Chapter 6: ANOVA Statistic and Test Results Implementation in Proposed Model This chapter is the core chapter of this project. This chapter all about focus on statistical test in intrusion detection systems particularly on ANOVA statistics. In this chapter first, the existing statistical techniques are analysed for intrusion detection. ANOVA calculation, deployment in intrusion detection system, backscatter-2008 data set distribution and other categories wise distribution will be explained in this chapter. Finally in the chapter, includes the graphs of the data sets and ANOVA and F statistic graphs are shown. Chapter 7: Discussion and conclusion Finally I will sum up my project in this chapter. It will include conclusion of research. Personal improvements of during that project because during that project I been through my experiences that later I found in the project that is helpful in other areas. Finally the goals that are achieved through entire project. Summary This chapter will enable reader to understand the general overview of the research. First of all the different research questions are identified. Then the objectives of the research are described which includes both core and advanced objectives. What is the nature of the research and which method will be used in it are in picture. The topic provides overall background information. Furthermore explanation of the report structure and brief description of all the chapters are also included in this chapter. Chapter 2 Research Background Introduction The focus of this chapter is to explain, what is intrusion and intrusion detection system. Why we need Intrusion Detection System. This chapter also discuss types and techniques used for Intrusion Detection Systems. Goals, challenges and problems are the main parts of the Intrusion Detection System are also explained in this chapter. Intrusion Detection System (IDS) A computer intrusion is the number of events that breaches the security of a system. Such number of events must be detected in proactive manner in order to guarantee the confidentiality, integrity and availability of resources of a computer system. An intrusion into an information system is a malicious activity that compromises its security (e.g. integrity, confidentiality, and availability) through a series of events in the information system. For example intrusion may compromise the integrity and confidentiality of an information system by gaining root level access and then modifying and stealing information. Another type of intrusion is denial-of-service intrusion that compromises the availability of an information system by flooding a server with an overwhelming number of service requests to the server over short period of time and thus makes services unavailable to legitimate users. According to D. Yang, A. Usynin W. Hines, they describe intrusion and intrusion detection as: An y action that is not legally allowed for a user to take towards an information system is called intrusion and intrusion detection is a process of detecting and tracing inappropriate, and incorrect, or anomalous activity targeted at computing and networking resources. Why we need Intrusion Detection System To provide guarantee of integrity, confidentiality and availability of the computer system resources, we need a system that supervise events, processes and actions within an information system [1]. The limitations of current traditional methods, misconfigured control access policies and also the misconfigured firewalls policies in computer systems and computer network security systems (Basic motivation to prevent security failures), along with increasing number of exploitable bugs in computer network software, have made it very obvious to design security oriented monitoring systems to supervise system events in context of security violations [1]. These traditional systems do not notify the system administrator about the misuses or anomaly events in the system. So we need a system which provides proactive decision about misuse or anomaly events, so therefore from last two decades the intrusion detection systems importance is growing day by day. Now a days intrusion detection system plays vital role in an organization computers security infrastructure. Types of Intrusion Detection System Intrusion detection system is a technique that supervises computers or networks for unauthorized login, events, activity, or file deletion or modifications [1]. Intrusion detection system can also be designed to monitor network traffic, so it can detect denial of service attacks, such as SYN, RST, ICMP attacks. Typically intrusion detection system can be classified into two types [1]. Host-Based Intrusion Detection System (HIDS) Network-Based Intrusion Detection System (NIDS) Each of the above two types of intrusion detection system has their own different approach to supervise, monitor and secure data, and each has distinct merits and demerits. In short words, host based intrusion detection system analyse activity occurrence on individual computers, while on the other hand network based IDSs examine traffic of the whole computer network. Host-Based Intrusion Detection System Host based intrusion detection gather and analyse audit records from a computer that provide services such as Password services, DHCP services, web services etc [1]. The host based intrusion detection systems (HIDS) are mostly platform dependent because each platform has different audit record from other platforms. It includes an agent on a host which detect intrusion by examining system audit records, for example audit record may be system calls, application logs, file-system modification (access control list data base modification, password file modification) and other system or users events or actions on the system. Intrusion detection system were first developed and implemented as a host based [1]. In host based intrusion detection systems once the audit records is aggregated for a specific computer, it can be sent to a central machine for analysis, or it can be examined for analysis on the local machine as well. These types of intrusion detection systems are highly effective for detecting inside intrusion events. An unauthorized modification, accesses, and retrieval of files can detect effectively by host based intrusion detection system. Issues involve in host based intrusion detection systems is the collection of audit records for thousands of computer may insufficient or ineffective. Windows NT/2000 security events logs, RDMS audit sources, UNIX Syslog, and Enterprises Management systems audit data (such as Tivoli) are the possible implementations of the host based intrusion detection system. Network-Based Intrusion Detection System Network-based intrusion detection system (NIDS) is completely platform independent intrusion detection system which predicts intrusion in network traffic by analysing network traffic such as frames , packets and TCP segments (network address, port number, protocols TCP headers, TCP flags etc) and network bandwidth as well. The NIDS examines and compared the captured packets with already analysed data to recognize their nature for anomaly or malicious activity. NIDS is supervising the whole network, so it should be more distributed than HIDS. NIDS does not examine information that originate from a computer but uses specials techniques like packet sniffing to take out data from TCP/IP or other protocols travelling along the computer network [1]. HIDS and NIDS can also be used as combination. My project focus on network based intrusion detection systems, in this project we analyse TCP flags for detecting intrusions. Techniques Used in Existing IDS In the above section we discussed about the general existing type of the intrusion detection system. Now the question arises that how these intrusion detection system detect the intrusion. There are two major techniques are used for above each intrusion detection system to detect intruder. Signature Detection or Misuse Detection Anomaly Detection Signature Detection or Misuse Detection This technique commonly called signature detection, this technique first derives a pattern for each known intrusive scenarios and then it is stored in a data base [3]. These patterns are called signatures. A signature can be as simple as a three failed login or a pattern that matches a specific portion of network traffic or it may be a sequence of string or bits [1]. Then this technique tests the current behaviour of the subject with store signature data base and signals an intrusion when there is a same pattern match. The main limitation in this technique, that it cannot detect new attacks whose signatures are unknown. Anomaly Detection In this technique the IDS develop a profile of the subjects normal behaviour (norm profile) or baseline of normal usage patterns. Subject of interest may be a host system, user, privileged program, file, computer network etc. Then this technique compare the observed behaviour of the subject with its normal profile and alarm an intrusion when the subjects observe activity departs from its normal profile [3]. For comparison, anomaly detection method use statistical techniques e,g ANOVA K-mean, Standard Deviations, Linear regressions, etc [2]. In my project, I am using ANOVA statistic for anomaly detection. Anomaly detection technique can detect both known and new intrusion in the information system if and only if, there is departure between norm and observed profile [3]. For example, in denial of service attack, intrusion occurs through flooding a server, the ratio of the events to the server is much higher than the events ratio of the norm operation condition [3]. Issues and Challenges in the IDS An intrusion detection system should recognize a substantial percentage of intrusion while maintain the false alarm rate at acceptable level [4]. The major challenge for IDS is the base rate fallacy. The base rate fallacy can be explained in false positive false negative. False positive means when there is no intrusion and the IDS detect intrusion in the event. False negative when there is an intrusion in the events and the IDS does not detect it. Unfortunately, the nature of the probability includes, and the overlapping area between the observed and training data, it is very difficult to keep the standard of the high rate of detections with low rate of false alarms [4]. According study held on the current intrusion detection systems depicted that the existing intrusion detection systems have not solved the problem of base rate fallacy [4]. Summary An intrusion into information system compromises security of the information system. A system, called intrusion detection is used to detect intrusion into information system. The two major types of IDS are HIDS and NIDS. The host based intrusion detection system monitor mostly the events on the host computer, while the NIDS monitor the activity of the computer network system. There are two approaches implemented for intrusion detection in IDS, anomaly and signature. Anomaly use statistical methods for detecting anomaly in the observed behaviour while signature check patterns in it. Base rate fallacy is the major challenge for IDS. Chapter 3 Security Vulnerabilities and threats in Networks Introduction In this chapter we are going to discuss the computer and network security. For computer security, there are some other terminologies like vulnerability, exploitability and threats are discussed as well in the chapter. Then chapter focus on Denial of Service attack, which is the most dominant attack in the wild of computer science. The chapter also concentrate the all aspects of the denial of service attack. Computer Security In the early days of the internet, network attacks have been a difficult problem. As the economy, business, banks and organization and society becomes more dependent on the internet, network attacks put a problem of huge significance. Computer security preclude attacker from getting the objectives through unauthorized use of computers and networks [5]. According to the Robert C. Searcord Security has developmental and operational elements [5]. Developmental security means, developing secure software with secure design and flawless implementation [5]. Operational Security means, securing the implemented system and networks from attacks. In computer security the following terminologies are used most commonly [5]. Security Policy: A set of rules and rehearses that are typically implemented by the network or system administrator to their system or network to protect it from attacks are called security policies. Security Flaw: A software fault that offers a potential security risk is called security flaw. Vulnerability: the term vulnerability is a set of conditions through malicious user implicitly or explicitly violates security policy. Exploit: a set of tools, software, or techniques that get benefit of security vulnerability to breach implicit or explicit security policy [5]. The term information security and network security are often used interchangeably. However, this project focus intrusion in computer networks, so we are going to discuss network security. The term network security is the techniques that are used to protect data from the hacker travelling on computer networks. Network security Issues There are many issued involved in the network security but the following are the most common. Known vulnerabilities are too many and new vulnerabilities are being discovered every day. In denial of service attack when the malicious user, attack on the resources of the remote server, so there is no typical way to distinguish bad and good requests. Vulnerability in TCP/IP protocols. Denial of service Attacks A denial of service attacks or distributed denial of service attack is an attempt to make computer resources exhausts or disable or unavailable to its legitimate users. These resources may be network bandwidth, computing power, computer services, or operating system data structure. When this attack is launched from a single machine, or network node then it is called denial of service attack. But now days in the computer wild the most serious threat is distributed denial of service attack [4]. In distributed denial of service attack, the attacker first gain access to the number of host throughout the internet, then the attacker uses these victims as launch pad simultaneously or in a coordinated fashion to launch the attack upon the targets. There are two basic classes of DoS attacks: logic attacks and resource attacks. Ping-of-Death, exploits current software flaws to degrade or crash the remote server is an example of the logic attacks. While on the other hand in resource attacks, the victims CPU, memory, or network resources are overwhelmed by sending large amount of wrong requests. Because the remote server, does not differentiate the bad and good request, so to defend attack on resources is not possible. Various denials of service attacks have some special characteristics Oleksii ignatenko explain the characteristics of the denial of service attacks as in the figure 1. Your browser may not support display of this image. Figure 1 Denial of service attack characteristics Attack type: a denial of service can be a distributed (when it comes from many sources) or non-distributed (when it comes from only one source). Attack Direction: attack direction may be network or system resources. Attack Scheme: Attack Scheme can be direct from malicious users source or it can be reflections form other victims systems, or it can be hidden. Attack Method: Method means that vulnerability that allows attack. Targeted attack utilizes vulnerability in protocols, software and services, while consumption method consumes all possible resources. Exploitive attacks take advantages of defects in operating system. operating system Methods for Implementing Denial of Service Attacks A denial of service attack can be implemented in many ways; the following are the most common implantation techniques Attempt to flood a network, thereby stopping legitimate network traffic Attempt to interrupt connections between two systems, thereby preclude access to a service Attempt to prevent a specific user from accessing a service The flood method can be deployed in many ways but the following are well known in the wild of networks system. TCP-SYN Flood ICMP Flood RST attack TCP-SYN Flood: In order to achieve the TCP-SYN flood the attacker tries to establish the connection to the server. Normally a client establishes a connection to the server through three way handshake. In three way handshake, The client or any sender sends the TCP packet with the SYN flag set. The server or receiver receives the TCP packet, it sends TCP packet with both SYN and ACK bits are set. The client receives SYN-ACK packet and send ACK packet to the server. The three way handshake can easily be understood in the figure 2: Client Server Your browser may not support display of this image.Your browser may not support display of this image.Your browser may not support display of this image. Your browser may not support display of this image. Your browser may not support display of this image. Figure 2 Three way Handshake This is called three way handshake of TCP connection establishment. So in SYN flood what the attacker does, he sends SYN packet to the server and the server responds with SYN-ACK packets but the attacker does not sends the ACK packet. If the server does not receive the ACK packet from the client it will resends a SYN-ACK packet again after waiting for 3 seconds. If SYN-ACK still does not arrive, the server will send another SYN-ACK after 6 seconds. This doubling in time continuous for a total of 4 or 6 attempts (the exact number depends upon the implementation of the TCP protocol on the server side) [8]. So in SYN flood the attacker install Zombies on Internet hosts and sends huge amount of SYN request from spoof IP to the server or any host on the internet and utilize all the server or host memory and data structure. In this way the server get busy and is not able to accept request or respond to