JOB WELL DONE !

Dec 9 2007 is the date when I am not working on my project! It's been couple of days ! The University deadline was Dec 7 2007 and somehow I made it ... no no no... WE MADE IT TOGETHER. (Thanks buddy, like every time you came to rescue me and made me through) "If He will get you to it then He will get you through it". I am always on the verge of giving up and somehow you put me back.

It wasn't somehow though. I remember the time when I was there. I can feel it from that hunger, that aching back, those sleepless nights and the crashing drowsiness, the confusion, the stress, the race to beat the deadline, hours of constant stare on the computer screen, repetitive efforts to attain something with perfection and all amidst millions of distractions. It may not be the best thing in the world but somehow it was the best thing for me. I remember coming out of the graduate studies office after submitting my project report. That feeling, that momentary happiness, that smile on my face coming out from deep within the heart(!). Nobody was watching me. Nobody was there to greet me. I just stretched my hand to Thank You and you greeted me well. In the best possible way anyone ever could.

These words are written to flatter you with the desire of having this time again and again. But all I can say is YOU KNOW THE BEST. I tried and I failed but what I received in the end was and will ever be a precious gift. I will make an effort not to stop this journey here. I still have a long way to go and I want to tell you I will try to do my best. You know that I includes 'YOU' too ! THANK YOU... THANK YOU.

And yea, some things left are fine tuning the code, testing the working of Telnet, SSH and SMTP Protocols, and understanding Linux Network Stack.

Project Report Submission Deadline

Today was a "tentative" deadline to submit a project report to department.
I am pretty much done with that, not completely though.

The last thing Dr Ouyang told me was to GET THE RIGHT THING RIGHT!
I have to compare the performance of SNIPHER with Wireshark and tcpdump. Some perfect quantitative data having user understandable form !

I have got 'vmstat' at work and I will get it done tomorrow.
I need to measure Memory usage, CPU usage, and Network Latency, in the sense difference in the RTT.

Anything left is FINE TUNING THE CODE ! WE ARE GOING TO DO IT, TOGETHER " . . Thank you Father . . .

SNIFFING THE PARSED DATA

At this point of time SNIPHER is not an exact sniffer!! or I think so.
Regular expressions are specified for originating (IP_CT_NEW)connection and I track ONLY those ones.
Other connections will simply pass through.

I need to do something like

1) REGEXes on the NEW connection.

2) On matching connections, it will MARK that connection using conntrack (I dont know how, at this time
!!) and SNIFF all the data on marked connection only.

--> Some thoughts like Creating a HASHTABLE using 4 fields [src_ip, src_port, dst_ip, dst_port].
I believe it will give a unique value on some operation and storing the data in that hashtable !!

3) Once this is done, also look for the utilities online which let you QUANTIFY PERFORMANCE !!

SNIPHER : 11_19_07

32. All these days I've been working on my Project Report and ya I was done with that REGEX thing.

I successfully wrote a REGEX for the http GET request, that was "GET .*HTTP/(0.9|1.0|1.1)"
I still have ftp, Telnet, ssh, SMTP regexes to write !!

OK, now after sticking on the same stuff for two days, I finally found (Thanks Master) why iptables wasn't working on the monitoring ports !! It wasn't able to track the data the data passing by for other machines ( I mean promiscuous mode)

YES, iptables DOESNT WORK IN PROMISCUOUS mode !! Want to know why ???

the data is delivered to the skb from ip_input.c --> ip_rcv(), which has a piece of code which looks like

/* When the interface is in promisc. mode, drop all the crap
* that it receives, do not try to analyse it.
*/
if (skb->pkt_type == PACKET_OTHERHOST)
goto drop;
.
.
.
drop:
kfree_skb(skb);

--> NO skb for the packets of other host !!

*** The next task is to make a change in the above function and make it accept all the packets and then test the code...
I am working on it, lets see how it goes!

And ya REPORT IS DUEEEEEE... !

33. include/linux/if_packet.h

/* Packet types */

#define PACKET_HOST 0 /* To us */
#define PACKET_BROADCAST 1 /* To all */
#define PACKET_MULTICAST 2 /* To group */
#define PACKET_OTHERHOST 3 /* To someone else */
#define PACKET_OUTGOING 4 /* Outgoing of any type */
/* These ones are invisible by user level */
#define PACKET_LOOPBACK 5 /* MC/BRD frame looped back */
#define PACKET_FASTROUTE 6 /* Fastrouted frame */


Packet is made to be one of these types by the Device Driver

11/19/07 || 04:35

SNIFFER_11_01_2007 Some Useful Data Structures

27. struct ip_conntrack

/usr/src/linux-2.4.18.2-34/include/linux/ip_conntrack.h

struct ip_conntrack
{
/* Usage count in here is 1 for hash table/destruct timer, 1 per skb,
plus 1 for any connection(s) we are `master' for */
struct nf_conntrack ct_general;

/* Have we seen traffic both ways yet? (bitset) */
unsigned long status;

/* Timer function; drops refcnt when it goes off. */
struct timer_list timeout;

#ifdef CONFIG_IP_NF_CT_ACCT
/* Accounting Information (same cache line as other written members) */
struct ip_conntrack_counter counters[IP_CT_DIR_MAX];
#endif
/* If we were expected by an expectation, this will be it */
struct ip_conntrack *master;

/* Current number of expected connections */
unsigned int expecting;

/* Unique ID that identifies this conntrack*/
unsigned int id;

/* Helper, if any. */
struct ip_conntrack_helper *helper;

/* Storage reserved for other modules: */
union ip_conntrack_proto proto;

union ip_conntrack_help help;
#ifdef CONFIG_IP_NF_NAT_NEEDED
struct {
struct ip_nat_info info;
union ip_conntrack_nat_help help;
#if defined(CONFIG_IP_NF_TARGET_MASQUERADE) || \
defined(CONFIG_IP_NF_TARGET_MASQUERADE_MODULE)
int masq_index;
#endif
} nat;
#endif /* CONFIG_IP_NF_NAT_NEEDED */

#if defined(CONFIG_IP_NF_CONNTRACK_MARK)
u_int32_t mark;
#endif

#ifdef CONFIG_IP_NF_CONNTRACK_SECMARK
u_int32_t secmark;
#endif

/* Traversed often, so hopefully in different cacheline to top */
/* These are my tuples; original and reply */
struct ip_conntrack_tuple_hash tuplehash[IP_CT_DIR_MAX];

#if defined(CONFIG_IP_NF_MATCH_LAYER7) || defined(CONFIG_IP_NF_MATCH_LAYER7_MODULE)
struct {
char * app_proto; /* e.g. "http". NULL before decision. "unknown" after decision if no match */
char * app_data; /* application layer data so far. NULL after match decision */
unsigned int app_data_len;
} layer7;
#endif

};


28. In ip_tables.h

#define ipt_match xt_match
#define ipt_target xt_target
#define ipt_table xt_table

29. Definition of struct ipt_match (i.e. xt_match) is in

/usr/src/linux-2.6.18.2-34/include/linux/netfilter/x_tables.h


struct xt_match
{
struct list_head list;

const char name[XT_FUNCTION_MAXNAMELEN-1];

/* Return true or false: return FALSE and set *hotdrop = 1 to
force immediate packet drop. */
/* Arguments changed since 2.6.9, as this must now handle
non-linear skb, using skb_header_pointer and
skb_ip_make_writable. */
int (*match)(const struct sk_buff *skb,
const struct net_device *in,
const struct net_device *out,
const struct xt_match *match,
const void *matchinfo,
int offset,
unsigned int protoff,
int *hotdrop);

/* Called when user tries to insert an entry of this type. */
/* Should return true or false. */
int (*checkentry)(const char *tablename,
const void *ip,
const struct xt_match *match,
void *matchinfo,
unsigned int matchinfosize,
unsigned int hook_mask);

/* Called when entry of this type deleted. */
void (*destroy)(const struct xt_match *match, void *matchinfo,
unsigned int matchinfosize);

--------------------------------------------------------------------------
/* Called when userspace align differs from kernel space one */
void (*compat_from_user)(void *dst, void *src);
int (*compat_to_user)(void __user *dst, void *src);
--------------------------------------------------------------------------

/* Set this to THIS_MODULE if you are a module, otherwise NULL */
struct module *me;

char *table;
unsigned int matchsize;
unsigned int compatsize;
unsigned int hooks;
unsigned short proto;

unsigned short family;
u_int8_t revision;
};

30. enum ip_conntrack_info

enum ip_conntrack_info
{
/* Part of an established connection (either direction). */
IP_CT_ESTABLISHED,

/* Like NEW, but related to an existing connection, or ICMP error
(in either direction). */
IP_CT_RELATED,

/* Started a new connection to track (only
IP_CT_DIR_ORIGINAL); may be a retransmission. */
IP_CT_NEW,

/* >= this indicates reply direction */
IP_CT_IS_REPLY,

/* Number of distinct IP_CT types (no NEW in reply dirn). */
IP_CT_NUMBER = IP_CT_IS_REPLY * 2 - 1
};

HOW TO DO REGEX on APP LAYER DATA 11_01_2007

SHARED HEADER FILE

struct ipt_sample {
---
---
char *proto;
char *pattern;
};

USER SPACE MODULE

--> iptables .... -m sample .... --proto [http|ftp] ...


--> switch(proto)
{
case http :
info->proto = http;
info->pattern = REGEX Pattern for http
break;
case ftp :
info->proto = ftp;
info->pattern = REGEX Pattern for ftp
break;
default :
info->proto = null;
info->pattern = null;
break;
}

KERNEL SPACE MODULE

--> COLLECT DATA
--> REGULAR EXPRESSION (proto and pattern)
--> if matches --> Print/Write the data to the file
else skip the packet

SNIFFER_10_31_2007

23. I have gone through the USER SPACE code of libipt_layer7.c and got the idea of how it functions.

24. NEXT TASK is to understand the KERNEL SPACE code and make ipt_sample adapt the methodology.

25. RULES

iptables -A OUTPUT -t mangle -m layer7 --l7proto ftp
iptables -A OUTPUT -t mangle -m sample --src-ip --dst-ip -j ACCEPT

26 . from looking at my code, I THINK MY FETCHING OF DATA IS NOT CORRECT ! Will confirm
it tonight.

Thanks Master !!! Let's see how it goes !

SNIFFER_10_28_2007

22. For Application Layer Data sniffing give support to APP protocols one by one. http, telnet, ftp, ssh. These are widely used protocols.

-> Study their PROTOCOL FORMAT and scan the HEADER and look for the application/agent information if the protocol is including it in there !!?

-> Use REGEX as the L7-FILTER is using (or something else like HASH or MD5 HASH?!!! )

-> for WRITING the sniffed data try to implement them as THREADS

[THREAD1] --------> [SHARED BUFFER ] <---------- [THREAD2] sniffs------------------------critical section-----------------------write to file ->Take help of Prof. Dick Smith..... (how to go for application layer data sniffing !!)

10/28/07||23:33

TILL SNIFFER_10_27_2007

1. libipt_sample.c is done.

2. Error in compilation of ipt_sample.c (also take care of ipt_sample.h**) Did I include it or saved it where it should be?
07/23/07||23:56

3. Definitions of the functions used in ipt_sample is changed. MAKE CHANGES ACCORDINGLY.
07/25/07||23:32

4. I checked all header files regarding the code which is not commented out. The thing is it's not being compiled, and I doubt its because I have to put it in the Kernel, make changes in Config file and then have to go for make or something. Try this steps first before continuing anything. Then work on it later.

"There is nothing what you and I together can't do !" ...toink!
08/06/07||23:46

5. Trying to compile kernel source. I think (not sure) may be something else to compile iptables source only.... but may be not, I guess. WIll do it tomorrow.
08/08/07||00:56

6.Compiled ipt_sample.c successfully. Not its not taking into 'insmod' giving error, FATAL : Module ipt_sample not found. Try to work on it tomorrow night.
08/09/07||00:25

7. insmod worked after "su -". Now packets are not being dropped
08/16/07||22:45

8. Netfilter's kernel modules should be kept here after compiling.
/lib/modules/2.6.18.2-34-default/kernel/net/ipv4/netfilter

9. iptables -I PREROUTING -t mangle -m sample --src-ip 127.0.0.1 --dst-ip 127.0.0.1 -j ACCEPT
08/22/07||00:39

10. struct iphdr(and others)is defined in /include/linux/

11. include/linux/in.h
****
/* Standard well-defined IP protocols. */
enum {
IPPROTO_IP = 0, /* Dummy protocol for TCP */
IPPROTO_ICMP = 1, /* Internet Control Message Protocol */
IPPROTO_IGMP = 2, /* Internet Group Management Protocol */
IPPROTO_IPIP = 4, /* IPIP tunnels (older KA9Q tunnels use 94) */
IPPROTO_TCP = 6, /* Transmission Control Protocol */
IPPROTO_EGP = 8, /* Exterior Gateway Protocol */
IPPROTO_PUP = 12, /* PUP protocol */
IPPROTO_UDP = 17, /* User Datagram Protocol */
IPPROTO_IDP = 22, /* XNS IDP protocol */
IPPROTO_DCCP = 33, /* Datagram Congestion Control Protocol */
IPPROTO_RSVP = 46, /* RSVP protocol */
IPPROTO_GRE = 47, /* Cisco GRE tunnels (rfc 1701,1702) */

IPPROTO_IPV6 = 41, /* IPv6-in-IPv4 tunnelling */

IPPROTO_ESP = 50, /* Encapsulation Security Payload protocol */
IPPROTO_AH = 51, /* Authentication Header protocol */
IPPROTO_PIM = 103, /* Protocol Independent Multicast */

IPPROTO_COMP = 108, /* Compression Header protocol */
IPPROTO_SCTP = 132, /* Stream Control Transport Protocol */

IPPROTO_RAW = 255, /* Raw IP packets */
IPPROTO_MAX
};

12.
iptables -D
rmmod ipt_sample

13. latest ipt_sample module got SUCCESSFULLY compiled with all 3 protocols support ICMP, TCP, UDP.
Now all I need is copy it to the kernel modules directory and insmod and write some good chains and SEE THE MAGIC !!

08/26/2007||01:51

14.
iptables -F PREROUTING -t mangle
iptables -L PREROUTING -t mangle

iptables -I PREROUTING -t mangle -m sample --src-ip 127.0.0.1 --dst-ip 130.86.241.250 -j ACCEPT

15. Jumbopacket
Sniffing the data >= 9000 Bytes which current sniffers dont provide.

16. One more thing, I think the iptables or skb has to be configured to get packets from Wifi Card !! I think it supports only Ethernet card ! Double check on this thing.

09/05/07||21:13

17. Made changes in the options. Can skip either of the IPs for src/dst and that works great, but it only takes "ping localhost"
and not any remote IP like "ping google.com". I think skb is configured to use Ethernet and not Wifi(I guess!!). and trying to find the net_device structure to get the info about the Device skb is using.
09/07/07||01:19

18. Module ipt_sample can track all traffic ICMP, UDP, TCP. The problem is I was inserting the chain on the wrong table.
iptables -I PREROUTING -t mangle -m sample --src-ip 127.0.0.1 --dst-ip 127.0.0.1 -j ACCEPT can track "ping localhost" but it can't track "ping google.com" as ping request for google.com arises from local machine so it appears on OUTPUT chain first and it doesnt go through PREROUTING. The reply can be matched as PREROUTING but it has the SRC and DST addresses altered.
To track "ping google.com" the correct rule should be "iptables -I POSTROUTING -t mangle -m sample --src-ip --dst-ip -j ACCEPT.

Now at this point of time everything works for ACCEPT and DROP target too!!! HURRAYYYYYYYYYY.....

one thing came in my mind... "There is nothing what you and I together can't do!"
09/12/07||01:14

19. Installed L7-Filter
I got L7-Filter to apply Regex on Application Layer Data to classify traffic.
I got three patch files to apply patch to the kernel and to the iptables

1. netfilter-layer7-v2.13
--> Applied patch "kernel-2.6.18-2.6.19-layer7-2.9.patch" to the kernel using

Copied the patch file to the "/usr/src/linux-2.6.18.2-34" directory and then
#cd /usr/src/linux-2.6.18.2-34
#patch -p1 <> Applied patch "iptables-for-kernel-pre2.6.20-layer7-2.13.patch" to the iptables using

Copied the patch file to the directory "/root/Desktop/iptables-1.3.6" and then
#cd /root/Desktop/iptables-1.3.6
#patch -p1 <> Then I did
#chmod +x extensions/.layer7-test"

-->Installed the iptables-1.3.6
#cd /root/Desktop/iptables-1.3.6/
#make KERNEL_DIR=/usr/src/linux-2.6.18.2-34
#make install KERNEL_DIR=/usr/src/linux-2.6.18.2-34

2. l7-protocols-2007-07-27.tar.gz

#tar -C /etc/ -xvf l7-protocols-2007-07-27.tar.gz
#cd /etc/
#mv l7-protocols-2007-07-27 l7-protocols

20. And compiled the kernel again

#cd /usr/src/linux-2.6.18.2-34
#make
Choose option N when asked for layer7 module !

It seems like it got successfully installed as it displayed help message on
#iptables -m layer7 -h

Now let start working on Matching APPLICATION LAYER DATA

09/14/07||21:28
21. Figure out how you can apply REGEX on Layer 7 Data. Its been a long time (See the date above)! Now get it done withing 2-3 weeks and start working on Report and HUNTING JOB!

10/19/07||02:05

21. For Application Level data I am thinking of doing something like

iptables -A POSTROUTING -t mangle
-m sample --src-ip SRCIP --sport SPORT --dst-ip DSTIP --dport DPORT --app APPLICATION -j ACCEPT

Where I can do a REGEX on APPLEVEL DATA and Sniff the traffic accordingly, something like packet matchingthe IPs and PORTs will scan the APPLICATION LAYER DATA or APPLIATION LAYER PROTOCOL and will do the sniffing!!

Keep it in mind and talk to Dr Ouyang !

10/22/07||00:00