Extract data from packet dump (pcap)

Archived from groups: comp.dcom.lans.ethernet (More info?)

In article <pan.2005.01.12.10.31.50.964778@privacy.net>,
nntp chip <me@privacy.net> wrote:
:I have some data recorded with tcpdump that needs to be reassembled. The
:idea is to filter out what is relevant and then glue together just the
 ata-part of every packet into a file.

With or without the possibility that the capture includes retransmissions?
If out of order data is detected, should the data be buffered or be
discarded? I've recently read that linux puts packet fragments out
in reverse order (last first), so some amount of packet reassembly
may have to be done. I guess when I framing the question I was thinking
of TCP ordering, but I notice now that you do not indicate TCP specifically.

If the data does -not- consist of nice unfragmented in-order post-
error recovery TCP packets, then I'd suggest grabbing the Linux
or FreeBSD IP static and hacking it to suit your needs. No point in
re-writing all the retransmission logic if you don't need to.
--
Oh, to be a Blobel!

Archived from groups: comp.dcom.lans.ethernet (More info?)

On Wed, 12 Jan 2005 11:30:54 +0000, Walter Roberson wrote:

> In article <pan.2005.01.12.10.31.50.964778@privacy.net>, nntp chip
> <me@privacy.net> wrote:
> :I have some data recorded with tcpdump that needs to be reassembled.
> The :idea is to filter out what is relevant and then glue together just
> the  ata-part of every packet into a file.
>
> With or without the possibility that the capture includes
> retransmissions? If out of order data is detected, should the data be
> buffered or be discarded? I've recently read that linux puts packet
> fragments out in reverse order (last first), so some amount of packet
> reassembly may have to be done. I guess when I framing the question I
> was thinking of TCP ordering, but I notice now that you do not indicate
> TCP specifically.
>
> If the data does -not- consist of nice unfragmented in-order post- error
> recovery TCP packets, then I'd suggest grabbing the Linux or FreeBSD IP
> static and hacking it to suit your needs. No point in re-writing all the
> retransmission logic if you don't need to.


Just getting the raw data would sometimes be good enough.

My main reason for reassembling data is to find bad behaviour in our
networks. Most things can be filtered and read in ethereal or ngrep. But when
it comes to streaming media and transferred files it gets harder to tell
when something bad is happening. That data can be any of TCP or UDP.

Since I cannot do any programming myself I am searching for existing
tools that can read a pcap-file, do all the needed magic and then write
the payload to a file/stdout.



---

Archived from groups: comp.dcom.lans.ethernet (More info?)

On Wed, 12 Jan 2005 15:58:21 +0100, nntp chip wrote:
> > :I have some data recorded with tcpdump that needs to be reassembled.
> > The :idea is to filter out what is relevant and then glue together just
> > the  ata-part of every packet into a file.
>
> My main reason for reassembling data is to find bad behaviour in our
> networks. Most things can be filtered and read in ethereal or ngrep. But when
> it comes to streaming media and transferred files it gets harder to tell
> when something bad is happening. That data can be any of TCP or UDP.
>
> Since I cannot do any programming myself I am searching for existing
> tools that can read a pcap-file, do all the needed magic and then write
> the payload to a file/stdout.

http://users.tpg.com.au/adsln4yb/chaosreader.html

I haven't tried it myself yet, but from the description it looks like it
might solve your problem.

Mirko

Archived from groups: comp.dcom.lans.ethernet (More info?)

In article <pan.2005.01.12.14.58.21.31286@privacy.net>,
nntp chip <me@privacy.net> wrote:

:My main reason for reassembling data is to find bad behaviour in our
:networks. Most things can be filtered and read in ethereal or ngrep. But when
:it comes to streaming media and transferred files it gets harder to tell
:when something bad is happening. That data can be any of TCP or UDP.

Urrr, what is "bad" in this context? 'bad' in a network sense
usually requires looking at the exact packets, not at the reassembled
stream, as one is looking for oddities like packets that have
an ACK without an initial SYN, or packets for which the checksum
is wrong, or packets that have ip source routing specified.

If you are trying to detect people using P2P and so on, then you
should be using an IDS such as 'snort' that wants the original
packets.

If, on the other hand, what you are looking for is matters such
as people sending trade secrets to competitors, or people
sending personal email through a company machine, then you get into
a moras of legal issues. In Canada and the USA, you can often
get around those issues by having a written security policy that
is literally signed off on by everyone... which works for -new-
employees but can be a problem if an existing employee refuses to
sign (unless you are an "at will" employer for US legal purposes;
in that case, you just tell them outright that if they don't sign they
will be fired, but if you aren't "at will" then you'll find that
refusing to agree to an impose -change- in employment conditions is not
considered adequate grounds for justifiable termination.)

In the EU, the EU privacy database policies are, I gather, interpreted
different ways in different places. About a year ago or so, someone
from, ummm Finland I think it was, posted indicating that in his
country recording the content of employee data packets was a
violation of employee privacy rights under the national implimentation
of the EU database privacy directives. I do not have information
about whether any particular national implimentation has exceptions;
the implication of the posting was that it at least could not be
-routinely- done the way it often is in the USA.
--
Preposterous!! Where would all the calculators go?!

Archived from groups: comp.dcom.lans.ethernet (More info?)

On Wed, 12 Jan 2005 15:55:48 +0000, Mirko Parthey wrote:

> On Wed, 12 Jan 2005 15:58:21 +0100, nntp chip wrote:
>> > :I have some data recorded with tcpdump that needs to be reassembled.
>> > The :idea is to filter out what is relevant and then glue together
>> > just the  ata-part of every packet into a file.
>>
>> My main reason for reassembling data is to find bad behaviour in our
>> networks. Most things can be filtered and read in ethereal or ngrep.
>> But when it comes to streaming media and transferred files it gets
>> harder to tell when something bad is happening. That data can be any
>> of TCP or UDP.
>>
>> Since I cannot do any programming myself I am searching for existing
>> tools that can read a pcap-file, do all the needed magic and then
>> write the payload to a file/stdout.
>
> http://users.tpg.com.au/adsln4yb/chaosreader.html
>
> I haven't tried it myself yet, but from the description it looks like it
> might solve your problem.


Sweet... Thank you very much.

Made some trial runs with known pcap-data that came out very nice. This
sure will be of good use around here.

---

Archived from groups: comp.dcom.lans.ethernet (More info?)

On Wed, 12 Jan 2005 20:55:00 +0000, Walter Roberson wrote:

> In article <pan.2005.01.12.14.58.21.31286@privacy.net>, nntp chip
> <me@privacy.net> wrote:
>
> :My main reason for reassembling data is to find bad behaviour in our
> :networks. Most things can be filtered and read in ethereal or ngrep.
> But when :it comes to streaming media and transferred files it gets
> harder to tell :when something bad is happening. That data can be any of
> TCP or UDP.
>
> Urrr, what is "bad" in this context? 'bad' in a network sense usually
> requires looking at the exact packets, not at the reassembled stream, as
> one is looking for oddities like packets that have an ACK without an
> initial SYN, or packets for which the checksum is wrong, or packets that
> have ip source routing specified.


That is what ethereal and similar tools are good for. When strange and
unexplained things happen I usually fire up at least one tcpdump first. If
possible more than one to be able to view problems from different angles.
Those dump-files are nice to have if things need to be analyzed in greater
detail. Should it turn out they are not needed I simply erase them again.




> If you are trying to detect people using P2P and so on, then you should
> be using an IDS such as 'snort' that wants the original packets.


I should really take the time needed to setup IDS again. The old box is no
more   Until then I watch for excessive traffic and unusual ports and
such things. It's not a big problem... yet.





> If, on the other hand, what you are looking for is matters such as
> people sending trade secrets to competitors, or people sending personal
> email through a company machine, then you get into a moras of legal
> issues. In Canada and the USA, you can often get around those issues by
> having a written security policy that is literally signed off on by
> everyone... which works for -new- employees but can be a problem if an
> existing employee refuses to sign (unless you are an "at will" employer
> for US legal purposes; in that case, you just tell them outright that if
> they don't sign they will be fired, but if you aren't "at will" then
> you'll find that refusing to agree to an impose -change- in employment
> conditions is not considered adequate grounds for justifiable
> termination.)


I check the networks of a municipality in Sweden. There are not really any
trade secrets since anyone can ask for public records. If you handle
records that are not public you must sign a contract of confidentiality. I
have done that.

The 'security policy' they have is simply telling employees that computers
and phones are for work only, but using common sense they are allowed to
call or surf for childcare, medical care, banking arrangements, car repair
shop and such things that are only open during the day. They are also told
that we can check things and will do just that if needed.

I know it would be illegal for me to sniff traffic for my own personal
use.

Hunting down people that surf pr0n, use p2p-applications or spend lots of
time in chatrooms and so on should be ok. My boss asked me to watch for
things like that, so if someone doesn't like that I guess he will have to
answer for it.

If anyone believe that I am totally wrong I would like to know what is
right and where I can read up on that.




> In the EU, the EU privacy database policies are, I gather, interpreted
> different ways in different places. About a year ago or so, someone
> from, ummm Finland I think it was, posted indicating that in his country
> recording the content of employee data packets was a violation of
> employee privacy rights under the national implimentation of the EU
> database privacy directives. I do not have information about whether any
> particular national implimentation has exceptions; the implication of
> the posting was that it at least could not be -routinely- done the way
> it often is in the USA.


Can you still get in trouble in USA/Canada for troubleshooting networks or
is it looking at transferred data that is forbidden?


---

Archived from groups: comp.dcom.lans.ethernet (More info?)

nntp chip wrote:


>> If the data does -not- consist of nice unfragmented in-order post- error
>> recovery TCP packets, then I'd suggest grabbing the Linux or FreeBSD IP
>> static and hacking it to suit your needs. No point in re-writing all the
>> retransmission logic if you don't need to.
>
>
> Just getting the raw data would sometimes be good enough.
>
> My main reason for reassembling data is to find bad behaviour in our
> networks. Most things can be filtered and read in ethereal or ngrep. But
> when it comes to streaming media and transferred files it gets harder to
> tell when something bad is happening. That data can be any of TCP or UDP.
>
> Since I cannot do any programming myself I am searching for existing
> tools that can read a pcap-file, do all the needed magic and then write
> the payload to a file/stdout.

Have you looked at what ethereal can do? It can read various dump formats,
filter the data, then export to various formats and also print to text
file, etc.