Sign in with
Sign up | Sign in
Your question
Closed

Best lossless compression for DV-AVI files?

Last response: in Graphics & Displays
Share
November 20, 2004 9:18:40 PM

Archived from groups: rec.video.desktop (More info?)

Are there lossless (or nearly so, perfection isn't required)
compressors that can scrunch DV-AVI files 2:1 or more? I'd like to
archive files that may later be converted back to DV for editing with
the likes of Premiere, so the process should be fairly undemanding of
CPU time.
Anonymous
November 21, 2004 7:41:24 AM

Archived from groups: rec.video.desktop (More info?)

"JT" <NgPoster@missing.org> wrote in message
news:6duvp0d7qvlt70eb1vdm9odr1ink9p42mi@4ax.com...
> Are there lossless (or nearly so, perfection isn't required)
> compressors that can scrunch DV-AVI files 2:1 or more? I'd like to
> archive files that may later be converted back to DV for editing with
> the likes of Premiere, so the process should be fairly undemanding of
> CPU time.
>
Sorry, but you aren't allowed to use lossless and compression
in the same sentence. DV is already pretty compressed. Even
recompression to the same codec is considered something
to avoid, if you can. If you cut that in half you will likely see
some deterioration, but if you then edit that in Premiere it
will suffer another recompression (at minimum everything that
changes (as in text overlays, color correction dissolves, etc.).

If you then put that to DVD, you will get another generate another
layer of heavy recompression. Compared to a VHS dub, it
will still look pretty good, but it will be quite a bit different from
the quality of your original DV footage.

When people talk lossless they are talking about schemes
that keep everything that changes within a frame. You save
a lot on a completely black screen with very little graphics
or text on it, but a frame with a lot of detail won't compress
very well without loosing information.

David
November 21, 2004 7:41:25 AM

Archived from groups: rec.video.desktop (More info?)

"david.mccall" <david.mccallUNDERLINE@comcast.net> wrote:

>
>"JT" <NgPoster@missing.org> wrote in message
>news:6duvp0d7qvlt70eb1vdm9odr1ink9p42mi@4ax.com...
>> Are there lossless (or nearly so, perfection isn't required)
>> compressors that can scrunch DV-AVI files 2:1 or more? I'd like to
>> archive files that may later be converted back to DV for editing with
>> the likes of Premiere, so the process should be fairly undemanding of
>> CPU time.
>>
>Sorry, but you aren't allowed to use lossless and compression
>in the same sentence. DV is already pretty compressed. Even
>recompression to the same codec is considered something
>to avoid, if you can. If you cut that in half you will likely see
>some deterioration, but if you then edit that in Premiere it
>will suffer another recompression (at minimum everything that
>changes (as in text overlays, color correction dissolves, etc.).
>
>If you then put that to DVD, you will get another generate another
>layer of heavy recompression. Compared to a VHS dub, it
>will still look pretty good, but it will be quite a bit different from
>the quality of your original DV footage.
>
>When people talk lossless they are talking about schemes
>that keep everything that changes within a frame. You save
>a lot on a completely black screen with very little graphics
>or text on it, but a frame with a lot of detail won't compress
>very well without loosing information.
>
>David
>
Lemme see, are you the guy with the perfection complex, or were you
trying to amuse? I make that mistake, too.

I assume you must know that there are dozens of lossless compression
scheme for all kinds of data, and were making a joke. Pkzip is a
simple one and for the range of avi files I sampled one day it ranged
from 10% to 50%, but unfortunately the 10% end was much too prevalant
- not enough to be worth the bother to me.

The best compression schemes are tailored to and designed with as much
knowledge as possible of the material being compressed and can almost
handily beat a general approach. RLL is a good place to begin, after
finding the best direction and/or frame size. I generally go for
deltas after that.

I've not worked with video data but I guess it's obvious that the
first two choices to investigate would be field/frame temporal, and
areal. I was/am hoping that someone suggests a piece of software that
does that rather than explain that it can't be done. It certainly can.
Related resources
Anonymous
November 21, 2004 7:41:26 AM

Archived from groups: rec.video.desktop (More info?)

"JT" wrote ...

> The best compression schemes are tailored to and designed with as much
> knowledge as possible of the material being compressed and can almost
> handily beat a general approach. RLL is a good place to begin, after
> finding the best direction and/or frame size. I generally go for
> deltas after that.
>
> I've not worked with video data but I guess it's obvious that the
> first two choices to investigate would be field/frame temporal, and
> areal. I was/am hoping that someone suggests a piece of software that
> does that rather than explain that it can't be done. It certainly can.

Of course youall realize that DV is already compressed 5:1 by that
methodology. It already creates artifacts visible to all but the most
casual TV viewer.
Anonymous
November 21, 2004 2:38:45 PM

Archived from groups: rec.video.desktop (More info?)

david.mccall wrote:


>
> Sorry, but you aren't allowed to use lossless and compression
> in the same sentence. DV is already pretty compressed. Even
> recompression to the same codec is considered something
> to avoid, if you can.

There *are* indeed situations were losless compression can be
accomplished. (e.g. RLE coding of cartoons)
However DV is not among them.
You can try and zip a DV file and probably get a few % compression. This
is because there is some repetitive (non-video) data in the dv frames
that is suitable for compression.
The compressed video will not compress further without loss of picture
quality.

However, if you decompress the DV frames and recompress them with a good
mpeg codec you will (admittedly, this is subjective) be able to get a
high rate of compression with a rather small loss in picture quality.

/Johan
Anonymous
November 21, 2004 3:48:43 PM

Archived from groups: rec.video.desktop (More info?)

On a sunny day (Sun, 21 Nov 2004 04:41:24 GMT) it happened "david.mccall"
<david.mccallUNDERLINE@comcast.net> wrote in
<U%Und.543166$mD.241587@attbi_s02>:

>
>
>
>"JT" <NgPoster@missing.org> wrote in message
>news:6duvp0d7qvlt70eb1vdm9odr1ink9p42mi@4ax.com...
>> Are there lossless (or nearly so, perfection isn't required)
>> compressors that can scrunch DV-AVI files 2:1 or more? I'd like to
>> archive files that may later be converted back to DV for editing with
>> the likes of Premiere, so the process should be fairly undemanding of
>> CPU time.
>>
>Sorry, but you aren't allowed to use lossless and compression
>in the same sentence.
Not exactly correct, for example '.zip' is compressed, AND lossless.

>DV is already pretty compressed.
True
You could for example zip a DV format file, but probably it will not make it a
lot shorter.
There are algos around that may still have some effect...
For the original poster he could encode to mpeg2 or DivX perhaps.
'slightly' lossy is vague, what you can do depends on the type of material,
say how much motion and detail, a factor 2 with DivX is no problem I think.
JP

>
Anonymous
November 21, 2004 6:36:04 PM

Archived from groups: rec.video.desktop (More info?)

On Sat, 20 Nov 2004 18:18:40 -0800, JT <NgPoster@missing.org> wrote:

>Are there lossless (or nearly so, perfection isn't required)
>compressors that can scrunch DV-AVI files 2:1 or more? I'd like to
>archive files that may later be converted back to DV for editing with
>the likes of Premiere, so the process should be fairly undemanding of
>CPU time.

This question keeps popping up again and again, but the answer is
clear . For long term archiving and backups, mini-DV tapes are the
only sensible way to go. Assuming you acquire footage on mini-DV,
your backups will be the same format. Just copy your original camera
masters from deck to deck or camera to camera (or even pc to deck
using Scenalyzer Live like I often do) The tapes cost only about
$4.50 apiece in bulk and they can hold over an hour of DV-AVI video
(13Gb+). Why try to compress to a file and lose quality and keep your
important and presumably irreplaceable footage on risky magnetic
temporary storage like a hard disk drive?
Anonymous
November 21, 2004 7:42:14 PM

Archived from groups: rec.video.desktop (More info?)

"JT" <NgPoster@missing.org> wrote in message
news:bga0q0l1ehn6givh3fces1319de2onh5u7@4ax.com...
> "david.mccall" <david.mccallUNDERLINE@comcast.net> wrote:
>
>>
>>"JT" <NgPoster@missing.org> wrote in message
>>news:6duvp0d7qvlt70eb1vdm9odr1ink9p42mi@4ax.com...
>>> Are there lossless (or nearly so, perfection isn't required)
>>> compressors that can scrunch DV-AVI files 2:1 or more? I'd like to
>>> archive files that may later be converted back to DV for editing with
>>> the likes of Premiere, so the process should be fairly undemanding of
>>> CPU time.
>>>
>>Sorry, but you aren't allowed to use lossless and compression
>>in the same sentence. DV is already pretty compressed. Even
>>recompression to the same codec is considered something
>>to avoid, if you can. If you cut that in half you will likely see
>>some deterioration, but if you then edit that in Premiere it
>>will suffer another recompression (at minimum everything that
>>changes (as in text overlays, color correction dissolves, etc.).
>>
>>If you then put that to DVD, you will get another generate another
>>layer of heavy recompression. Compared to a VHS dub, it
>>will still look pretty good, but it will be quite a bit different from
>>the quality of your original DV footage.
>>
>>When people talk lossless they are talking about schemes
>>that keep everything that changes within a frame. You save
>>a lot on a completely black screen with very little graphics
>>or text on it, but a frame with a lot of detail won't compress
>>very well without loosing information.
>>
>>David
>>
> Lemme see, are you the guy with the perfection complex, or were you
> trying to amuse? I make that mistake, too.
>
I wouldn't call it a perfectionist complex, but I am a professional
that feels a need to provide his clients with appropriate levels of
quality when I do work for them. However, not everybody needs
the very best quality possible and corners can sometimes be cut
if the needs are modest. For instance home movies, items that
will only be viewed on the web at small sizes, documentation that
needs to be kept small to fit on a CD, etc.

> I assume you must know that there are dozens of lossless compression
> scheme for all kinds of data, and were making a joke. Pkzip is a
> simple one and for the range of avi files I sampled one day it ranged
> from 10% to 50%, but unfortunately the 10% end was much too prevalant
> - not enough to be worth the bother to me.
>
Perhaps someone with better knowlege of compression will jump in,
but there is a term for that kind of compression, and in fact, I think that
type of compression is the is first step in most compression scheme.
After everything that is redundent has been removed, then the compressor
starts looking for stuff to throw ayay that you might not notice, then it
starts
in on the stuff you might be able to notice if you look close, but still
might
not care. DV is already at this level, so PK-Zip isn't going to make much
of an impact on DV footage. The only way you can further compress DV
is to throw away more important information that was removed on earlier
passes. Al you can do is expiriment to see how much loss is acceptable
to your aplication.

> The best compression schemes are tailored to and designed with as much
> knowledge as possible of the material being compressed and can almost
> handily beat a general approach. RLL is a good place to begin, after
> finding the best direction and/or frame size. I generally go for
> deltas after that.
>
But, you see, that has already been done with the first DV compression
before going to tape. Chances are RLL algorythms won't do much for
DV compressed material. You could gain a little more by doing some
delta compression, but that makes editing harder. I'm not even sure
many editing programs can deal with delta compressed video. The
problems is that you can't just jump in the middle of delta compressed
video and play or make a cut. The editing system has to backtrack to
the previous I frame and then reconstruct the video for that section
before you can scrub backwards or nake a cut. HDV compression
has to do this which is why you can just edit it in every software. I don't
know how smooth it is to edit HDV at this point, but it will get better
as processor speed goes up.

> I've not worked with video data but I guess it's obvious that the
> first two choices to investigate would be field/frame temporal, and
> areal. I was/am hoping that someone suggests a piece of software that
> does that rather than explain that it can't be done. It certainly can.
>
If only it were true. I wish you luck, but I think you will find that your
video
looks somewhat different once you throw away enough detail to achieve
2 to 1 on top of 5 to 1 compression. This level of quality may be
acceptable to your application. As I said before, it will still beat a VHS
dub,
and god knows a lot of people found that acceptable for years. Even if you
do that and output MPEG-2 (much heavier compression than DV) it still
might be good enough for your needs. However many applications would
find that much recompression unacceptable.

I did intend a touch of levity in that first statement about not being
allowed
to use lossless and compression in the same sentence, but beyond that,
I'm being serious.

Your best bet would be to dump the DV into an editor and cut out the
useless parts, then dump the remainder back to DV tape or cut it into
4 gig, or smaller, chunks and then store the data on DVDs as raw DV.

David
Anonymous
November 21, 2004 7:49:44 PM

Archived from groups: rec.video.desktop (More info?)

"Jan Panteltje" <pNaonStpealmtje@yahoo.com> wrote in message
news:1101041299.mlvxwhjZNGeXX3Qx0uttww@teranews...
>
> True
> You could for example zip a DV format file, but probably it will not make
> it a
> lot shorter.
> There are algos around that may still have some effect...
> For the original poster he could encode to mpeg2 or DivX perhaps.
> 'slightly' lossy is vague, what you can do depends on the type of
> material,
> say how much motion and detail, a factor 2 with DivX is no problem I
> think.
> JP
>
It may be acceptable for his needs, but it may not be much fun
to edit with, even if his editor can handle MPEG-2 or DivX,
and the recompression to store the result will add another hit.
It may still be acceptable to some, but to think you can do any
decent level of further compression, without loss, with a DV
source is a pipe dream.

David
Anonymous
November 22, 2004 12:20:07 AM

Archived from groups: rec.video.desktop (More info?)

On a sunny day (Sun, 21 Nov 2004 15:36:04 -0500) it happened Jim Gunn
<Jim_Gunn@Yahoo.com> wrote in <vqu1q09eg8ov9diorurrpuvk10tanre1rj@4ax.com>:
>
>This question keeps popping up again and again, but the answer is
>clear . For long term archiving and backups, mini-DV tapes are the
>only sensible way to go. Assuming you acquire footage on mini-DV,
>your backups will be the same format. Just copy your original camera
>masters from deck to deck or camera to camera (or even pc to deck
>using Scenalyzer Live like I often do) The tapes cost only about
>$4.50 apiece in bulk and they can hold over an hour of DV-AVI video
>(13Gb+). Why try to compress to a file and lose quality and keep your
>important and presumably irreplaceable footage on risky magnetic
>temporary storage like a hard disk drive?
>
My experience with tape is that it is much more unreliable then say CDR
or DVD-R.
Especially in moist and tropical oriented climates all sorts of stuff
starts growing on tapes... and floppies etc...
So, why not binary split the DV and make a couple of DVD+R?
I have hundreds full of video...
And, for Dave, regarding his remarks.. about editing. Of cause it is correct
that you can cut DV material easy, it is just a collection of jpgs right?
But for a mpeg2 or DivX encoded movie you can also cut on one frame.
There will hardly be any quality loss if you work with the right bitrates and
encoding parameters.
So it all depends on what you have to store it on.
But in no circumstance would I use tapes... I have never seen an old tape
without some sort of problem.... dropouts.
Go optical if you MUST keep the original.
JP
Anonymous
November 22, 2004 3:55:15 AM

Archived from groups: rec.video.desktop (More info?)

"Jan Panteltje" <pNaonStpealmtje@yahoo.com> wrote in message
news:1101071981.ER2YwvIq6WxshJjmHQ3zlg@teranews...
>
> My experience with tape is that it is much more unreliable then say CDR
> or DVD-R.
> Especially in moist and tropical oriented climates all sorts of stuff
> starts growing on tapes... and floppies etc...
> So, why not binary split the DV and make a couple of DVD+R?
> I have hundreds full of video...
> And, for Dave, regarding his remarks.. about editing. Of cause it is
> correct
> that you can cut DV material easy, it is just a collection of jpgs right?
> But for a mpeg2 or DivX encoded movie you can also cut on one frame.
> There will hardly be any quality loss if you work with the right bitrates
> and
> encoding parameters.
> So it all depends on what you have to store it on.
> But in no circumstance would I use tapes... I have never seen an old tape
> without some sort of problem.... dropouts.
> Go optical if you MUST keep the original.
> JP
>
You make some very good points. I have seen tape fail a lot over the years.
Unfortunately, recordable DVD media hasn't been around long enough to
know what they look like in 30 years under various conditions.

But, you are right. We know that tapes can fail. I've had
very little luck with tape backups on a computer, and I'm
not sure why DV tapes would fare better.

David
Anonymous
November 22, 2004 3:55:16 AM

Archived from groups: rec.video.desktop (More info?)

"david.mccall" wrote ...
> But, you are right. We know that tapes can fail. I've had
> very little luck with tape backups on a computer, and I'm
> not sure why DV tapes would fare better.

And yet likely way over 90% of the planet's computer data is
backed up on tape! There are whole mines filled with backup
tapes from every industrial company you've ever heard of.
Anonymous
November 22, 2004 6:44:29 AM

Archived from groups: rec.video.desktop (More info?)

"Richard Crowley" <rcrowley7@xprt.net> wrote in message
news:10q2jtlgj47kn41@corp.supernews.com...
> "david.mccall" wrote ...
>> But, you are right. We know that tapes can fail. I've had
>> very little luck with tape backups on a computer, and I'm
>> not sure why DV tapes would fare better.
>
> And yet likely way over 90% of the planet's computer data is
> backed up on tape! There are whole mines filled with backup
> tapes from every industrial company you've ever heard of.
Ain't it the truth.

I used to handle my records wit reasonable care,
but they all went "clickity popity"

I had a friend that just piled them up on the table,
occasionally knocking them on to the floor, then
drop an ash tray on the mess then play the record
and it sounded better than mine, even though his
system wasn't as expensive as mine. Go figure.
Perhaps if I had kept my records in a mine?

David
Anonymous
November 22, 2004 12:48:07 PM

Archived from groups: rec.video.desktop (More info?)

"david.mccall" <david.mccallUNDERLINE@comcast.net> wrote in message news:<TNaod.651152$8_6.356137@attbi_s04>...
> "Jan Panteltje" <pNaonStpealmtje@yahoo.com> wrote in message
> news:1101071981.ER2YwvIq6WxshJjmHQ3zlg@teranews...
> >
> > My experience with tape is that it is much more unreliable then say CDR
> > or DVD-R.
> > Especially in moist and tropical oriented climates all sorts of stuff
> > starts growing on tapes... and floppies etc...
> > So, why not binary split the DV and make a couple of DVD+R?
> > I have hundreds full of video...
> > And, for Dave, regarding his remarks.. about editing. Of cause it is
> > correct
> > that you can cut DV material easy, it is just a collection of jpgs right?
> > But for a mpeg2 or DivX encoded movie you can also cut on one frame.
> > There will hardly be any quality loss if you work with the right bitrates
> > and
> > encoding parameters.
> > So it all depends on what you have to store it on.
> > But in no circumstance would I use tapes... I have never seen an old tape
> > without some sort of problem.... dropouts.
> > Go optical if you MUST keep the original.
> > JP
> >
> You make some very good points. I have seen tape fail a lot over the years.
> Unfortunately, recordable DVD media hasn't been around long enough to
> know what they look like in 30 years under various conditions.
>
> But, you are right. We know that tapes can fail. I've had
> very little luck with tape backups on a computer, and I'm
> not sure why DV tapes would fare better.
>
> David

No offense intended, but I do find comments about 'will I be able to
get data from this media in 30 years' a little silly. Since the data
is in digital form, one can move it about from medium to medium to
one's heart's content. I would anticipate that even in 10 years,
we'll look back at DVD's and laugh at their puny capacity. In 30
years, we'll probably have holographic media that can hold collosal
amounts of data.

If I were you, I'd split your DV AVI files across DVDs and store them
carefully, perhaps multiple copies in multiple locations if the data
is that precious. Then, when the next generation of media comes
along, copy the files to this format. And let your children carry on
with this tradition, and then your childrens' children, and your
childrens' childrens' children...
Anonymous
November 22, 2004 10:07:18 PM

Archived from groups: rec.video.desktop (More info?)

"stankley" <pstankley@hotmail.com> wrote in message
news:728c4cea.0411220948.110c789f@posting.google.com...
>
> No offense intended, but I do find comments about 'will I be able to
> get data from this media in 30 years' a little silly. Since the data
> is in digital form, one can move it about from medium to medium to
> one's heart's content.
>
Only if you have a drive that will take the disk, the integrity of the
recorded
material is still near perfect, and you still have access to the original
codec.

Sure, bits are bits, but if you were to be given an 8" floppy and had
to get a file off of it, you might be in trouble. Likewise, in the analog
universe, a 1" type A analog tape would be difficult, and even the
various 1/2" reel to reel formats are getting difficult to access. Some
laser disc (videodisc) are quite noisy due to the "laser rot". I have
heard that some DVDs and CDs are starting to show signs of "laser rot"
so that could be an issue, especially for digital information. Analog media
can be pretty damaged and still work even though the image might be of
poor quality. With digital you get some error correction, but once the
damage
gets beyond what the error correction can deal with you get nothing but
garbage.

David
Anonymous
November 22, 2004 10:20:50 PM

Archived from groups: rec.video.desktop (More info?)

In article <GNqod.85151$5K2.67163@attbi_s03>,
david.mccallUNDERLINE@comcast.net says...
> Subject: Re: Best lossless compression for DV-AVI files?
> From: "david.mccall" <david.mccallUNDERLINE@comcast.net>
> Newsgroups: rec.video.desktop
>
>
> "stankley" <pstankley@hotmail.com> wrote in message
> news:728c4cea.0411220948.110c789f@posting.google.com...
> >
> > No offense intended, but I do find comments about 'will I be able to
> > get data from this media in 30 years' a little silly. Since the data
> > is in digital form, one can move it about from medium to medium to
> > one's heart's content.
> >
> Only if you have a drive that will take the disk, the integrity of the
> recorded
> material is still near perfect, and you still have access to the original
> codec.
>
> Sure, bits are bits, but if you were to be given an 8" floppy and had
> to get a file off of it, you might be in trouble. Likewise, in the analog
> universe, a 1" type A analog tape would be difficult, and even the
> various 1/2" reel to reel formats are getting difficult to access. Some
> laser disc (videodisc) are quite noisy due to the "laser rot". I have
> heard that some DVDs and CDs are starting to show signs of "laser rot"
> so that could be an issue, especially for digital information. Analog media
> can be pretty damaged and still work even though the image might be of
> poor quality. With digital you get some error correction, but once the
> damage
> gets beyond what the error correction can deal with you get nothing but
> garbage.
>
> David
>
>


Of course the key is to switch it over before the old format becomes a
footnote in the history books. I hope someday we have star trek
isolinear chips but as you know even those are vulnerable to power
surges in the computer cord caused by Romulin fire :-)
--
_________________________
Chris Phillipo - Cape Breton, Nova Scotia
http://www.ramsays-online.com
Anonymous
November 24, 2004 12:11:12 AM

Archived from groups: rec.video.desktop (More info?)

"david.mccall" <david.mccallUNDERLINE@comcast.net> wrote in message
news:xgdod.129019$HA.16330@attbi_s01...
>
> I had a friend that just piled them up on the table,
> occasionally knocking them on to the floor, then
> drop an ash tray on the mess then play the record
> and it sounded better than mine, even though his
> system wasn't as expensive as mine. Go figure.
> Perhaps if I had kept my records in a mine?
>
> David
It's those ashes that polished the scratches out of the grooves. Learned
that back in the 60's




>
>
November 20, 2010 4:40:33 PM

This topic has been closed by Mousemonkey
!