Closed

Best lossless compression for DV-AVI files?

Archived from groups: rec.video.desktop (More info?)

Are there lossless (or nearly so, perfection isn't required)
compressors that can scrunch DV-AVI files 2:1 or more? I'd like to
archive files that may later be converted back to DV for editing with
the likes of Premiere, so the process should be fairly undemanding of
CPU time.
17 answers Last reply
More about best lossless compression files
  1. Archived from groups: rec.video.desktop (More info?)

    "JT" <NgPoster@missing.org> wrote in message
    news:6duvp0d7qvlt70eb1vdm9odr1ink9p42mi@4ax.com...
    > Are there lossless (or nearly so, perfection isn't required)
    > compressors that can scrunch DV-AVI files 2:1 or more? I'd like to
    > archive files that may later be converted back to DV for editing with
    > the likes of Premiere, so the process should be fairly undemanding of
    > CPU time.
    >
    Sorry, but you aren't allowed to use lossless and compression
    in the same sentence. DV is already pretty compressed. Even
    recompression to the same codec is considered something
    to avoid, if you can. If you cut that in half you will likely see
    some deterioration, but if you then edit that in Premiere it
    will suffer another recompression (at minimum everything that
    changes (as in text overlays, color correction dissolves, etc.).

    If you then put that to DVD, you will get another generate another
    layer of heavy recompression. Compared to a VHS dub, it
    will still look pretty good, but it will be quite a bit different from
    the quality of your original DV footage.

    When people talk lossless they are talking about schemes
    that keep everything that changes within a frame. You save
    a lot on a completely black screen with very little graphics
    or text on it, but a frame with a lot of detail won't compress
    very well without loosing information.

    David
  2. Archived from groups: rec.video.desktop (More info?)

    "david.mccall" <david.mccallUNDERLINE@comcast.net> wrote:

    >
    >"JT" <NgPoster@missing.org> wrote in message
    >news:6duvp0d7qvlt70eb1vdm9odr1ink9p42mi@4ax.com...
    >> Are there lossless (or nearly so, perfection isn't required)
    >> compressors that can scrunch DV-AVI files 2:1 or more? I'd like to
    >> archive files that may later be converted back to DV for editing with
    >> the likes of Premiere, so the process should be fairly undemanding of
    >> CPU time.
    >>
    >Sorry, but you aren't allowed to use lossless and compression
    >in the same sentence. DV is already pretty compressed. Even
    >recompression to the same codec is considered something
    >to avoid, if you can. If you cut that in half you will likely see
    >some deterioration, but if you then edit that in Premiere it
    >will suffer another recompression (at minimum everything that
    >changes (as in text overlays, color correction dissolves, etc.).
    >
    >If you then put that to DVD, you will get another generate another
    >layer of heavy recompression. Compared to a VHS dub, it
    >will still look pretty good, but it will be quite a bit different from
    >the quality of your original DV footage.
    >
    >When people talk lossless they are talking about schemes
    >that keep everything that changes within a frame. You save
    >a lot on a completely black screen with very little graphics
    >or text on it, but a frame with a lot of detail won't compress
    >very well without loosing information.
    >
    >David
    >
    Lemme see, are you the guy with the perfection complex, or were you
    trying to amuse? I make that mistake, too.

    I assume you must know that there are dozens of lossless compression
    scheme for all kinds of data, and were making a joke. Pkzip is a
    simple one and for the range of avi files I sampled one day it ranged
    from 10% to 50%, but unfortunately the 10% end was much too prevalant
    - not enough to be worth the bother to me.

    The best compression schemes are tailored to and designed with as much
    knowledge as possible of the material being compressed and can almost
    handily beat a general approach. RLL is a good place to begin, after
    finding the best direction and/or frame size. I generally go for
    deltas after that.

    I've not worked with video data but I guess it's obvious that the
    first two choices to investigate would be field/frame temporal, and
    areal. I was/am hoping that someone suggests a piece of software that
    does that rather than explain that it can't be done. It certainly can.
  3. Archived from groups: rec.video.desktop (More info?)

    "JT" wrote ...

    > The best compression schemes are tailored to and designed with as much
    > knowledge as possible of the material being compressed and can almost
    > handily beat a general approach. RLL is a good place to begin, after
    > finding the best direction and/or frame size. I generally go for
    > deltas after that.
    >
    > I've not worked with video data but I guess it's obvious that the
    > first two choices to investigate would be field/frame temporal, and
    > areal. I was/am hoping that someone suggests a piece of software that
    > does that rather than explain that it can't be done. It certainly can.

    Of course youall realize that DV is already compressed 5:1 by that
    methodology. It already creates artifacts visible to all but the most
    casual TV viewer.
  4. Archived from groups: rec.video.desktop (More info?)

    david.mccall wrote:


    >
    > Sorry, but you aren't allowed to use lossless and compression
    > in the same sentence. DV is already pretty compressed. Even
    > recompression to the same codec is considered something
    > to avoid, if you can.

    There *are* indeed situations were losless compression can be
    accomplished. (e.g. RLE coding of cartoons)
    However DV is not among them.
    You can try and zip a DV file and probably get a few % compression. This
    is because there is some repetitive (non-video) data in the dv frames
    that is suitable for compression.
    The compressed video will not compress further without loss of picture
    quality.

    However, if you decompress the DV frames and recompress them with a good
    mpeg codec you will (admittedly, this is subjective) be able to get a
    high rate of compression with a rather small loss in picture quality.

    /Johan
  5. Archived from groups: rec.video.desktop (More info?)

    On a sunny day (Sun, 21 Nov 2004 04:41:24 GMT) it happened "david.mccall"
    <david.mccallUNDERLINE@comcast.net> wrote in
    <U%Und.543166$mD.241587@attbi_s02>:

    >
    >
    >
    >"JT" <NgPoster@missing.org> wrote in message
    >news:6duvp0d7qvlt70eb1vdm9odr1ink9p42mi@4ax.com...
    >> Are there lossless (or nearly so, perfection isn't required)
    >> compressors that can scrunch DV-AVI files 2:1 or more? I'd like to
    >> archive files that may later be converted back to DV for editing with
    >> the likes of Premiere, so the process should be fairly undemanding of
    >> CPU time.
    >>
    >Sorry, but you aren't allowed to use lossless and compression
    >in the same sentence.
    Not exactly correct, for example '.zip' is compressed, AND lossless.

    >DV is already pretty compressed.
    True
    You could for example zip a DV format file, but probably it will not make it a
    lot shorter.
    There are algos around that may still have some effect...
    For the original poster he could encode to mpeg2 or DivX perhaps.
    'slightly' lossy is vague, what you can do depends on the type of material,
    say how much motion and detail, a factor 2 with DivX is no problem I think.
    JP

    >
  6. Archived from groups: rec.video.desktop (More info?)

    On Sat, 20 Nov 2004 18:18:40 -0800, JT <NgPoster@missing.org> wrote:

    >Are there lossless (or nearly so, perfection isn't required)
    >compressors that can scrunch DV-AVI files 2:1 or more? I'd like to
    >archive files that may later be converted back to DV for editing with
    >the likes of Premiere, so the process should be fairly undemanding of
    >CPU time.

    This question keeps popping up again and again, but the answer is
    clear . For long term archiving and backups, mini-DV tapes are the
    only sensible way to go. Assuming you acquire footage on mini-DV,
    your backups will be the same format. Just copy your original camera
    masters from deck to deck or camera to camera (or even pc to deck
    using Scenalyzer Live like I often do) The tapes cost only about
    $4.50 apiece in bulk and they can hold over an hour of DV-AVI video
    (13Gb+). Why try to compress to a file and lose quality and keep your
    important and presumably irreplaceable footage on risky magnetic
    temporary storage like a hard disk drive?
  7. Archived from groups: rec.video.desktop (More info?)

    "JT" <NgPoster@missing.org> wrote in message
    news:bga0q0l1ehn6givh3fces1319de2onh5u7@4ax.com...
    > "david.mccall" <david.mccallUNDERLINE@comcast.net> wrote:
    >
    >>
    >>"JT" <NgPoster@missing.org> wrote in message
    >>news:6duvp0d7qvlt70eb1vdm9odr1ink9p42mi@4ax.com...
    >>> Are there lossless (or nearly so, perfection isn't required)
    >>> compressors that can scrunch DV-AVI files 2:1 or more? I'd like to
    >>> archive files that may later be converted back to DV for editing with
    >>> the likes of Premiere, so the process should be fairly undemanding of
    >>> CPU time.
    >>>
    >>Sorry, but you aren't allowed to use lossless and compression
    >>in the same sentence. DV is already pretty compressed. Even
    >>recompression to the same codec is considered something
    >>to avoid, if you can. If you cut that in half you will likely see
    >>some deterioration, but if you then edit that in Premiere it
    >>will suffer another recompression (at minimum everything that
    >>changes (as in text overlays, color correction dissolves, etc.).
    >>
    >>If you then put that to DVD, you will get another generate another
    >>layer of heavy recompression. Compared to a VHS dub, it
    >>will still look pretty good, but it will be quite a bit different from
    >>the quality of your original DV footage.
    >>
    >>When people talk lossless they are talking about schemes
    >>that keep everything that changes within a frame. You save
    >>a lot on a completely black screen with very little graphics
    >>or text on it, but a frame with a lot of detail won't compress
    >>very well without loosing information.
    >>
    >>David
    >>
    > Lemme see, are you the guy with the perfection complex, or were you
    > trying to amuse? I make that mistake, too.
    >
    I wouldn't call it a perfectionist complex, but I am a professional
    that feels a need to provide his clients with appropriate levels of
    quality when I do work for them. However, not everybody needs
    the very best quality possible and corners can sometimes be cut
    if the needs are modest. For instance home movies, items that
    will only be viewed on the web at small sizes, documentation that
    needs to be kept small to fit on a CD, etc.

    > I assume you must know that there are dozens of lossless compression
    > scheme for all kinds of data, and were making a joke. Pkzip is a
    > simple one and for the range of avi files I sampled one day it ranged
    > from 10% to 50%, but unfortunately the 10% end was much too prevalant
    > - not enough to be worth the bother to me.
    >
    Perhaps someone with better knowlege of compression will jump in,
    but there is a term for that kind of compression, and in fact, I think that
    type of compression is the is first step in most compression scheme.
    After everything that is redundent has been removed, then the compressor
    starts looking for stuff to throw ayay that you might not notice, then it
    starts
    in on the stuff you might be able to notice if you look close, but still
    might
    not care. DV is already at this level, so PK-Zip isn't going to make much
    of an impact on DV footage. The only way you can further compress DV
    is to throw away more important information that was removed on earlier
    passes. Al you can do is expiriment to see how much loss is acceptable
    to your aplication.

    > The best compression schemes are tailored to and designed with as much
    > knowledge as possible of the material being compressed and can almost
    > handily beat a general approach. RLL is a good place to begin, after
    > finding the best direction and/or frame size. I generally go for
    > deltas after that.
    >
    But, you see, that has already been done with the first DV compression
    before going to tape. Chances are RLL algorythms won't do much for
    DV compressed material. You could gain a little more by doing some
    delta compression, but that makes editing harder. I'm not even sure
    many editing programs can deal with delta compressed video. The
    problems is that you can't just jump in the middle of delta compressed
    video and play or make a cut. The editing system has to backtrack to
    the previous I frame and then reconstruct the video for that section
    before you can scrub backwards or nake a cut. HDV compression
    has to do this which is why you can just edit it in every software. I don't
    know how smooth it is to edit HDV at this point, but it will get better
    as processor speed goes up.

    > I've not worked with video data but I guess it's obvious that the
    > first two choices to investigate would be field/frame temporal, and
    > areal. I was/am hoping that someone suggests a piece of software that
    > does that rather than explain that it can't be done. It certainly can.
    >
    If only it were true. I wish you luck, but I think you will find that your
    video
    looks somewhat different once you throw away enough detail to achieve
    2 to 1 on top of 5 to 1 compression. This level of quality may be
    acceptable to your application. As I said before, it will still beat a VHS
    dub,
    and god knows a lot of people found that acceptable for years. Even if you
    do that and output MPEG-2 (much heavier compression than DV) it still
    might be good enough for your needs. However many applications would
    find that much recompression unacceptable.

    I did intend a touch of levity in that first statement about not being
    allowed
    to use lossless and compression in the same sentence, but beyond that,
    I'm being serious.

    Your best bet would be to dump the DV into an editor and cut out the
    useless parts, then dump the remainder back to DV tape or cut it into
    4 gig, or smaller, chunks and then store the data on DVDs as raw DV.

    David
  8. Archived from groups: rec.video.desktop (More info?)

    "Jan Panteltje" <pNaonStpealmtje@yahoo.com> wrote in message
    news:1101041299.mlvxwhjZNGeXX3Qx0uttww@teranews...
    >
    > True
    > You could for example zip a DV format file, but probably it will not make
    > it a
    > lot shorter.
    > There are algos around that may still have some effect...
    > For the original poster he could encode to mpeg2 or DivX perhaps.
    > 'slightly' lossy is vague, what you can do depends on the type of
    > material,
    > say how much motion and detail, a factor 2 with DivX is no problem I
    > think.
    > JP
    >
    It may be acceptable for his needs, but it may not be much fun
    to edit with, even if his editor can handle MPEG-2 or DivX,
    and the recompression to store the result will add another hit.
    It may still be acceptable to some, but to think you can do any
    decent level of further compression, without loss, with a DV
    source is a pipe dream.

    David
  9. Archived from groups: rec.video.desktop (More info?)

    On a sunny day (Sun, 21 Nov 2004 15:36:04 -0500) it happened Jim Gunn
    <Jim_Gunn@Yahoo.com> wrote in <vqu1q09eg8ov9diorurrpuvk10tanre1rj@4ax.com>:
    >
    >This question keeps popping up again and again, but the answer is
    >clear . For long term archiving and backups, mini-DV tapes are the
    >only sensible way to go. Assuming you acquire footage on mini-DV,
    >your backups will be the same format. Just copy your original camera
    >masters from deck to deck or camera to camera (or even pc to deck
    >using Scenalyzer Live like I often do) The tapes cost only about
    >$4.50 apiece in bulk and they can hold over an hour of DV-AVI video
    >(13Gb+). Why try to compress to a file and lose quality and keep your
    >important and presumably irreplaceable footage on risky magnetic
    >temporary storage like a hard disk drive?
    >
    My experience with tape is that it is much more unreliable then say CDR
    or DVD-R.
    Especially in moist and tropical oriented climates all sorts of stuff
    starts growing on tapes... and floppies etc...
    So, why not binary split the DV and make a couple of DVD+R?
    I have hundreds full of video...
    And, for Dave, regarding his remarks.. about editing. Of cause it is correct
    that you can cut DV material easy, it is just a collection of jpgs right?
    But for a mpeg2 or DivX encoded movie you can also cut on one frame.
    There will hardly be any quality loss if you work with the right bitrates and
    encoding parameters.
    So it all depends on what you have to store it on.
    But in no circumstance would I use tapes... I have never seen an old tape
    without some sort of problem.... dropouts.
    Go optical if you MUST keep the original.
    JP
  10. Archived from groups: rec.video.desktop (More info?)

    "Jan Panteltje" <pNaonStpealmtje@yahoo.com> wrote in message
    news:1101071981.ER2YwvIq6WxshJjmHQ3zlg@teranews...
    >
    > My experience with tape is that it is much more unreliable then say CDR
    > or DVD-R.
    > Especially in moist and tropical oriented climates all sorts of stuff
    > starts growing on tapes... and floppies etc...
    > So, why not binary split the DV and make a couple of DVD+R?
    > I have hundreds full of video...
    > And, for Dave, regarding his remarks.. about editing. Of cause it is
    > correct
    > that you can cut DV material easy, it is just a collection of jpgs right?
    > But for a mpeg2 or DivX encoded movie you can also cut on one frame.
    > There will hardly be any quality loss if you work with the right bitrates
    > and
    > encoding parameters.
    > So it all depends on what you have to store it on.
    > But in no circumstance would I use tapes... I have never seen an old tape
    > without some sort of problem.... dropouts.
    > Go optical if you MUST keep the original.
    > JP
    >
    You make some very good points. I have seen tape fail a lot over the years.
    Unfortunately, recordable DVD media hasn't been around long enough to
    know what they look like in 30 years under various conditions.

    But, you are right. We know that tapes can fail. I've had
    very little luck with tape backups on a computer, and I'm
    not sure why DV tapes would fare better.

    David
  11. Archived from groups: rec.video.desktop (More info?)

    "david.mccall" wrote ...
    > But, you are right. We know that tapes can fail. I've had
    > very little luck with tape backups on a computer, and I'm
    > not sure why DV tapes would fare better.

    And yet likely way over 90% of the planet's computer data is
    backed up on tape! There are whole mines filled with backup
    tapes from every industrial company you've ever heard of.
  12. Archived from groups: rec.video.desktop (More info?)

    "Richard Crowley" <rcrowley7@xprt.net> wrote in message
    news:10q2jtlgj47kn41@corp.supernews.com...
    > "david.mccall" wrote ...
    >> But, you are right. We know that tapes can fail. I've had
    >> very little luck with tape backups on a computer, and I'm
    >> not sure why DV tapes would fare better.
    >
    > And yet likely way over 90% of the planet's computer data is
    > backed up on tape! There are whole mines filled with backup
    > tapes from every industrial company you've ever heard of.
    Ain't it the truth.

    I used to handle my records wit reasonable care,
    but they all went "clickity popity"

    I had a friend that just piled them up on the table,
    occasionally knocking them on to the floor, then
    drop an ash tray on the mess then play the record
    and it sounded better than mine, even though his
    system wasn't as expensive as mine. Go figure.
    Perhaps if I had kept my records in a mine?

    David
  13. Archived from groups: rec.video.desktop (More info?)

    "david.mccall" <david.mccallUNDERLINE@comcast.net> wrote in message news:<TNaod.651152$8_6.356137@attbi_s04>...
    > "Jan Panteltje" <pNaonStpealmtje@yahoo.com> wrote in message
    > news:1101071981.ER2YwvIq6WxshJjmHQ3zlg@teranews...
    > >
    > > My experience with tape is that it is much more unreliable then say CDR
    > > or DVD-R.
    > > Especially in moist and tropical oriented climates all sorts of stuff
    > > starts growing on tapes... and floppies etc...
    > > So, why not binary split the DV and make a couple of DVD+R?
    > > I have hundreds full of video...
    > > And, for Dave, regarding his remarks.. about editing. Of cause it is
    > > correct
    > > that you can cut DV material easy, it is just a collection of jpgs right?
    > > But for a mpeg2 or DivX encoded movie you can also cut on one frame.
    > > There will hardly be any quality loss if you work with the right bitrates
    > > and
    > > encoding parameters.
    > > So it all depends on what you have to store it on.
    > > But in no circumstance would I use tapes... I have never seen an old tape
    > > without some sort of problem.... dropouts.
    > > Go optical if you MUST keep the original.
    > > JP
    > >
    > You make some very good points. I have seen tape fail a lot over the years.
    > Unfortunately, recordable DVD media hasn't been around long enough to
    > know what they look like in 30 years under various conditions.
    >
    > But, you are right. We know that tapes can fail. I've had
    > very little luck with tape backups on a computer, and I'm
    > not sure why DV tapes would fare better.
    >
    > David

    No offense intended, but I do find comments about 'will I be able to
    get data from this media in 30 years' a little silly. Since the data
    is in digital form, one can move it about from medium to medium to
    one's heart's content. I would anticipate that even in 10 years,
    we'll look back at DVD's and laugh at their puny capacity. In 30
    years, we'll probably have holographic media that can hold collosal
    amounts of data.

    If I were you, I'd split your DV AVI files across DVDs and store them
    carefully, perhaps multiple copies in multiple locations if the data
    is that precious. Then, when the next generation of media comes
    along, copy the files to this format. And let your children carry on
    with this tradition, and then your childrens' children, and your
    childrens' childrens' children...
  14. Archived from groups: rec.video.desktop (More info?)

    "stankley" <pstankley@hotmail.com> wrote in message
    news:728c4cea.0411220948.110c789f@posting.google.com...
    >
    > No offense intended, but I do find comments about 'will I be able to
    > get data from this media in 30 years' a little silly. Since the data
    > is in digital form, one can move it about from medium to medium to
    > one's heart's content.
    >
    Only if you have a drive that will take the disk, the integrity of the
    recorded
    material is still near perfect, and you still have access to the original
    codec.

    Sure, bits are bits, but if you were to be given an 8" floppy and had
    to get a file off of it, you might be in trouble. Likewise, in the analog
    universe, a 1" type A analog tape would be difficult, and even the
    various 1/2" reel to reel formats are getting difficult to access. Some
    laser disc (videodisc) are quite noisy due to the "laser rot". I have
    heard that some DVDs and CDs are starting to show signs of "laser rot"
    so that could be an issue, especially for digital information. Analog media
    can be pretty damaged and still work even though the image might be of
    poor quality. With digital you get some error correction, but once the
    damage
    gets beyond what the error correction can deal with you get nothing but
    garbage.

    David
  15. Archived from groups: rec.video.desktop (More info?)

    In article <GNqod.85151$5K2.67163@attbi_s03>,
    david.mccallUNDERLINE@comcast.net says...
    > Subject: Re: Best lossless compression for DV-AVI files?
    > From: "david.mccall" <david.mccallUNDERLINE@comcast.net>
    > Newsgroups: rec.video.desktop
    >
    >
    > "stankley" <pstankley@hotmail.com> wrote in message
    > news:728c4cea.0411220948.110c789f@posting.google.com...
    > >
    > > No offense intended, but I do find comments about 'will I be able to
    > > get data from this media in 30 years' a little silly. Since the data
    > > is in digital form, one can move it about from medium to medium to
    > > one's heart's content.
    > >
    > Only if you have a drive that will take the disk, the integrity of the
    > recorded
    > material is still near perfect, and you still have access to the original
    > codec.
    >
    > Sure, bits are bits, but if you were to be given an 8" floppy and had
    > to get a file off of it, you might be in trouble. Likewise, in the analog
    > universe, a 1" type A analog tape would be difficult, and even the
    > various 1/2" reel to reel formats are getting difficult to access. Some
    > laser disc (videodisc) are quite noisy due to the "laser rot". I have
    > heard that some DVDs and CDs are starting to show signs of "laser rot"
    > so that could be an issue, especially for digital information. Analog media
    > can be pretty damaged and still work even though the image might be of
    > poor quality. With digital you get some error correction, but once the
    > damage
    > gets beyond what the error correction can deal with you get nothing but
    > garbage.
    >
    > David
    >
    >


    Of course the key is to switch it over before the old format becomes a
    footnote in the history books. I hope someday we have star trek
    isolinear chips but as you know even those are vulnerable to power
    surges in the computer cord caused by Romulin fire :-)
    --
    _________________________
    Chris Phillipo - Cape Breton, Nova Scotia
    http://www.ramsays-online.com
  16. Archived from groups: rec.video.desktop (More info?)

    "david.mccall" <david.mccallUNDERLINE@comcast.net> wrote in message
    news:xgdod.129019$HA.16330@attbi_s01...
    >
    > I had a friend that just piled them up on the table,
    > occasionally knocking them on to the floor, then
    > drop an ash tray on the mess then play the record
    > and it sounded better than mine, even though his
    > system wasn't as expensive as mine. Go figure.
    > Perhaps if I had kept my records in a mine?
    >
    > David
    It's those ashes that polished the scratches out of the grooves. Learned
    that back in the 60's


    >
    >
  17. This topic has been closed by Mousemonkey
Ask a new question

Read More

Tuner Cards AVI Lossless Compression Graphics