Sign in with
Sign up | Sign in
Your question

Pentax RAW file size

Last response: in Digital Camera
Share
Anonymous
January 5, 2005 8:18:46 PM

Archived from groups: rec.photo.digital (More info?)

Hey all... getting closer to purchasing a *istDS, but I'm a bit troubled by a
design choice of Pentaxs'. It would appear that the RAW file size is *HUGE* by
comparison to other brands. I've heard rumors that they both pad the 12-bits off the
A/D to 16 bits, AND don't do compression. It would seem like a silly thing to not do
a lossless compression on the RAW files unless computational power was an issue. Can
anyone confirm or deny such a thing? I've been scouring the 'net looking for a sample
RAW .PEF file to look through, but haven't had any luck with that either.

If it is too difficult to compress on-the-fly at the time of the shot, at
least a "pack photos" option or something would be a very good upgrade to the
firmware. One could let the camera chew on the already taken pictures and crunch them
down to 50% of their original size to fit 100 pics on a 512MB card, rather than 45.

Anyone care to comment?

-Cory

--

*************************************************************************
* Cory Papenfuss *
* Electrical Engineering candidate Ph.D. graduate student *
* Virginia Polytechnic Institute and State University *
*************************************************************************

More about : pentax raw file size

Anonymous
January 5, 2005 9:56:29 PM

Archived from groups: rec.photo.digital (More info?)

John Bean <waterfoot@gmail.com> wrote:
: The DS file is about 10MB, about 3MB smaller than the D.

: Both include full resolution JPEGs within them along with VGA and thumbnail
: sizes. Plus the sensor dump of course.

They've got *full-resolution* JPEGs in them? That's pretty significant
then.... seems like that should be an option, not enabled all the time. Hrm... no
accounting for taste, I guess. :) 

-Cory

--

*************************************************************************
* Cory Papenfuss *
* Electrical Engineering candidate Ph.D. graduate student *
* Virginia Polytechnic Institute and State University *
*************************************************************************
Anonymous
January 5, 2005 10:17:56 PM

Archived from groups: rec.photo.digital (More info?)

On Wed, 5 Jan 2005 18:56:29 +0000 (UTC),
papenfussDIESPAM@juneauDOTmeDOTvt.edu wrote:

> John Bean <waterfoot@gmail.com> wrote:
>: The DS file is about 10MB, about 3MB smaller than the D.
>
>: Both include full resolution JPEGs within them along with VGA and thumbnail
>: sizes. Plus the sensor dump of course.
>
> They've got *full-resolution* JPEGs in them? That's pretty significant
> then.... seems like that should be an option, not enabled all the time. Hrm... no
> accounting for taste, I guess. :) 

Nothing to do with taste, it's a good design feature. It's used for quick
high-resolution reviewing and scrolling of the image on camera without
having to reprocess the raw data every time.

--
John Bean

Great spirits often meet violent opposition with mediocre minds (Albert
Einstein)
Related resources
Anonymous
January 5, 2005 11:33:31 PM

Archived from groups: rec.photo.digital (More info?)

John Bean <waterfoot@gmail.com> wrote:
: Nothing to do with taste, it's a good design feature. It's used for quick
: high-resolution reviewing and scrolling of the image on camera without
: having to reprocess the raw data every time.

I don't disagree that's it's a nice feature to have, I just don't agree that
it should be mandatory. I managed to find a couple of sample PEF files online and did
a little hexdumping. I found the three sizes within the PEF file of size 10000kB

160x120: 8kB
640x480: 40kB
3008x2000: 912kB

That's 10% worth of overhead on every shot. It's a fairly compressed JPEG too, so
it's probably not the same one you'd get if you shot in JPEG. I think it'd be better
to have an option to include big JPEG or not and have it be the same high-quality one
as if shot JPEG.

--

*************************************************************************
* Cory Papenfuss *
* Electrical Engineering candidate Ph.D. graduate student *
* Virginia Polytechnic Institute and State University *
*************************************************************************
Anonymous
January 6, 2005 1:26:14 AM

Archived from groups: rec.photo.digital (More info?)

On Wed, 5 Jan 2005 20:33:31 +0000 (UTC),
papenfussDIESPAM@juneauDOTmeDOTvt.edu wrote:

> John Bean <waterfoot@gmail.com> wrote:
>: Nothing to do with taste, it's a good design feature. It's used for quick
>: high-resolution reviewing and scrolling of the image on camera without
>: having to reprocess the raw data every time.
>
> I don't disagree that's it's a nice feature to have, I just don't agree that
> it should be mandatory. I managed to find a couple of sample PEF files online and did
> a little hexdumping. I found the three sizes within the PEF file of size 10000kB
>
> 160x120: 8kB
> 640x480: 40kB
> 3008x2000: 912kB

Yes, that's what I told you initially though I didn't bother quoting the
number chapter and verse, but thank you for confirming my facts.

If you don't like the way it works tell Pentax, not me. Personally, I like
it the way it is. Memory is cheap, convenience is worth more.

--
John Bean

I think computer viruses should count as life. I think it says something
about human nature that the only form of life we have created so far is
purely destructive. We've created life in our own image (Stephen Hawking)
January 6, 2005 1:26:15 AM

Archived from groups: rec.photo.digital (More info?)

"John Bean" <waterfoot@gmail.com> wrote in message
news:1uogk66dlf42i$.dlg@waterfoot.net...
> On Wed, 5 Jan 2005 20:33:31 +0000 (UTC),
> papenfussDIESPAM@juneauDOTmeDOTvt.edu wrote:
>
> > John Bean <waterfoot@gmail.com> wrote:
> >: Nothing to do with taste, it's a good design feature. It's used for
quick
> >: high-resolution reviewing and scrolling of the image on camera without
> >: having to reprocess the raw data every time.
> >
> > I don't disagree that's it's a nice feature to have, I just don't agree
that
> > it should be mandatory. I managed to find a couple of sample PEF files
online and did
> > a little hexdumping. I found the three sizes within the PEF file of
size 10000kB
> >
> > 160x120: 8kB
> > 640x480: 40kB
> > 3008x2000: 912kB
>
> Yes, that's what I told you initially though I didn't bother quoting the
> number chapter and verse, but thank you for confirming my facts.
>
> If you don't like the way it works tell Pentax, not me. Personally, I like
> it the way it is. Memory is cheap, convenience is worth more.
>
Maybe disabling the embedded jpeg will be in the next firmware, or perhaps
in the next Pentax dSLR.
Anonymous
January 6, 2005 1:59:19 AM

Archived from groups: rec.photo.digital (More info?)

John Bean <waterfoot@gmail.com> wrote:
: Yes, that's what I told you initially though I didn't bother quoting the
: number chapter and verse, but thank you for confirming my facts.

I was just excited that I was able to hack through the file enough to extract
the stuff. Also, I wanted to know how big the JPEGs were. As you know, they could be
compressed to 100kB, or take 5MB... depends on the compression setting. Just saying
the resolution doesn't nail down the bytes consumed.

: If you don't like the way it works tell Pentax, not me. Personally, I like
: it the way it is. Memory is cheap, convenience is worth more.

I plan to suggest it to Pentax. I doubt I'll get anywhere, but it's worth a
shot. Memory is (relatively) cheap, but I'd say the inconvenience of having to swap
cards 2x as many times than necessary is annoying if you don't plan to use the
full-size JPEGs. Again... personal preference, which is why I submit it should be an
*option*... :) 

Also in my musings through the dcraw source and some utils on linux, I
discovered that the .PEF format appears to be a perverted TIFF file with custom tag
numbers.

*************************************************************************
* Cory Papenfuss *
* Electrical Engineering candidate Ph.D. graduate student *
* Virginia Polytechnic Institute and State University *
*************************************************************************
Anonymous
January 6, 2005 2:15:47 AM

Archived from groups: rec.photo.digital (More info?)

On Wed, 5 Jan 2005 22:59:19 +0000 (UTC),
papenfussDIESPAM@juneauDOTmeDOTvt.edu wrote:

> I was just excited that I was able to hack through the file enough to extract
> the stuff. Also, I wanted to know how big the JPEGs were. As you know, they could be
> compressed to 100kB, or take 5MB... depends on the compression setting. Just saying
> the resolution doesn't nail down the bytes consumed.

Ah, youthful exuberance :-)

Yes, the big JPEG is very compressed. It only really needed for checking
fine detail (which is not lost of course) for focus accuracy and checking
for blown highlights. You never see it "full size" unless you extract it
later, and it's pretty rough.

> Also in my musings through the dcraw source and some utils on linux, I
> discovered that the .PEF format appears to be a perverted TIFF file with custom tag
> numbers.

Yes, near enough for even old utilities that can parse a TIFF to extract
TIFF-like EXIF information. Not as TIFF-like as Nikon's PEF format, which
even Windows Explorer can extract the thumbnail from if you tell it it's a
TIFF.

--
John Bean

Where a calculator on the ENIAC is equpped with 18,000 vacuum tubes and
weighs 30 tons, computers in the future may have only 1,000 vaccuum tubes
and perhaps weigh 1.5 tons (Popular Mechanics, March 1949)
Anonymous
January 6, 2005 2:43:43 AM

Archived from groups: rec.photo.digital (More info?)

John Bean <waterfoot@gmail.com> wrote:
: Ah, youthful exuberance :-)

: Yes, the big JPEG is very compressed. It only really needed for checking
: fine detail (which is not lost of course) for focus accuracy and checking
: for blown highlights. You never see it "full size" unless you extract it
: later, and it's pretty rough.

... So it still begs the question why have such a large, high-res "thumbnail"
taking up space that's not good enough to keep as JPEG-from-camera?


: Yes, near enough for even old utilities that can parse a TIFF to extract
: TIFF-like EXIF information. Not as TIFF-like as Nikon's PEF format, which
: even Windows Explorer can extract the thumbnail from if you tell it it's a
: TIFF.

Even without the JPEGs in there, however, the RAW data seems to take up more
room than on most. For a 4mp Cannon, the *entire* RAW file was 3-3.5 MB. For 6mp
with the same compression, it should be 4.5-5.5 MB... not 9. The TIFF extracted via
tiffsplit indicates:
tiffinfo xaa.tif
TIFF Directory at offset 0x8cd488
Image Width: 3040 Image Length: 2024
Resolution: 72, 72 pixels/inch
Bits/Sample: 12
Compression Scheme: PackBits
Photometric Interpretation: 32803 (0x8023)
Date & Time: "2004:11:29 08:38:25"
Software: "*ist DS Ver 1.00 "
Make: "PENTAX Corporation "
Model: "PENTAX *ist DS "
Orientation: row 0 top, col 0 lhs
Samples/Pixel: 1
Rows/Strip: 2024
Planar Configuration: single image plane

I don't know if the PackBits compression scheme is correct, but it certainly doesn't
seem to compress much of anything. A Bayer grid that's 3040x2024 is 6.15mp. At 12
bits, that's 9.0 MB.... the size of the extracted RAW data that's not JPEG.

-Cory

*************************************************************************
* Cory Papenfuss *
* Electrical Engineering candidate Ph.D. graduate student *
* Virginia Polytechnic Institute and State University *
*************************************************************************
Anonymous
January 6, 2005 2:51:33 AM

Archived from groups: rec.photo.digital (More info?)

On Wed, 5 Jan 2005 23:43:43 +0000 (UTC),
papenfussDIESPAM@juneauDOTmeDOTvt.edu wrote:

> Even without the JPEGs in there, however, the RAW data seems to take up more
> room than on most. For a 4mp Cannon, the *entire* RAW file was 3-3.5 MB. For 6mp
> with the same compression, it should be 4.5-5.5 MB... not 9. The TIFF extracted via
> tiffsplit indicates:
> tiffinfo xaa.tif
> TIFF Directory at offset 0x8cd488
> Image Width: 3040 Image Length: 2024
> Resolution: 72, 72 pixels/inch
> Bits/Sample: 12
> Compression Scheme: PackBits
> Photometric Interpretation: 32803 (0x8023)
> Date & Time: "2004:11:29 08:38:25"
> Software: "*ist DS Ver 1.00 "
> Make: "PENTAX Corporation "
> Model: "PENTAX *ist DS "
> Orientation: row 0 top, col 0 lhs
> Samples/Pixel: 1
> Rows/Strip: 2024
> Planar Configuration: single image plane
>
> I don't know if the PackBits compression scheme is correct, but it certainly doesn't
> seem to compress much of anything. A Bayer grid that's 3040x2024 is 6.15mp. At 12
> bits, that's 9.0 MB.... the size of the extracted RAW data that's not JPEG.

PackBits does what it says. On the D the 12-bit data was simply stored in a
16-bit TIFF so it occupied 12M. The DS packs the data but neither compress
it, hence the 9M data size of the DS. Round up to 13M and 10M respectively
including the JPEGs and red-tape stuff.

--
John Bean

What is a cynic? A man who knows the price of everything and the value of
nothing (Oscar Wilde)
Anonymous
January 6, 2005 3:31:51 AM

Archived from groups: rec.photo.digital (More info?)

John Bean <waterfoot@gmail.com> wrote:
: PackBits does what it says. On the D the 12-bit data was simply stored in a
: 16-bit TIFF so it occupied 12M. The DS packs the data but neither compress
: it, hence the 9M data size of the DS. Round up to 13M and 10M respectively
: including the JPEGs and red-tape stuff.

That's what I thought initially. Within the context of TIFF files, PackBits
compression is supposed to be an RLE encoding/compression... not just removing zero
padding. The file size doesn't agree with that, however, and seems to imply no
compresison at all.

For something as noisy as raw sensor data, PackBits would be a lousy
compression scheme. I wouldn't doubt it if the Pentax engineers abused the TIFF
format further by calling it PackBits when it's really just uncompressed non-padded
12-bit data.

-Cory

--

*************************************************************************
* Cory Papenfuss *
* Electrical Engineering candidate Ph.D. graduate student *
* Virginia Polytechnic Institute and State University *
*************************************************************************
Anonymous
January 6, 2005 12:28:36 PM

Archived from groups: rec.photo.digital (More info?)

On Thu, 6 Jan 2005 00:31:51 +0000 (UTC),
papenfussDIESPAM@juneauDOTmeDOTvt.edu wrote:

> John Bean <waterfoot@gmail.com> wrote:
>: PackBits does what it says. On the D the 12-bit data was simply stored in a
>: 16-bit TIFF so it occupied 12M. The DS packs the data but neither compress
>: it, hence the 9M data size of the DS. Round up to 13M and 10M respectively
>: including the JPEGs and red-tape stuff.
>
> That's what I thought initially. Within the context of TIFF files, PackBits
> compression is supposed to be an RLE encoding/compression... not just removing zero
> padding. The file size doesn't agree with that, however, and seems to imply no
> compresison at all.

Ah, sorry, I didn't look up the definition I just assumed it would pack
only.

> For something as noisy as raw sensor data, PackBits would be a lousy
> compression scheme. I wouldn't doubt it if the Pentax engineers abused the TIFF
> format further by calling it PackBits when it's really just uncompressed non-padded
> 12-bit data.

Having made their choice - for whatever reason - not to compress the data,
what would they call it? I'm no TIFF expert but AFAIK there's no explicit
option to "store" 12-bit data, whereas it could be argued that uncompressed
but otherwise packed data would fall under the PackBits scheme. I don't know
why Pentax deviated from TIFF unlike (say) Nikon, but my guess is may be the
other way around. Perhaps they had no intention of making the data TIFF
compatible but simply made the container sufficiently TIFF-like to allow
EXIF to be parsed. But it's pointless to try to outguess them.

Personally I would like to see PEF - and others - dropped in favour of
another TIFF variant DNG which uses lossless JPEG compression of the RAW
data, but that's another story.

--
John Bean

All programmers are playwrights and all computers are lousy actors (Anon)
Anonymous
January 6, 2005 11:30:18 PM

Archived from groups: rec.photo.digital (More info?)

I guess you will also not like the fact that the jpeg compression is very
variable as well, taken at maximum settings depending on the subject the
files will vary from 1.8 - 4.2 Mbs.

This camera is not for you, this is a consumer level camera aimed at
amateurs that want more, not professionals like yourself. RAW mode will be
used sometimes but users of this camera will be happy with what it provides.

If you post with a useable email address I will send you a .pef file, any
prefs on the subject?

<papenfussDIESPAM@juneauDOTmeDOTvt.edu> wrote in message
news:crh7hm$eqd$1@solaris.cc.vt.edu...
> Hey all... getting closer to purchasing a *istDS, but I'm a bit troubled
> by a
> design choice of Pentaxs'. It would appear that the RAW file size is
> *HUGE* by
> comparison to other brands. I've heard rumors that they both pad the
> 12-bits off the
> A/D to 16 bits, AND don't do compression. It would seem like a silly
> thing to not do
> a lossless compression on the RAW files unless computational power was an
> issue. Can
> anyone confirm or deny such a thing? I've been scouring the 'net looking
> for a sample
> RAW .PEF file to look through, but haven't had any luck with that either.
>
> If it is too difficult to compress on-the-fly at the time of the shot, at
> least a "pack photos" option or something would be a very good upgrade to
> the
> firmware. One could let the camera chew on the already taken pictures and
> crunch them
> down to 50% of their original size to fit 100 pics on a 512MB card, rather
> than 45.
>
> Anyone care to comment?
>
> -Cory
>
> --
>
> *************************************************************************
> * Cory Papenfuss *
> * Electrical Engineering candidate Ph.D. graduate student *
> * Virginia Polytechnic Institute and State University *
> *************************************************************************
>
Anonymous
January 7, 2005 12:12:20 AM

Archived from groups: rec.photo.digital (More info?)

Pete D <no@email.com> wrote:
: This camera is not for you, this is a consumer level camera aimed at
: amateurs that want more, not professionals like yourself. RAW mode will be
: used sometimes but users of this camera will be happy with what it provides.

Had to read that carefully... thought you called me an amateur... :)  I'm
nothing resembling a professional, but I'm cursed with being a perfectionist and an
electrical engineer. I guess that's an engineered amateur... That means whatever I
do, I try to get the best possible performance out of it.

My opinion of DSLRs isn't that much different from that of their 35mm
counterparts. Basically, it's a light box with a sensor and memory in it. The DSLR
has the added advantage of instant review, autometer/focus, etc, but most of the
"features" are irrelevent when shooting raw. I'm certainly fine with JPEGs for
snap-n-shoot stuff (birthday parties, etc), and I'm fine with them taking great
pictures with "Auto." For "professional" uses, I'll use it reverted to a metered
manual lightbox with a raw sensor. It's just that it's not quite as optimized in that
regard as it could be (big but low-quality JPEG and uncompressed data). It'll still
get the job done.

I'm partial to Pentax because the have great bang-for-the-buck, and haven't
alienated their following with a new lens format.

: If you post with a useable email address I will send you a .pef file, any
: prefs on the subject?

I did actually find a few .PEF files, but I'm still looking for some good
sensor noise shots. The same subject exposure at ISO200, 800, 3200 would be great.
Probably something with a wide dynamic range to pull what's in the shadows (and how
much noise there is). My email is obfuscated, not unusable... remove DIESPAM and
change and DOT to a '.'

Thanks (this whole thing has been an interesting banter)... :) 

-Cory

--

*************************************************************************
* Cory Papenfuss *
* Electrical Engineering candidate Ph.D. graduate student *
* Virginia Polytechnic Institute and State University *
*************************************************************************
!