Archival scans, 48bit, Nikon Coolscan 5000 ED, ? Gamma 1.0 ?

[Guest]

Archived from groups: alt.comp.periphs.scanner,comp.graphics.apps.photoshop (More info?)

Hi All,

I wish to scan ~3-4000 slides, for two reasons, one to have the images
available electronically but mainly to have a safe archive/backup of the
images (most of these slides cover the period when I used to work for the
British Antarctic Survey, are c20 years old and I would be gutted if I lost
them...).

I've had a Nikon Coolscan 5000 ED for a couple of months, and have spent the
time becoming familiar with it.... and colour management. On the colour
management issues I am now just starting to get a good overall idea of how
things work (and I must admit it was not simple, and I am speaking as a
lapsed physicist!).

I would like to scan these slides ONCE - i.e. I would like to get it right
the first time. I intend to scan at 48bits and 4000dpi (i.e. the max
resolution of the scanner).

Can anyone comment on the scenario below:

---------------------

* 16bits/channel / 4000dpi
* Raw scanner RGB at - gamma 1.0 - (Nikon colour management turned off).
* Only processing performed by the scanner being digital ICE
* Scanner calibrated using it8 targets and resultant icc profiles used to
perform conversion to the working colour space (presently Wide Gamut RGB)
on import of the raw gamma 1.0 files to Photoshop

----------------------

I am aware that there is a somewhat heated discussion on the subject of
gamma 1.0 editing, which is not what I am proposing here. My concern is
complete retention of the data delivered by the scanner. My reasoning is:

* The scanner sensor has a 16bit resolution.
* I acknowledge the sense in outputing a higher gamma file when using 8
bits/channel in order to space the resultant resolution perceptually.
However when performing such a transform on the full bit data all I see is
an increase in spacing of the scanner resolution at the shadow end at the
cost of lost information in the highlights. I.e. I see no gain.
* The scans are archival - I might wish to use the data in a couple of
decades, with display technologies that may be completely different from
today (i.e. why gamma encode the data with a value that derives from
today's display technology).

I would be particularly interested to hear from people in the high gamma
camp(!), since I would guess from the gamma 1.0 camp I am going to hear "Go
for it". The only potential problem that I can see here is whether the
application of a gamma 2.2 curve through Photoshop/icc profile is any less
accurate than in the scanner itself. I acknowledge that there may be
others I have missed....

Cheers,

Gary Whitehead.



N.B. I too fought with the colour management on the scanner, and gave up in
near disgust. Wolf Faust's targets, and resultant ICC profiles gave the
best results I had seen within minutes of generating them!

 

[Jim]

Archived from groups: alt.comp.periphs.scanner (More info?)

"Gary Whitehead" <gary.whitehead@t-online.de> wrote in message
news:cfr6qe$ad9$05$1@news.t-online.com...
> Hi All,
>
> Can anyone comment on the scenario below:
>
> ---------------------
>
> * 16bits/channel / 4000dpi
> * Raw scanner RGB at - gamma 1.0 - (Nikon colour management turned off).
> * Only processing performed by the scanner being digital ICE
> * Scanner calibrated using it8 targets and resultant icc profiles used to
> perform conversion to the working colour space (presently Wide Gamut RGB)
> on import of the raw gamma 1.0 files to Photoshop
>
This seems to me to be the preferred method. Create a profile of your
scanner with color management turned off. You then take the resultant
output to whichever program you prefer, apply the profile, and convert to
whichever working color space you desire. It works for me.
Jim

 

[Guest]

Archived from groups: alt.comp.periphs.scanner (More info?)

Sounds good ... but jus to be sure include a raw scan of a well-known target ... such as
an IT-8 target for which you have the data, or an image of an Macbeth color checker ...
with your archives.





"Gary Whitehead" <gary.whitehead@t-online.de> wrote in message
news:cfr6qe$ad9$05$1@news.t-online.com...
> Hi All,
>
> I wish to scan ~3-4000 slides, for two reasons, one to have the images
> available electronically but mainly to have a safe archive/backup of the
> images (most of these slides cover the period when I used to work for the
> British Antarctic Survey, are c20 years old and I would be gutted if I lost
> them...).
>
> I've had a Nikon Coolscan 5000 ED for a couple of months, and have spent the
> time becoming familiar with it.... and colour management. On the colour
> management issues I am now just starting to get a good overall idea of how
> things work (and I must admit it was not simple, and I am speaking as a
> lapsed physicist!).
>
> I would like to scan these slides ONCE - i.e. I would like to get it right
> the first time. I intend to scan at 48bits and 4000dpi (i.e. the max
> resolution of the scanner).
>
> Can anyone comment on the scenario below:
>
> ---------------------
>
> * 16bits/channel / 4000dpi
> * Raw scanner RGB at - gamma 1.0 - (Nikon colour management turned off).
> * Only processing performed by the scanner being digital ICE
> * Scanner calibrated using it8 targets and resultant icc profiles used to
> perform conversion to the working colour space (presently Wide Gamut RGB)
> on import of the raw gamma 1.0 files to Photoshop
>
> ----------------------
>
> I am aware that there is a somewhat heated discussion on the subject of
> gamma 1.0 editing, which is not what I am proposing here. My concern is
> complete retention of the data delivered by the scanner. My reasoning is:
>
> * The scanner sensor has a 16bit resolution.
> * I acknowledge the sense in outputing a higher gamma file when using 8
> bits/channel in order to space the resultant resolution perceptually.
> However when performing such a transform on the full bit data all I see is
> an increase in spacing of the scanner resolution at the shadow end at the
> cost of lost information in the highlights. I.e. I see no gain.
> * The scans are archival - I might wish to use the data in a couple of
> decades, with display technologies that may be completely different from
> today (i.e. why gamma encode the data with a value that derives from
> today's display technology).
>
> I would be particularly interested to hear from people in the high gamma
> camp(!), since I would guess from the gamma 1.0 camp I am going to hear "Go
> for it". The only potential problem that I can see here is whether the
> application of a gamma 2.2 curve through Photoshop/icc profile is any less
> accurate than in the scanner itself. I acknowledge that there may be
> others I have missed....
>
> Cheers,
>
> Gary Whitehead.
>
>
>
> N.B. I too fought with the colour management on the scanner, and gave up in
> near disgust. Wolf Faust's targets, and resultant ICC profiles gave the
> best results I had seen within minutes of generating them!

 

[Guest]

Archived from groups: alt.comp.periphs.scanner (More info?)

Have you considered the practical realities of creating files this large,
about 100mbs per file?

Have you considered the real world utility of 16 bit color vs the ideal
(i.e. no monitor or printing method can utilize that much color information
and will in some way arbitrarily truncate it anyway)?

I hope your are a very young man so that you may live to complete this
project.

 

[Greg]

Archived from groups: alt.comp.periphs.scanner (More info?)

Go for it.

The scanner hardware, I believe, *always* scans with a gamma of 1, with the
gamma adjustment
being performed by software.

The *only* reason I don't use a gamma of 1 is that the profiler I use (the
Little CMS scanner profiler)
will only use 8-bit scans. If I feed the profiler higher bit depth target
scans, the profiler definitely
does not make use of the extra resolution. So, I use a gamma of 2.2 to get
around this problem.

I think your workflow makes complete sense, and aside from the difference in
gamma, it's
the same as my workflow for all intents & purposes.

Greg.

"Gary Whitehead" <gary.whitehead@t-online.de> wrote in message
news:cfr6qe$ad9$05$1@news.t-online.com...
> Hi All,
>
> I wish to scan ~3-4000 slides, for two reasons, one to have the images
> available electronically but mainly to have a safe archive/backup of the
> images (most of these slides cover the period when I used to work for the
> British Antarctic Survey, are c20 years old and I would be gutted if I
> lost
> them...).
>
> I've had a Nikon Coolscan 5000 ED for a couple of months, and have spent
> the
> time becoming familiar with it.... and colour management. On the colour
> management issues I am now just starting to get a good overall idea of how
> things work (and I must admit it was not simple, and I am speaking as a
> lapsed physicist!).
>
> I would like to scan these slides ONCE - i.e. I would like to get it right
> the first time. I intend to scan at 48bits and 4000dpi (i.e. the max
> resolution of the scanner).
>
> Can anyone comment on the scenario below:
>
> ---------------------
>
> * 16bits/channel / 4000dpi
> * Raw scanner RGB at - gamma 1.0 - (Nikon colour management turned off).
> * Only processing performed by the scanner being digital ICE
> * Scanner calibrated using it8 targets and resultant icc profiles used to
> perform conversion to the working colour space (presently Wide Gamut RGB)
> on import of the raw gamma 1.0 files to Photoshop
>
> ----------------------
>
> I am aware that there is a somewhat heated discussion on the subject of
> gamma 1.0 editing, which is not what I am proposing here. My concern is
> complete retention of the data delivered by the scanner. My reasoning is:
>
> * The scanner sensor has a 16bit resolution.
> * I acknowledge the sense in outputing a higher gamma file when using 8
> bits/channel in order to space the resultant resolution perceptually.
> However when performing such a transform on the full bit data all I see is
> an increase in spacing of the scanner resolution at the shadow end at the
> cost of lost information in the highlights. I.e. I see no gain.
> * The scans are archival - I might wish to use the data in a couple of
> decades, with display technologies that may be completely different from
> today (i.e. why gamma encode the data with a value that derives from
> today's display technology).
>
> I would be particularly interested to hear from people in the high gamma
> camp(!), since I would guess from the gamma 1.0 camp I am going to hear
> "Go
> for it". The only potential problem that I can see here is whether the
> application of a gamma 2.2 curve through Photoshop/icc profile is any less
> accurate than in the scanner itself. I acknowledge that there may be
> others I have missed....
>
> Cheers,
>
> Gary Whitehead.
>
>
>
> N.B. I too fought with the colour management on the scanner, and gave up
> in
> near disgust. Wolf Faust's targets, and resultant ICC profiles gave the
> best results I had seen within minutes of generating them!

 

[Guest]

Archived from groups: alt.comp.periphs.scanner (More info?)

DAT tapes!

Having the disc space for this is not a problem nowadays, but I certainly
would not be doing it without backup. The entire collection should fit on
around 30 tapes (12GB), at a cost of about 150 euros.

I'm also reckoning on us being at a cusp, i.e. in a few years the amount of
data that this will produce will fit easily into normal spec machines.

Cheers,

Gary.

bmoag wrote:

> Have you considered the practical realities of creating files this large,
> about 100mbs per file?
>
> Have you considered the real world utility of 16 bit color vs the ideal
> (i.e. no monitor or printing method can utilize that much color
> information and will in some way arbitrarily truncate it anyway)?
>
> I hope your are a very young man so that you may live to complete this
> project.

 

[Greg]

Archived from groups: alt.comp.periphs.scanner (More info?)

Good point - I forgot to mention that I use JPEG 2000 to archive my scans.

Gary - I suggest you look into JPEG 2000 as a possibility for storing your
achives. JPEG 2000 supports greater than 8-bit per channel encoding, whereas
standard JPEG doesn't.

Yes, I use JPEG 2000 in lossy mode, but it does have a lossless mode as
well.

Greg.
"bmoag" <aetoo@yahoo.com> wrote in message
news:n3eUc.5016$oL7.4353@newssvr27.news.prodigy.com...
> Have you considered the practical realities of creating files this large,
> about 100mbs per file?
>
> Have you considered the real world utility of 16 bit color vs the ideal
> (i.e. no monitor or printing method can utilize that much color
> information
> and will in some way arbitrarily truncate it anyway)?
>
> I hope your are a very young man so that you may live to complete this
> project.
>
>
>

 

[Guest]

Archived from groups: alt.comp.periphs.scanner (More info?)

External Hard drive or CDR or even the new DVD data recorders.

Tape has the disadvantage of having to play the tape until you find the
selection you are looking for. Also Tape is slow in transfer rate.

For 150 Euro you can get a 120 GB external hard drive and case. USB 2.0 or
Firewire. In the USA a external hard drive case is about $40 USD and 120 GB
hard drive is about $90 USD.

--
CSM1
http://www.carlmcmillan.com
--
"Gary Whitehead" <gary.whitehead@t-online.de> wrote in message
news:cfs9qd$a28$05$1@news.t-online.com...
> DAT tapes!
>
> Having the disc space for this is not a problem nowadays, but I certainly
> would not be doing it without backup. The entire collection should fit
on
> around 30 tapes (12GB), at a cost of about 150 euros.
>
> I'm also reckoning on us being at a cusp, i.e. in a few years the amount
of
> data that this will produce will fit easily into normal spec machines.
>
> Cheers,
>
> Gary.
>
> bmoag wrote:
>
> > Have you considered the practical realities of creating files this
large,
> > about 100mbs per file?
> >
> > Have you considered the real world utility of 16 bit color vs the ideal
> > (i.e. no monitor or printing method can utilize that much color
> > information and will in some way arbitrarily truncate it anyway)?
> >
> > I hope your are a very young man so that you may live to complete this
> > project.
>

 

[Guest]

Archived from groups: alt.comp.periphs.scanner (More info?)

On Tue, 17 Aug 2004 15:29:10 GMT, "CSM1" <nomoremail@nomail.com>
wrote:

>External Hard drive or CDR or even the new DVD data recorders.
>
And here I'm up to something on the order of 2 sets of 60 DVDs plus a
complete backup on a second computer. I have about 3,000 to 4,000
negatives left to scan and about a 100# worth of *old* prints all the
way back to tintypes and the "petrified cardboard" prints.
The unedited TIFFs run about 60 megs each.

All slides and negatives are indexed in notebooks.

Roger Halstead (K8RI & ARRL life member)
(N833R, S# CD-2 Worlds oldest Debonair)
www.rogerhalstead.com

>Tape has the disadvantage of having to play the tape until you find the
>selection you are looking for. Also Tape is slow in transfer rate.
>
>For 150 Euro you can get a 120 GB external hard drive and case. USB 2.0 or
>Firewire. In the USA a external hard drive case is about $40 USD and 120 GB
>hard drive is about $90 USD.
>
>--
>CSM1
>http://www.carlmcmillan.com

 

[Al]

Archived from groups: alt.comp.periphs.scanner (More info?)

You have an excellent film scanner there. Though I may not know much about
that type of scanning, at 4000dpi x 3-4000 scans, I'm assuming you have a
terrabyte server handy

I can only assume that if you scan at a default 1.0 gamma, you still have
more than enough information to change it to a better gamma/icc profile
through photoshop, possibly even through using a repetitive action. But I
also think that not all pictures will be the same, therefore a pic that
looks decent with a gamma 1 may not be applicable to another pic. But
that's just my newbie thinking

Good luck anyway in whatever you choose to do.

Al

in article cfr6qe$ad9$05$1@news.t-online.com, Gary Whitehead at
gary.whitehead@t-online.de wrote on 17/8/04 6:52 AM:

> Hi All,
>
> I wish to scan ~3-4000 slides, for two reasons, one to have the images
> available electronically but mainly to have a safe archive/backup of the
> images (most of these slides cover the period when I used to work for the
> British Antarctic Survey, are c20 years old and I would be gutted if I lost
> them...).
>
> I've had a Nikon Coolscan 5000 ED for a couple of months, and have spent the
> time becoming familiar with it.... and colour management. On the colour
> management issues I am now just starting to get a good overall idea of how
> things work (and I must admit it was not simple, and I am speaking as a
> lapsed physicist!).
>
> I would like to scan these slides ONCE - i.e. I would like to get it right
> the first time. I intend to scan at 48bits and 4000dpi (i.e. the max
> resolution of the scanner).
>
> Can anyone comment on the scenario below:
>
> ---------------------
>
> * 16bits/channel / 4000dpi
> * Raw scanner RGB at - gamma 1.0 - (Nikon colour management turned off).
> * Only processing performed by the scanner being digital ICE
> * Scanner calibrated using it8 targets and resultant icc profiles used to
> perform conversion to the working colour space (presently Wide Gamut RGB)
> on import of the raw gamma 1.0 files to Photoshop
>
> ----------------------
>
> I am aware that there is a somewhat heated discussion on the subject of
> gamma 1.0 editing, which is not what I am proposing here. My concern is
> complete retention of the data delivered by the scanner. My reasoning is:
>
> * The scanner sensor has a 16bit resolution.
> * I acknowledge the sense in outputing a higher gamma file when using 8
> bits/channel in order to space the resultant resolution perceptually.
> However when performing such a transform on the full bit data all I see is
> an increase in spacing of the scanner resolution at the shadow end at the
> cost of lost information in the highlights. I.e. I see no gain.
> * The scans are archival - I might wish to use the data in a couple of
> decades, with display technologies that may be completely different from
> today (i.e. why gamma encode the data with a value that derives from
> today's display technology).
>
> I would be particularly interested to hear from people in the high gamma
> camp(!), since I would guess from the gamma 1.0 camp I am going to hear "Go
> for it". The only potential problem that I can see here is whether the
> application of a gamma 2.2 curve through Photoshop/icc profile is any less
> accurate than in the scanner itself. I acknowledge that there may be
> others I have missed....
>
> Cheers,
>
> Gary Whitehead.
>
>
>
> N.B. I too fought with the colour management on the scanner, and gave up in
> near disgust. Wolf Faust's targets, and resultant ICC profiles gave the
> best results I had seen within minutes of generating them!

 

[Guest]

Archived from groups: alt.comp.periphs.scanner (More info?)

On Mon, 16 Aug 2004 22:52:30 +0200, Gary Whitehead
<gary.whitehead@t-online.de> wrote:

Hi Gary,

As I mentioned in a previous post I've gone through many a thousand
images so far. They ranged from Kodachrome (many versions with some
as old as 50 years) Ektachrome, and a wide variety of 35mm negatives.

It might help if you sort your slides by age, but you will find some
pretty wide variations.

>Hi All,
>
>I wish to scan ~3-4000 slides, for two reasons, one to have the images
>available electronically but mainly to have a safe archive/backup of the
>images (most of these slides cover the period when I used to work for the
>British Antarctic Survey, are c20 years old and I would be gutted if I lost
>them...).

I think the backup is a good idea, but if the slides were properly
processed they will probably be around in good shape longer than the
original digital media to which you save them.

>
>I've had a Nikon Coolscan 5000 ED for a couple of months, and have spent the
>time becoming familiar with it.... and colour management. On the colour

I've been using the same since some where around Feb., or March.

>management issues I am now just starting to get a good overall idea of how
>things work (and I must admit it was not simple, and I am speaking as a
>lapsed physicist!).
>
>I would like to scan these slides ONCE - i.e. I would like to get it right
>the first time. I intend to scan at 48bits and 4000dpi (i.e. the max
>resolution of the scanner).

At that resolution and depth the basic slide with no cropping will run
about 126 to 130 megs. That comes out to about 520 Gigs of storage.
or about 113 DVDs.

I scan at 8 bits and haven't found any reason to go beyond that. The
files are still roughly 60 megs as TIFFs.

>
>Can anyone comment on the scenario below:
>
>---------------------
>
>* 16bits/channel / 4000dpi
>* Raw scanner RGB at - gamma 1.0 - (Nikon colour management turned off).
>* Only processing performed by the scanner being digital ICE

My experience, using the SF210 automated slide feeder using digital
ice only will be roughly 30 to 40 seconds per image.

>* Scanner calibrated using it8 targets and resultant icc profiles used to
>perform conversion to the working colour space (presently Wide Gamut RGB)
>on import of the raw gamma 1.0 files to Photoshop
>
>----------------------
>
>I am aware that there is a somewhat heated discussion on the subject of
>gamma 1.0 editing, which is not what I am proposing here. My concern is
>complete retention of the data delivered by the scanner. My reasoning is:
>
>* The scanner sensor has a 16bit resolution.
>* I acknowledge the sense in outputing a higher gamma file when using 8
>bits/channel in order to space the resultant resolution perceptually.
>However when performing such a transform on the full bit data all I see is
>an increase in spacing of the scanner resolution at the shadow end at the
>cost of lost information in the highlights. I.e. I see no gain.
>* The scans are archival - I might wish to use the data in a couple of
>decades, with display technologies that may be completely different from

Here lies a problem. To maintain data integrity you are going to have
to refresh it at least once every ten years, or more frequently, as
there is no real data to support storage life beyond that. Beyond a
few years data lifetime is based on accelerated lifetime testing and
projections.

When archiving, the general approach is to make two copies and store
them in separate locations that are friendly to the medium. (Dark,
with temperature and humidity control)

As you are not going to be using a rolling backup that rules out
magnetic storage. Tapes and hard drives may have lifetimes of many
years, but the lifetime to maintain data integrity is very short.

Even with today's advances in hard drives which last 100s of thousands
of hours you can not depend on that kind of life for the data.
Typically HDs are refreshed on a monthly basis. I'd not want to trust
one even in storage beyond a year.

Optical on the other hand is potentially very long lived, but not
proven. How long a particular medium remains viable (will the hard
ware remain available to read and write said medium) and what media is
practical for the amount of storage you need? Beyond 10 years there
is the very real possibility of the need to change to a different
media.

Currently about the only thing I see that would meet the goal are DVDs
and they come with no guarantee, only a projected lifetime.
Unfortunately DVDs tend to vary in quality and can be susceptible to
damage from handeling, which pretty much describes most optical disks.

Store them on edge in Jewel cases and keep away from sun light. They
should also be kept in a relatively low humidity and cool, but not
cold temperatures. Check the manufacturers specs on storage. Above
all, do not flex DVDs as most do when taking them out of the case.
Press down in the center of the case and the DVD will pop out. Do not
pull up on the edges. That will cause flexing and the DVD is a two
layer device which can separate, or fracture.

You are really on your own as far as quality control and how often you
check them for data integrity. If you ever find a corrupt file you
know you have waited too long. Hopefully the ones on the second backup
will still be good, or at least the same files will not have failed.

Roger Halstead (K8RI & ARRL life member)
(N833R, S# CD-2 Worlds oldest Debonair)
www.rogerhalstead.com
>today (i.e. why gamma encode the data with a value that derives from
>today's display technology).
>
>I would be particularly interested to hear from people in the high gamma
>camp(!), since I would guess from the gamma 1.0 camp I am going to hear "Go
>for it". The only potential problem that I can see here is whether the
>application of a gamma 2.2 curve through Photoshop/icc profile is any less
>accurate than in the scanner itself. I acknowledge that there may be
>others I have missed....
>
>Cheers,
>
>Gary Whitehead.
>
>
>
>N.B. I too fought with the colour management on the scanner, and gave up in
>near disgust. Wolf Faust's targets, and resultant ICC profiles gave the
>best results I had seen within minutes of generating them!

 

[Guest]

Archived from groups: alt.comp.periphs.scanner (More info?)

Roger Halstead wrote:

> Here lies a problem. To maintain data integrity you are going to have
> to refresh it at least once every ten years, or more frequently, as
> there is no real data to support storage life beyond that. Beyond a
> few years data lifetime is based on accelerated lifetime testing and
> projections.
>
> When archiving, the general approach is to make two copies and store
> them in separate locations that are friendly to the medium. (Dark,
> with temperature and humidity control)
>
> As you are not going to be using a rolling backup that rules out
> magnetic storage. Tapes and hard drives may have lifetimes of many
> years, but the lifetime to maintain data integrity is very short.
>
> Even with today's advances in hard drives which last 100s of thousands
> of hours you can not depend on that kind of life for the data.
> Typically HDs are refreshed on a monthly basis. I'd not want to trust
> one even in storage beyond a year.
>
> Optical on the other hand is potentially very long lived, but not
> proven. How long a particular medium remains viable (will the hard
> ware remain available to read and write said medium) and what media is
> practical for the amount of storage you need? Beyond 10 years there
> is the very real possibility of the need to change to a different
> media.
>
> Currently about the only thing I see that would meet the goal are DVDs
> and they come with no guarantee, only a projected lifetime.
> Unfortunately DVDs tend to vary in quality and can be susceptible to
> damage from handeling, which pretty much describes most optical disks.
>
> Store them on edge in Jewel cases and keep away from sun light. They
> should also be kept in a relatively low humidity and cool, but not
> cold temperatures. Check the manufacturers specs on storage. Above
> all, do not flex DVDs as most do when taking them out of the case.
> Press down in the center of the case and the DVD will pop out. Do not
> pull up on the edges. That will cause flexing and the DVD is a two
> layer device which can separate, or fracture.
>
> You are really on your own as far as quality control and how often you
> check them for data integrity. If you ever find a corrupt file you
> know you have waited too long. Hopefully the ones on the second backup
> will still be good, or at least the same files will not have failed.
>

Long term, I do not see too much of a problem in that relatively cheap
devices that can hold the entire collection without raising a sweat are
only around the corner - 300GB consumer drives are now available. Then I
can easily manage redundant online storage.

Short term I will have the set online on my server (for myself and my
girlfriend I run a house (Linux) server, files, printing email etc), and
rely on DAT tape for the offline backup. Basically, I can manage it now
(with some hassle) and have redundancy, and wait in hope for the whatever
cheap terabyte devices that are available in a couple of years. DAT tapes
are advertised to have an archival lifetime in the range of at least a
decade (and tape is a mature technolgy) which is much longer than I intend
to rely on them for. They are also relatively cheap in cost/GB.

 

[Guest]

Archived from groups: alt.comp.periphs.scanner (More info?)

CSM1 wrote:

> Tape has the disadvantage of having to play the tape until you find the
> selection you are looking for. Also Tape is slow in transfer rate.

Aye, but I plan to have them online as well (a couple of discs nowadays),
the tape is for redundancy. However, I must admit that restoring the
collection in the case of a disc failure would not be fun (c3hrs per tape).

 

[Guest]

Archived from groups: alt.comp.periphs.scanner (More info?)

Greg wrote:

> The *only* reason I don't use a gamma of 1 is that the profiler I use (the
> Little CMS scanner profiler)
> will only use 8-bit scans. If I feed the profiler higher bit depth target
> scans, the profiler definitely
> does not make use of the extra resolution. So, I use a gamma of 2.2 to get
> around this problem.

I have been thinking about this and looking at my scans of the IT8 targets.
I don't think that it actually matters much that we profile the scanners
using 8 bit scans.

Reasoning:

1. The scanned targets are actually quite noisy (film grain/target surface),
giving in the midtones (using Photoshop's histogram tool) a standard
deviation ~4-5bit (out of 256) on each channel, and maybe 1.5bits in the
highlights, and 0.5bits in the shadows (for a single target square).
Before anyone shouts at Wolf Faust, as I will explain below this is "Good
Thing".

2. At 4000dpi we are averaging around 10000 samples for a colour square and
maybe 40000 for a greyscale square. The resolution of the average can be
approximated to the standard error which will be

SD/SquareRoot(NoOfSamples).

This gives me in the case of the midtones a standard error of around 1/25th
of a bit for a colour square, which is almost an extra 5 bits of precision,
i.e. around 13bits. No useful improvement would be seen by using a 16 bit
scan. The other way of looking at it is that the quantisation noise of
8bit sampling (0.5bit) is insignificant compared to the target noise.
(Errors should be added as a sum of squares).

This is actually a standard technique in digital measurement, where it is
recognised that the mixture of some noise (greater than the bit resolution)
and averaging multiple samples allows sub bit resolution.

The only area that I have some doubt on this argument is in the deep
highlights (1-3 bits) and shadows. It is possible that there may be some
clipping of the noise component which would tend to shift the average
towards the midtones. I will take a look at a 16bit image later which I
will range expand (i.e. expand levels 0-2 -> 0-256). The test here is
whether I see a normal distribution of points. If I do, then this should
also be safe at the extremes.

Of course this does all depend on the profiler perfoming its calculation in
a sufficiently accurate data type and exporting the profiles in 16bits.
Little CMS does appear to use the LUT16Type in its output, but I have not
checked the code to see how it is calculated.

 

[Guest]

Archived from groups: alt.comp.periphs.scanner (More info?)

On Wed, 18 Aug 2004 19:49:06 +0200, Gary Whitehead
<gary.whitehead@t-online.de> wrote:

>Roger Halstead wrote:
>
>> Here lies a problem. To maintain data integrity you are going to have
>> to refresh it at least once every ten years, or more frequently, as
>> there is no real data to support storage life beyond that. Beyond a
>> few years data lifetime is based on accelerated lifetime testing and
>> projections.
>>

<snip>

>Long term, I do not see too much of a problem in that relatively cheap
>devices that can hold the entire collection without raising a sweat are
>only around the corner - 300GB consumer drives are now available. Then I

I hope to end up with a pair of 320s on each of three machines with a
serial RAID on the 4th. That 4th machine is remote in another
building.

>can easily manage redundant online storage.

Currently I run redundant on line (mine) storage as do you.
I have the work machine, one for backup and archive on DVD.

I figure Hard drive data should be refreshed at least once a year if
not more often. So far I've had good luck and it's quite fast across
the 100 Mbps network.

>
>Short term I will have the set online on my server (for myself and my
>girlfriend I run a house (Linux) server, files, printing email etc), and
>rely on DAT tape for the offline backup. Basically, I can manage it now
>(with some hassle) and have redundancy, and wait in hope for the whatever
>cheap terabyte devices that are available in a couple of years. DAT tapes

I have close to that now across my network at a relatively reasonable
cost. Going to the 6 300 Gig plus drives and the serial RAID will not
be trivial. They will make the 3 to 4 gig dual processor machine
coming up look pretty reasonable.

I run too much *stuff* and can bog down a single 2.8 gig processor.

>are advertised to have an archival lifetime in the range of at least a
>decade (and tape is a mature technolgy) which is much longer than I intend

True, but with he advertised life of a decade I'd want to refresh at 5
years, which is only three refreshes over 20 years.

>to rely on them for. They are also relatively cheap in cost/GB.

Good luck, It sounds like you have thought out your system well.

Roger Halstead (K8RI & ARRL life member)
(N833R, S# CD-2 Worlds oldest Debonair)
www.rogerhalstead.com

 

[Greg]

Archived from groups: alt.comp.periphs.scanner (More info?)

Gary,
I have not digested your notes in detail yet, but, I do know that the
version of the Little
CMS profiler I am using definitely will *not* use anything greater than
8-bits per channel
in the IT8 scans - I have even had this confirmed by the author himself. It
will load
the scan and process it, but it will not use the extra precision - it will
discard it.
Because of this, the author himself has said that it is very important *not*
to use a gamma
1 scan - it is important to use a perceptually uniform gamma, such as 2.2.
As you say, this issue is specific to the profiling software - other
profilers which really
can use the full precision would be entirelly suitable for use with gamma 1
IT8 target
scans.

Now, after saying all this, I am not sure whether there is a more recent
version of the
profiler available now - it's possible that this limitation has been
removed.

Greg.

"Gary Whitehead" <gary.whitehead@t-online.de> wrote in message
news:cg0624$d85$03$1@news.t-online.com...
> Greg wrote:
>
>> The *only* reason I don't use a gamma of 1 is that the profiler I use
>> (the
>> Little CMS scanner profiler)
>> will only use 8-bit scans. If I feed the profiler higher bit depth target
>> scans, the profiler definitely
>> does not make use of the extra resolution. So, I use a gamma of 2.2 to
>> get
>> around this problem.
>
> I have been thinking about this and looking at my scans of the IT8
> targets.
> I don't think that it actually matters much that we profile the scanners
> using 8 bit scans.
>
> Reasoning:
>
> 1. The scanned targets are actually quite noisy (film grain/target
> surface),
> giving in the midtones (using Photoshop's histogram tool) a standard
> deviation ~4-5bit (out of 256) on each channel, and maybe 1.5bits in the
> highlights, and 0.5bits in the shadows (for a single target square).
> Before anyone shouts at Wolf Faust, as I will explain below this is "Good
> Thing".
>
> 2. At 4000dpi we are averaging around 10000 samples for a colour square
> and
> maybe 40000 for a greyscale square. The resolution of the average can be
> approximated to the standard error which will be
>
> SD/SquareRoot(NoOfSamples).
>
> This gives me in the case of the midtones a standard error of around
> 1/25th
> of a bit for a colour square, which is almost an extra 5 bits of
> precision,
> i.e. around 13bits. No useful improvement would be seen by using a 16 bit
> scan. The other way of looking at it is that the quantisation noise of
> 8bit sampling (0.5bit) is insignificant compared to the target noise.
> (Errors should be added as a sum of squares).
>
> This is actually a standard technique in digital measurement, where it is
> recognised that the mixture of some noise (greater than the bit
> resolution)
> and averaging multiple samples allows sub bit resolution.
>
> The only area that I have some doubt on this argument is in the deep
> highlights (1-3 bits) and shadows. It is possible that there may be some
> clipping of the noise component which would tend to shift the average
> towards the midtones. I will take a look at a 16bit image later which I
> will range expand (i.e. expand levels 0-2 -> 0-256). The test here is
> whether I see a normal distribution of points. If I do, then this should
> also be safe at the extremes.
>
> Of course this does all depend on the profiler perfoming its calculation
> in
> a sufficiently accurate data type and exporting the profiles in 16bits.
> Little CMS does appear to use the LUT16Type in its output, but I have not
> checked the code to see how it is calculated.

 

[Greg]

Archived from groups: alt.comp.periphs.scanner (More info?)

Gary,
I have checked with the author (Marti Maria), and there is in fact a new
version of the scanner
program, available here: http://www.littlecms.com/profiler_qs.htm and this
really
does work with high resolution scans, and so in theory, gamma 1 should be
safe.
Marti cautions us that a) there is no support, and b) we should test it
thoroughly
with gamma 1 before relying on it too much, as gamma 1 still needs great
care.

If this is the version you're already using, then ignore my earlier warning.

Greg.
p.s I post this information with permission from Marti.