UFF (universal file format) idea!

By now, you've probably all heard about my universal file format project that encodes binary data into ASCII symbols. At this point in time, my project is beginning to mature from simply encoding messages to encoding entire files (multiple files in fact) into an encoded archive. The clearest advantage to this format is to be able to transfer this data to paper. However I'm beginning to get a new idea for this format. Up here on Tom's Hardware, we can't post pics within the forum (we can only paste links). On MSN, AIM and ICQ, sometimes it's impossible to get a file or send a file to someone with a router. Also, some people reject e-mail with attachments. What if you could simply post these symbols or send these symbols in an IM or email and then copy and paste them in my Archive utility, where it will reconstitute the data and let you manipulate it and extract it as you would a zip archive. Best of all, my format is far more reliable than Zip but just as secure if not more.

<b>Note:</b> Unlike Zip, UFF's main objective is versatility not maximum compression.

Intelligence is not merely the wealth of knowledge but the sum of perception, wisdom, and knowledge.

Well? Anyone think this might be a useful program? I'm seriously considering making this public for people to use online.

Intelligence is not merely the wealth of knowledge but the sum of perception, wisdom, and knowledge.

Fabulous idea....Can I beta test?

Get your own sig!!!

Sure! In a couple of weeks, I'll make it open for beta testing.

Intelligence is not merely the wealth of knowledge but the sum of perception, wisdom, and knowledge.

Yup, that was my thinking exactly. It should allow for the transfer of files over the forum in a much easier fashion. You will need the .NET Framework from Windows Update if you don't have it yet though.

Intelligence is not merely the wealth of knowledge but the sum of perception, wisdom, and knowledge.

Well, work on WinUFF (that's what I decided to call it) is well under way. All the basic functionality is complete. You can create archives, modify archives by adding and deleting files within them, load archives to view the files within them, and extract files from within the archives. The program associates itself with .UFF files.

The current limitations of the file format are as follows:

Maximum file header + trailer size = 1MB
Maximum file size (that can be added to the archive)= 2GB
Maximum archive size = 2^23 GB
Maximum number of files within archive = over 2 billion (2^31 to be exact) files

As you can see, the most major limitation here is the size of the file header + trailer so that limiation will be reached far before the others. This limitation exists because I'm using 2 unsigned 16-bit integers, one to represent the length of the header and one to represent the length of the trailer. However, because each byte only represents a single bit, then a 16-bit int allows for a file header of the size of 2^16 * 8. The same applies for the trailer. However, I can easily raise that limit to 2^32 * 8.

The following features I'm in the process of finishing:

Redo/Undo action
Test/Repair Archive
Adding Comments
Renaming files within archive
Adding complete folders to archive
Password protection
Spanning and reconstitution

Finally, the user interface is nearly identical to Winzip.

EDIT: I forgot to mention that WinUFF automatically creates a backup of the archive if you delete files from the archive.

Intelligence is not merely the wealth of knowledge but the sum of perception, wisdom, and knowledge.

I also would love to beta it, hope you get it all compiled soon.

A womans body is a temple built upon a sewer.

Ok, there are now two forms of UFF. Hexadecimal UFF and Binary UFF. Hexadecimal UFF's are smaller in size (exactly 1/4 the size of Binary UFF), but binary is better for print. Performance wise, both are about the same.

Speaking of which, performance optimization is a huge issue here. I have to find the right balance between performance and memory usage which is very difficult. I can get it to encode a 4MB file in 3 or 4 sec, and extract that same file in 40sec. In my opinion that's not acceptable so I'll continue optimizing until I can cut those times in half.

Intelligence is not merely the wealth of knowledge but the sum of perception, wisdom, and knowledge.

Well, I can scan in UFF encoded archives and OCR them but dust is a nightmare. My stupid OCR algorithm misdetects a small bit of dust on the glass bed of the scanner as a symbol. It's ANNOYING! I have no idea how to write a subroutine to remove dust pixels. The best I can do is only accept the very dark black colours. That should hopefully remove 99% of dust particles in scans but I can't be sure.

Intelligence is not merely the wealth of knowledge but the sum of perception, wisdom, and knowledge.

Holy peeperdepeep... talking about programming skills.

I love my Delta 60HP 7000 RPM fan that puts out more dB then CFM

Well, I've just had an inspiration in developing a new dust removal filter that should improve accuracy SIGNIFICANTLY for higher density information storage on paper.
This was my idea:

#1: The idea is when you print out a paper archive produced by WinUFF all the horizontal spacing between characters and the vertical spacing between lines MUST exist.

#2: Under that assumption, the filter basically takes the scanned image and converts it into a grid based on currently apparent spacing.

#3: For each square region in the grid it performs a horizontal and vertical pass.

#4: The vertical pass counts the number of vertical pixels each line and finds the average number of pixels per line in the region minus one (as a safety precaution ). This average becomes a theshold that I will explain later.

#5: Now, chances are those number of pixels per line is not constant, but they shouldn't be off more than a couple of pixels. However, chances are also that the top of the characters and the bottom aren't aligned perfectly or there are dust particles linking two lines together. In those lines, the pixel count will be well under the threshold.

#6: Those lines that are under the threshold average will be cleared (converted to white pixels).

#7 The same idea applies for the horizontal pass. It builds a threshold average and removes the anomalous vertical lines between characters.

Basically, this filter will fix problems line \ / looking like a V.

This new filter would probably take a few seconds to complete for a 600dpi scanned page but it will render two older filter I had written obsolete so that will save some time.

Intelligence is not merely the wealth of knowledge but the sum of perception, wisdom, and knowledge.