Best Way to Move/Sync Data| DevonThink Database Software

Status
Not open for further replies.

commissarmo

Distinguished
Jan 5, 2010
179
0
18,690
This is a Complex file-transfer problem - I hope someone with more experience can offer their candid advice - I already know I'm probably trying to solve a a semi-unsolvable problem...

BACKGROUND
I am trying to use DevonThink, the Mac-based personal database software. While it has its strengths, it's very poorly designed for data input (they basically want you to put all your data into the database proprietary binary format). But I want it to just have a copy of all my data which remains safely in my filesystem(s).

1. So I set up a file-sync system on my machines where target folders are copied to a WebDav folder. The files are then moved off the WebDav into DevonThink on a the Mac where it lives.

1A. I also want NEW files I create (and either save or rename into the monitored folders) on ANY sync-ed machine to transfer to the WebDav, and then get uploaded to DevonThink on the Mac, as well. So it's a bit circuitous, but I haven't figured out a better way to do it yet.

2. But this has not been NEARLY as simple as I thought it would be.

PROBLEMS

3. DevonThink has very poor file import abilities. It requires a flat-file import (it can't read new data in subfolders). This means I need to maintain the WebDav folder in FLAT file format, stripping the normal directory structure.

4. I actually managed to accomplish THIS too with Syncovery (a sync program which lets you sync to a target folder and strip the directory to FLAT file.)

5. Unfortunately, the FLAT file method created a lot of unforeseen problems:

A. Since the folder is flat, the sync software often encounters duplicate files, which it cannot handle well = errors.

B. It's very difficult to check the target folder against the origin folder for any files that were not transferred for whatever reason (using various folder compare tools fails because when a directory structure is stripped out, the dates attached to the files does not update, meaning in binary comparison, EVERY file 'looks' different).

When both folders showed a file count discrepancy of 200 files out of 3000 transfers, I could not easily determine which 200 files had been NOT transferred = terrible.

6. Additionally, importing data into DevonThink is also made somewhat difficult because of the import vector. DevonThink uses an open drop-folder (called Inbox), where files are dropped and are IMMEDIATELY Copied + Deleted into the database proper.

This means that sync-software CANNOT sync properly to this folder because they always SEE the folder as being EMPTY (and thus cannot compare file deltas, and they just copy everything in the folder again, only to have it emptied again)

7. In order to fix Problem #6, I found a Mac program called Hazel, which is NOT a sync program, but a task automator - it is capable, with some annoyances, of moving files into this empty Inbox folder, WITHOUT comparing the destination folder against the source, so THIS COMPONENT OF THE PROBLEM IS SOLVED.

8. Of course having all these chains of automated sync/copy/WebDav folders is probably unstable, and furthermore, it's likely for various reasons that various errors will always be going on in the background.

---

Other than "Don't Use DevonThink" (which is something I've been told several times, but there's not similar alternative I've found), is there a way to do this that I HAVE NOT THOUGHT OF? Even if it means sacrificing some automation, etc.?

-- I've considered just copying the bulk of the data into the database, and then just not worrying about newly created files; obviously this is a LOT simpler, but it means I will have to manually track new files somehow and then add them into the database at a later date...

I'd appreciate any advice.
 

commissarmo

Distinguished
Jan 5, 2010
179
0
18,690
Well to be honest, it's very simple to use (it's not technically a "database" I've been told by those who know such things - it's more of a file organization system apparently), has an AI engine which runs various comparison/similitude functions, allows in-system opening of various filetypes, automatically wiki-style links data, and allows file annotations/meta-tagging, and doesn't require any setup or configuration.

Even whilst being someone who (clearly) doesn't really even know what a formal database platform 'is,' DevonThink's only commonly quoted 'competitor' is EverNote, which I also use, but in my opinion, Evernote is only 10% similar to DevonThink - the reason it even gets quoted as similar, I believe, is just that people use both to throw various random data types into - 'everything bucket'.

*** I don't want to have to 'construct' my own database software in order to do this - if there's an off-the-shelf personal data management tool that's like this though, please tell me about it!
 

commissarmo

Distinguished
Jan 5, 2010
179
0
18,690
DevonThink is highly praised in the academic (my) community, for its ability to centralize one's personal data.

http://alternativeto.net/software/devonthink/

Other database programs like FileMaker, are much more set up for very formal "database" needs like inventory or personnel directories.

--

That said - I'm still trying to make DevonThink work as well, probably because I purchased it already...
 

commissarmo

Distinguished
Jan 5, 2010
179
0
18,690
I have managed to get my Goldberg machine for data transfer working... it transfers the data, but it does miss some files, which are then basically impossible for me to detect which ones out of thousands of total files...

But for now at least 90% of the data I have gets sent to the database software on the database computer.

Does anyone have any alternate personal database/information management solutions, or perhaps any other advice on how to sync/transfer data across WAN between multiple machines?

Thanks in advance!
 

jeroensangers

Honorable
Oct 24, 2013
1
0
10,510
DEVONthink store its contents in the original file format in the Library folder. It does NOT convert files to a proprietary format, like other applications (such as Evernote) do.

Besides putting files in the DEVONthink database, you also have the possibility to index an external folder. Any file you save in an indexed folder will be processed by DEVONthink and will appear alongside files stored inside DEVONthink. In my opinion, indexing folders is the solution to your problem.
 

commissarmo

Distinguished
Jan 5, 2010
179
0
18,690
Unfortunately, the problem is not NEARLY that simple.

Basically, DevonThink works ok (import OR Indexing data) so long as several things are true:

1. You only have one local machine where you store ALL your data (and can thus import/index it into DevonThink)

OR

2. You only use DevonThink to manage specific projects where you can move all the relevant data directly into it.

--

My problem is I have multiple machines (all Windows, which isn't the problem but does add an annoying layer of complexity), where I store data, and I WANT local copies of that data on the local machine.

e.g. I have a machine setup for photoshop - I want the photoshop images kept on THAT local machine.

I ALSO want that data in DevonThink however so DevonThink can function like a huge META organizer/index of ALL my worldly data.

This has proven nigh impossible to set up though with DevonThink because of design constraints.

A. I tried centralizing ALL my data in an NAS repository (through sync software, I have all the data copied to a giant folder on the NAS) - I wanted DevonThink to then operate with this folder as its media, thus allowing me to have copies of all my data on all machines in a unified folder which updates when new data is added, AND which DevonThink can access.

B. This failed for numerous reasons - DevonThink has poor data-entry design, where it maintains NO connection of knowledge of the filesystem from which the data comes. As a result, every time it attempts to update against the repository, it copies the ENTIRE thing, creating duplicates of the entire database - NOT a solution.

C. It also failed because DevonThink cannot update any files that are in FOLDERS - so I had to get a sync script that STRIPS the folder structure of my entire database and leaves FLAT files.

D. It also failed because DevonThink has problems with certain filetypes - I don't know why, as its listed as being compatible with almost all files.

E. Finally, it also failed because it is unable to detect when NEW files are added to the central data folder, so new files are not added to DevonThink's database.

--

Basically, the simplest answer is - DevonThink is NOT what I want, despite seeming to be that on the surface.

I am looking for basically a Database Manager which can OVERLAY on top of an OS filesystem, and perform some of DevonThink's search, meta-tagging, reorganization, and other features.

DevonThink is designed as a CLOSED software BOX where you can put things into it and use the features on them, but that's it - it maintains its own SEPARATE filesystem basically - I have found this to be NOT what I want, despite my attempts to rig up some sort of complex data system.
--
Just briefly - Indexing doesn't solve any of these problems because, 1. Indexing files on an NAS is tremendously problematic for DevonThink as it cannot always see the files depending on NAS availability.
2. The same problems involving files in folders still occurs - all data needs to be in FLAT file format (a giant flat folder with 15,000 data-types isn't especially wieldy)
3. Updating the database doesn't work - when a new file is created on Machine XYZ, the sync software WILL send it in flat format into the giant NAS repository, but DevonThink Will NOT see it.
--
I have tried using Hazel Automator, various Sync softwares, etc. I'm fairly confident at this point that this cannot be done - and I have good reason to want it to work because I already purchased the expensive DevonThink...

--
I am hoping someone has some advice for how to accomplish what I want, such as other database software I might be able to use.

Thanks!
 

TLK3

Reputable
Jun 21, 2014
1
0
4,510



I know this is an old thread, but I thought I'd reply for anyone else who looks here.

What you claim above is simply not the case. DEVONthink (since v2.5) has included a sync option for Dropbox, local LAN storage, USB drives, etc. Any form of storage whether local or cloud-based is usable.




DEVONthink creates its own directory structure for storing files. When you import, files are moved into these directories - typically a single character (a..z, 0..9) under a higher level directory. This is a "unified" folder. No, you can't dump files directory into here, but you don't need to. It handles placement by itself and you organize (sort, label, tag, etc.) through DEVONthink's interface.




After importing files, you have DEVONthink delete them. If you wish to maintain you own "master" repository, then use an Inbox folder for files that have not yet been added to DEVONthink and import from there. Simple.




You absolutely can import files/folders. And yes, it does work with virtually every type of file. Again, not sure why you'd be having trouble with these features. Have you contacted their tech support?




I'm not 100% certain DEVONthink will do automatic importing of files from a predefined directory. But it's likely there is a way. Since I manually import as needed from my "inbox" / dump folder I don't worry about this issue. Primarily because I sort after importing, so automated importing does me no good.




You should have put this at the TOP of the post. This is the first time I've seen you make any reference to keeping files in YOUR OWN folder structure. DEVONthink just copies them to its own. If you're using DEVONthink to search/access files, who cares what the underlying directories look like?



I think I finally understand your gripe about DEVONthink. It won't let you maintain YOUR files at the filesystem level in YOUR directory structure, and use THEIR database to tag/label/index these files. Is that your issue?
 
Status
Not open for further replies.