Parallel Operations

English support for the software AllDup
Post Reply
GMS
Posts: 3
Joined: 21 May 2020, 03:48

Parallel Operations

Post by GMS » 22 May 2020, 05:07

Have you considered reading files in parallel and calculating hashes in parallel? It would really speed things up with multiple CPUs and separate drives.

User avatar
Administrator
Site Admin
Posts: 3529
Joined: 04 Oct 2004, 18:38
Location: Thailand
Contact:

Re: Parallel Operations

Post by Administrator » 22 May 2020, 09:34

Yes, we already thought about that but the process to convert the current search engine to work with threads is very time consuming...maybe somewhere in the future...

mz1888
Posts: 25
Joined: 24 Jul 2019, 14:00

Re: Parallel Operations

Post by mz1888 » 26 May 2020, 01:47

From the users' point of view, parallelism is great but not the only a small part of performance boost:

1. Ability to refresh a search. This would eliminate the re-comparison operations on the whole underlying set. Sometimes it's just faster to manually change the directories directly without going through AllDup but AllDup would still show stale results. To show the fresh result on the same inputs, users have to reconduct search.

2.. Ability to conduct search on "parameter delta". For instance, users tend to go from 100%, 99%....80% similarity. Currently, there's no way to schedule or conduct a subtraction of 99% - 100%, 98%-99%...80%-81%, or make the results group by % similarity bucket. The user has to search 20 times.

3. Ability to "cache" comparisons to reduce the total numbers of file comparison performed. Some files are already compared and yielded low % similar. By saving/indexing that info, users trade space for speed, and speed increases with each incremental search.

Lastly, many are long time users of AllDup who are only concerned about incoming files by now. It would further cut search time if there's a custom schedule search to compare just those incoming files.

Post Reply