MP3 byte-for-byte audio content comparison - issues

[GC_mp3]

Hello everyone:

I have been a heavy user of AllDup for my mp3 collection over a few years - and a muliple time financial donator, since there's no other program like it. AllDup can ignore tags and focus on comparing byte-for-byte of the audio content, which is exceptional.

Here's the issue - and I hope that the software developer(s) can chime in.

I've used AllDup exclusively on a byte-for-byte audio comparison for tens of thousands of mp3 files, but I came across multiple pairs that I know are byte-for-byte audio content identical, because they're the only copies that I have, yet, AllDupe did NOT mark them as duplicates.

I have the options set in AllDupe to ignore all tags and to scan the entire file. I suspected that maybe replay gain or mp3gain tags caused the difference between the files and that AllDupe did not see them as identical duplictes, but that was not the case.

I added both mp3 audio files into ChatGPT on "pro" settings as well as Claude Fable, max effort. ChatGPT was the most throrough at analysis, by the way.

What ChatGPT and Claude relayed is that both files have identical byte-for-byte audio content (as I suspected) and decoded both files with identical audio streams but one file of the pair had a single Xing/Info bookeeping/LAME header frame at the start. This caused the difference between the two files, yet, the byte-for-byte audio for both files was correct. Both AI tools said, explicitly and confidently that AllDupe's algorithm is exact.

ChatGPT put it this way:

The complete MPEG stream after removing ID3 tags is not byte-for-byte identical, because the first MPEG frame contains differing Info/LAME bookkeeping data. Yet, the actual byte-for-byte audio content is exactly identical after ignoring the Xing/Info bookeeping frame.


REQUEST:

In addition to allowing AllDupe to ignore ID3 tags and meta with the settings in the file content "search method," also show settings to

[Pasted from ChatGPT] (respectfully submitting to the developer here because this AllDup program has been so helpful to me)

Exclude the complete initial MPEG frame when that frame is positively identified as a Xing, Info, Xing/LAME, Info/LAME, or VBRI header frame. Then compare or hash only the subsequent MPEG audio frames.

It is important to exclude the entire recognized header frame, not merely:

the 100-byte seek table;
the four-byte Xing or Info identifier;
the LAME3.99r string;
or selected fields inside the frame.

In your two files, multiple fields within that same first frame differ. Ignoring only the seek table would still leave other differences that cause AllDup to reject the files.

Ready-to-send feature request

You can send the developer this:

Feature request: Ignore MP3 Xing/Info/LAME/VBRI header frames during audio-content comparison

AllDup’s “Ignore ID3 tags” option currently excludes ID3v1, ID3v2, Lyrics3 v2, and APEv2 metadata, but it still compares the first MPEG frame when that frame contains a Xing/Info/LAME header.

Please add an option such as:

“Compare encoded MP3 audio frames only — ignore Xing/Info/LAME/VBRI header frames.”

Suggested behavior:

Skip recognized leading metadata and locate the first valid MPEG audio frame.
Parse the first MPEG Layer III frame.
Check for a valid Xing or Info structure at the standard location after the MPEG side-information area, or for a valid VBRI structure at its standard location.
When one of those structures is positively identified, exclude the entire first MPEG frame, using the frame length calculated from its MPEG header.
Exclude the optional LAME extension along with that frame because the LAME information is embedded inside the Xing/Info frame.
Compare or hash every subsequent complete MPEG audio frame byte-for-byte.
Continue excluding recognized trailing ID3v1, APE, and Lyrics metadata.
Do not skip the first frame unconditionally. Skip it only when a structurally valid Xing, Info, or VBRI header is found.
Do not find these headers by searching arbitrarily for the text Xing, Info, LAME, or VBRI. Validate the MPEG frame and inspect the format-defined offsets.
Use the resulting normalized audio-frame length and checksum when grouping duplicate candidates, rather than requiring the complete files to have the same physical size.

FFmpeg’s MP3 demuxer already follows this general approach: it detects Xing/Info and VBRI structures and then seeks past the complete VBR-header frame before reading the normal MP3 packets.

Ideally, AllDup could report these as:

“Identical encoded audio frames; different MP3 stream header or metadata.”



THANK YOU!