[Help wanted] Some tests are failing

[migperfer]

Hi! As discussed in #123 some tests are failing. It is for now unclear what could lead to these errors.

So far the tests that are failing are:

• CFP

  • cfp_new
  • cfp_original

• CQT

  • test_cqt_1992_v2_linear (phase differs from reference)
  • test_cqt_1992_v2_log (phase differs from reference)

• STFT

  • stft_complex
  • stft_magnitude
  • stft_phase

• VQT

  • test_vqt

These are failing at least for CPU. If someone with access to a GPU wants to collaborate please feel free to post the results of the tests for the GPU.

[migperfer]

test_CQT_1992_v2

I've been trying to track which function might the one breaking these tests, and I'm ultra confused now. I rewrote the test a bit, so only the complex CQT ground truth is loaded; and magnitude and phase ground truths are extracted from there.

I'm even more confused now:

@pytest.mark.parametrize("device", [*device_args])
def test_cqt_1992_v2_linear(device):
    # Linear sweep case
    fs = 44100
    t = 1
    f0 = 55
    f1 = 22050
    s = np.linspace(0, t, fs * t)
    x = chirp(s, f0, 1, f1, method="linear")
    x = x.astype(dtype=np.float32)


    # Get the complex ground truth for CQT1992v2, magnitude and phase can be derived from it
    complex_ground_truth = np.load(
        os.path.join(
            dir_path, "ground-truths/linear-sweep-cqt-1992-complex-ground-truth.npy"
        )
    )
    ground_truth_complex_real = complex_ground_truth[..., 0]
    ground_truth_complex_img = complex_ground_truth[..., 1]

    magnitude_ground_truth = np.log(np.sqrt(np.power(ground_truth_complex_real, 2) + np.power(ground_truth_complex_img, 2)) + 1e-5)
    
    phase_ground_truth_atan2 = np.arctan2(ground_truth_complex_img, ground_truth_complex_real)
    phase_ground_truth_real = np.cos(phase_ground_truth_atan2)
    phase_ground_truth_img = np.sin(phase_ground_truth_atan2)
    phase_ground_truth = np.stack([phase_ground_truth_real, phase_ground_truth_img], axis=-1)

    # Magnitude
    stft = CQT1992v2(
        sr=fs, fmin=55, output_format="Magnitude", n_bins=207, bins_per_octave=24
    ).to(device)
    X = stft(torch.tensor(x, device=device).unsqueeze(0))
    X = torch.log(X + 1e-5)
    assert np.allclose(X.cpu(), magnitude_ground_truth, rtol=1e-3, atol=1e-3)

    # Complex
    stft = CQT1992v2(
        sr=fs, fmin=55, output_format="Complex", n_bins=207, bins_per_octave=24
    ).to(device)
    X = stft(torch.tensor(x, device=device).unsqueeze(0))
    assert np.allclose(X.cpu(), complex_ground_truth, rtol=1e-3, atol=1e-3)
    assert np.allclose(np.arctan2(X[..., 1].cpu(), X[..., 0].cpu()), phase_ground_truth_atan2, rtol=1e-3, atol=1e-3)
    assert np.allclose(torch.atan2(X[..., 1].cpu(), X[..., 0].cpu()), phase_ground_truth_atan2, rtol=1e-3, atol=1e-3)

    # Phase
    stft = CQT1992v2(
        sr=fs, fmin=55, output_format="Phase", n_bins=207, bins_per_octave=24
    ).to(device)
    X = stft(torch.tensor(x, device=device).unsqueeze(0))

    assert np.allclose(X.cpu(), phase_ground_truth, rtol=1e-3, atol=1e-3)

This is quite surprising, as the complex CQT seems to be extracted correctly. Am I doing something wrong when calculating the phase?
How can it be possible that not even np.arctan2 gets the same result? (Look at second assertion of the # Complex section). But what is surprising to me is that it fails only on the CPU; I'm testing this on a MacBook with MPS, and when I use MPS the test pass perfectly.

[KinWaiCheuk]

Thank you @migperfer for the investigation.
May I know which pytorch and numpy version are you using?
I tried torch 1.8.1 and numpy=1.19.5. It seems the results for CQT are all fine now. So I believe that it is related to the notorious floating-point error. Can you try adjusting the tolerance a bit such as rtol=1e-2 and atol=1e-2?

I have attached the full unit test result below
unit_test_report.txt

There are still problems for CFP, STFT, and VQT. I am investigating the issues now.

[KinWaiCheuk]

I managed to resolve all failed test cases. Turns out it's all floating-point error. I make the conditions less strict and all tests pass now.

[migperfer]

They are working indeed!

All tests are working for me except the CQT ones.
I think there could be a potential issue with MacBooks. I'm using a MacBook, phase is still wrong even with tolerances 1e-1.

I see that #126 do not change at all the CQT and still is working on Github actions (I Imagine they run on linux), so I'd say the Macbook case a very nitpick one.

[KinWaiCheuk]

Are you using a MacBook with M1/M2 processor? Not sure if it induces extra problems.
Well I would say floating-point error is really a big headache.

[migperfer]

An M2 indeed. And yes, it is a headache 😂

[dorienh]

The failing tests were a result of nnAudio falling behind modern PyTorch, NumPy, and SciPy. These have all been fixed in nnAudio2, which has a rebuilt test suite that passes cleanly on current dependencies: https://github.com/AMAAI-Lab/nnAudio2

pip install nnaudio2