TORCH_CHECK used within torch.compile does not throw legible errors

[lezcano]

🐛 Describe the bug

At best, you get a

terminate called after throwing an instance of 'c10::Error'
terminate called after throwing an instance of 'c10::Error'
terminate called after throwing an instance of 'c10::Error'
terminate called recursively
[1]    2964411 IOT instruction (core dumped)

At worst, you just get the "terminate called recursively" with not mention to c10::Error

To repro, you can perform any out of bounds error. I got it with

import torch

@torch.compile
def copy(a):
    b = torch.arange(start=1, end=-a.numel() - 1, step=-1, device=a.device)
    return a[b]
x = torch.rand(1024, 128, device="cpu")
copy(x)

when testing #114471 (may not repro in master).

Versions

#114471

[lezcano]

cc @jgong5

[lezcano]

It doesn't happen for every input. The following input throws a nice error:

@torch.compile
def fn(x, y):
    b = torch.arange(x.size(0) - 1, device=x.device) + 3
    return x[b] + y[b]
x = torch.rand(1024, 128, device="cpu")
y = torch.rand(1024, 128, device="cpu")
fn(x, y)

Note that the indexing error in the OP is much more egregious than the one in this second example.

[jgong5]

when testing #114471 (may not repro in master).

How to reproduce it? It cannot be repro on master.

[lezcano]

Patch in that PR. There are quite a few issues in master when it comes to issuing device_asserts. See that in the repro in the OP we don't even generate a TORCH_CHECK and rather we just read out of bounds.

[zhuhaozhe]

I have compared the difference with ATen ops, we need to change 2 things to throw as legible errors as aten does here:

• Inductor path dose not translate c++ exception into python like aten
• Inductor path do not have logic to catch exception inside omp paralell region, while at::parallel_for can catch and re throw it. The difference behavior of difference inputs is also because 1 will encounter omp parallel and one is not.

I will submit a PR for it.

[zhuhaozhe]

Submitted a PR here. #127868