`select()` silently drops hidden fields such as `etag`

[dmpetrov]

Description

etag is in File._hidden_fields (src/datachain/lib/file.py:244-251), so a request for it is
dropped rather than refused:

chain.select("scenario.frag.video.etag", "scenario.predicate")   # etag column simply absent

For lineage work, where the etag is the entire point, a silent omission is worse than an error. The
workaround is mutate(), which is not discoverable from the failure since there is no failure.

Version Info

[modelpath-dev]

I will take this issue. Please assign it to me.

The problem seems to be that select() is not handling hidden fields like etag correctly. I will first look at the select() function in src/datachain/lib/file.py to see how it processes fields. My initial fix would involve modifying the logic to either include hidden fields when explicitly requested or raise an error if they cannot be selected. I will test the fix by ensuring etag is either included or an error is raised when requested.

[modelpath-dev]

I am looking into the select() function to address the issue with hidden fields like etag. I will update the logic to handle these fields correctly. If anyone has suggestions or insights on this, feel free to share.