`dd` fails on gnu test `tests/dd/direct.sh` - dynamic removal of `O_DIRECT` missing

[cre4ture]

I started fixing remaining issues from the GNU test suite that are related to dd as this was the last util I was working on.

I investigated what the issue is by manual tests and using strace.
Summary:
The test tries to do file copy with the direct input and output mode.
direct mode is special as it has some restrictions regarding the buffer locations and its size:
While a direct read apparently succeds in the case when it reads the last chunk from a file that is irregularily sized,
a direct write fails if we try to write a buffer that doesn't match the blocksize.

I used strace to figure out how GNU is solving it.
Aparently it gets rid of the O_DIRECT status flag before writing the last chunk by using the fcntl system call.

GNU:

write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
read(0, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
read(0, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 511
read(0, "", 512)                        = 0
fcntl(1, F_GETFL)                       = 0xc001 (flags O_WRONLY|O_DIRECT|O_LARGEFILE)
fcntl(1, F_SETFL, O_WRONLY|O_LARGEFILE) = 0
lseek(1, 0, SEEK_CUR)                   = 7680
fadvise64(1, 4096, 0, POSIX_FADV_DONTNEED) = 0
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 511) = 511
fsync(1)                                = 0
fadvise64(1, 4096, 0, POSIX_FADV_DONTNEED) = 0
close(0)                                = 0
close(1)                                = 0
write(2, "15+1 records in\n15+1 records out"..., 3315+1 records in
15+1 records out
) = 33
write(2, "8191 bytes (8.2 kB, 8.0 KiB) cop"..., 598191 bytes (8.2 kB, 8.0 KiB) copied, 0.00712461 s, 1.1 MB/s) = 59
write(2, "\n", 1
)                       = 1
close(2)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

uutils:

write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 511
read(3, "", 512)                        = 0
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 511) = -1 EINVAL (Invalid argument)
close(4)                                = 0
futex(0x5dca26e81db8, FUTEX_WAKE_PRIVATE, 1) = 1
close(3)                                = 0
write(2, "dd", 2dd)                       = 2
write(2, ": ", 2: )                       = 2
write(2, "IO error: Invalid input", 23IO error: Invalid input) = 23
write(2, "\n", 1
)                       = 1
sigaltstack({ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=8192}, NULL) = 0
munmap(0x79831cf07000, 12288)           = 0
exit_group(1)                           = ?
+++ exited with 1 +++

I'm alread working on a solution

[renanrodrigo]

Hello there!
FWIW this issue is affecting the HA resource agents in Ubuntu (as we are moving to uutils coreutils): https://bugs.launchpad.net/ubuntu/+source/resource-agents/+bug/2125968
a) we are having problems with O_DIRECT in reads, and
b) Not sure if it only affects the last chunk in our case because we are reading just a single chunk. But still alignment problems seem to be there.

If you think this is a different issue, I can report it separately.

[cre4ture]

Hi @renanrodrigo,

I think I could spend a bit more time on it continue working on the fix.
I just never contributed via launchpad, so I could use a hint for starting.