I've got a sticky wicket here. On

macosx

I used the FreeBSD tool

paste

to build a command.

paste

, unfortunately, did not implement

macOSX

policy regarding carriage returns, which apparently

macOSX

was using on the file. As a result,

paste

injected

\r

characters (

0x0D

) that even

vim

was insensitive to (although

tr

was, so it does appear the file was valid, in some sense), and which bypassed python sanitization to be deposited directly in the datalake unescaped, in raw form. When querying these files via the

lakectl

tool, they are not accessible. A

lakectl fs ls

chokes on the result, yielding a corrupted `common_prefix`:

common_prefix                                                    1536.1350/
/ommon_prefix                                                    1539.1980

-- here you can see the `\r`s are not handled. As a result I can neither rename nor access the files. Has anyone encountered this problem, and if you have, is there a solution?

Ooch… Sounds annoying. I’ve never run into that before, so what I’m suggesting is a bit theoretical. Maybe try using

lakectl local

to clone the data locally, run a script to sanitize and rename the files, and then commit the changes back? I’d definitely recommend doing this in a separate branch first, just to be safe.

ok that is a good idea, I'll give it a shot and see if it works.

It will be extremely beneficial to open a bug on this and provide all details, especially the reproduction scenario so we could prioritize and resolve this

@Iddo Avneri that way would work. The files come back:

xyz$'\r' # where 'xyz' is the name they should be

You were able to recover?

Here's another way.

lakefs://...

URLs really are URLs, so you can encode a carriage return in them as

%0D

. I would try e.g.

lakectl fs rm '<lakefs://my-repo/branch/common_prefix%0D1536.1350/>'

. I would hesitate to add another encoding for doing this, as it would be nonstandard. Would a better error message have helped? The current error message is:

path URI malformed lakefs URI
Error executing command.

@Chris - trying to close the loop on this. Are you good to go?

yes, thank you! Your method worked