Forthcoming dateutil code reorganization

[pganssle]

I know that pandas makes use of some of the private interfaces in dateutil (and I'm hoping to start adding features so that this is no longer necessary), so I thought I'd give you a heads up about a proposed changed to the organization of the parser module that will change the location of some of the private interfaces.

Basically parser.py will move to parser/_parser.py, and only the public parts will be imported into parser/__init__.py, as part of the introduction of a dedicated ISO-8601 parser. See PR dateutil/dateutil#489 for the current discussion.

[TomAugspurger]

Thanks for the heads up! We test against dateutil master in one of our jobs (here), so we should be able to catch these and patch them in pandas quickly.

Do you have a rough plan for the next dateutil release?

[pganssle]

@TomAugspurger I'm trying to finalize some things, I want to get isoparse in there and possibly a few improvements to the time zones. I am hoping a new feature release before the end of the year. I usually finalize a release on master and let it sit for a week or so, I can ping you when that happens so you'll have some notice.

[TomAugspurger]

Thanks. If pandas doesn't have another release in that timeframe I can backport any necessary fixes and to a quick 0.21.1 release.

…

On Mon, Nov 6, 2017 at 10:43 AM, Paul Ganssle ***@***.***> wrote: @TomAugspurger <https://github.com/tomaugspurger> I'm trying to finalize some things, I want to get isoparse in there and possibly a few improvements to the time zones. I am hoping a new feature release before the end of the year. I usually finalize a release on master and let it sit for a week or so, I can ping you when that happens so you'll have some notice. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18141 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIkrd2Jf6fILc5MxUHvvNTNmdlihwks5szzc0gaJpZM4QTdJk> .

[jbrockmendel]

@pganssle I'm -1 on rushing the isoparse. The best-case scenario is that the dateutil implementation can drop-in replace the _string_to_dts pandas currently uses for parsing these formats. Since the current pandas implementation isn't fully spec compliant (as you mentioned in an issue somewhere), it isn't obvious how easy (or feasible) this will be. So yah, advocating patience and thorough in-house testing/profiling.

As for pandas accessing dateutil internals, there are a couple of workarounds pandas uses that we can probably make unnecessary: https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/tslibs/parsing.pyx#L207, https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/tslibs/parsing.pyx#L315

Note the tail end of tslibs.parsing.dateutil_parse is almost identical to _build_tzinfo from dateutil#493.

[pganssle]

I'm not really planning on "rushing" the isoparse implementation, but at the end of the day because it's spec-driven, a lot of the behavior questions are actually made ahead of time (assuming we stick to the spec). The main question then comes down to the public API, and decisions about that can always be pushed off to later by releasing a highly constrained version (e.g. the only supported interface is isoparse(dt_str) - which we know we want to support anyway).

In any case, I'd like some form of it to be in dateutil 2.7.0.

[pganssle]

@jbrockmendel I would also guess that a lot of this stuff will have to be in conditional import logic or something, unfortunately. I suspect it could cause a decent number of problems if pandas bumped their dependency to the latest version of dateutil, since that could cause inconsistencies in pinned versions.

[jreback]

@pganssle if you are moving things to private methods please deprecate first; simply moving will break lots of outstanding installs.

[pganssle]

@jreback I'm not moving things to private methods, I'm moving private functions/classes (i.e. in the physical layout of the code). I've pulled out the reorganization portion of the isoparser PR into dateutil/dateutil#501 so it's easier to see what's happening.

Basically, if you're calling dateutil.parser._tzparser or something, it will have moved, but even stuff that is not in dateutil.parser.__all__ will all still be importable from dateutil.parser if it doesn't start with _

[jreback]

what i am saying is if you are changing your publicly exposed functions you should deprecate

[jbrockmendel]

@jreback pandas should not be affected by the changes pganssle is discussing, with the exception of the private import of _timelex in tslibs.parsing. I'll make sure to patch that try/except import when appropriate.

[jbrockmendel]

I'm not really planning on "rushing" the isoparse implementation

@pganssle it probably only seems rushed to me because I haven't made time to look it over closely. Will do so this evening.

[jreback]

@pganssle we should patch this on the pandas side
but the issue is that the upstream needs to deprecate or completely break existing installations which is not nice

[TomAugspurger]

It's fine for packages to change private methods without a deprecation cycle. Let's see how bad the breakage actually is.

…

On Tue, Nov 7, 2017 at 8:21 AM, Jeff Reback ***@***.***> wrote: @pganssle <https://github.com/pganssle> we should patch this on the pandas side but the issue is that the upstream needs to deprecate or completely break existing installations which is not nice — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18141 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIu9ITCzGPRiA59X9xHivyAgAVxkTks5s0GdsgaJpZM4QTdJk> .

[pganssle]

I'm sure some people will break if they are relying on the private interface, that's a consequence of Python's philosophy of "nothing is really private", but I think there's a better than average chance that anyone who notices / pays attention to DeprecationWarning is also likely to not use private methods in the first place, and if they do use private methods, they're also likely to test against the master of their dependencies (as you guys are doing, presumably at least partially because you've been burned before).

Not saying things won't break out there, just saying that deprecation warnings (which aren't even on by default) probably won't mitigate this by much. In the short term people can pin their dependencies until they fix their fragile code. Worst case scenario, the library is BSD licensed, so there are no viral concerns from vendoring it.

That said, for some of the stuff where I'm actually planning on exposing it publicly anyway (_tzparser, _timelex), it might not be the worst thing in the world to go through a deprecation cycle. Still, the message is weird: "This private method has moved, you shouldn't be using it in the first place, but if you must, pick it up from somewhere else now."