Datestring

Well, this is annoying (but complete, at least). I've been able to convert longs to Python DateTime Objects based on a frequency, but what about outputting them? It would be nice to have a function that could print out the date in a readable format, no? Unfortunately, there's a few problems with this seemingly trivial process.

First off, there's a lot of different ways to print the date. Do we print the weekday? If so, where do we print it? Do we print months first, or years first, or days first? I realize most of this has become "standardized" (whatever that means these days...) but with so many possibilities, it's difficult to create a "perfect" format that could satisfy anyone.

The second (real) problem is that PyStrings are horrible with formatting. There's a function in the Python C - API called PyString_FromFormat() which works very similarly to printf(). It does not, unfortunately, work similarly enough. PyString_FromFormat() completely ignores any minimum length formats and precisions placed on the input data. If I want to print a YYYY-MM-DD string format with exactly four numbers for years, 2 for months, and 2 for days, I can't do that without formatting the strings myself (in C). Is this catastrophic? No. But it's certainly annoying...

And to finish off, a quick aside: standardizing years at 4 digits is a bad idea. What about years post 9999? Maybe someone in some scientific research lab is going to grow really old (cryogenic technology is advancing fairly steadily, after all...). But years below 1000 look... awkward in a YYYY-MM-DD format (which is the current one the tests are written for...).

Oh, and Python DateTime has a printing method of it's own:

>>> print datetime.datetime.now()
2009-07-25 03:55:31.884781
>>> print datetime.datetime(999,1,1)
0999-01-01 00:00:00
>>> print datetime.datetime(10001,1,1)
Traceback (most recent call last):
File "", line 1, in
ValueError: year is out of range

See? They standardize at YYYY and error out above the year 9999... but of course none of this is available on the C end (as far as I can tell), so this is of little use to me... Besides, NumPy should be able to handle dates above 9999. It's the principle of the thing.

Also, sorry to Matt Knox about not reading your comment until now. That calculation would have saved me a lot of time (and blood pressure). I didn't have the blog set up to email me when people commented. I do now. Thank you, again.

Long to Datetime

I'm absolutely stuck on this one calculation. Everything else is working perfectly. I can't figure out how to calculate the calendar given a long "number of business days" since 1970. The funny thing is, I figured it out fine going the other way (that is, datetime to long). Something is fishy in my calculations (I've tested the test numbers again and again).

Also, since you suggested using smaller structs (since femtosecond has no need of knowing month, etc), I started using ymdstruct (year, month, day structs) and hmsstruct (hour, minute, second structs). Send a long "number of days" to long_to_ymdstruct() or "number of seconds" to long_to_hmsstruct() and the function will return the appropriate struct. So to calculate a calendar date, it's just a matter of converting given frequency to days or seconds (depending on precision of frequencies... for the Business Day case, we need to convert to days by using the ymdstruct).

I may be overly complicating things. I'm not totally confident in the efficiency of structs in C (I know you should try to have structs contain a base two number of items so the compiler can do a cheap shift instead of an expensive multiplication to find members). Unfortunately, both ymdstruct and hmsstruct seem like unnecessary baggage for the long_to_datestruct function, since this function returns neither a ymdstruct nor an hmsstruct (it returns a datestruct, which is a seperate struct entirely...). There really has to be a better way of doing this... I just haven't thought of it yet...

But regardless, I still can't figure out these business day calculations. The most annoying part of the entire endeavor is that January 1, 1970 is on a Thursday... So I know I have to add some offset or subtract some other offset... but what are they? The closest calculation I could try to find (remember, I'm only trying to turn business days into absolute days to fill my ymdstruct):

absdays = 7 * (dlong / 5) + dlong % 5

where dlong is the long long value representing a date and absdays are the absolute number of days (since Jan 1, 1970, specifically).

Git

Sorry for the blogless time lapse. I needed a little push to pump out some code. Check it out for yourself:

Clone URL: git://github.com/martyfuhry/npy_datetime.git

The important code is in the parsing directory (you'll need to navigate there and run the setup.py build and install). The instructions in the readme are a little antiquated, but the tests directory (in the parsing folder) should clear things up.

From Here On Out, It's Math

The code is coming along nicely.

>>> p.parse_date("01/01/1980", "Y")
(10L, 9L)

Real simple. First, we import the parsedates module, then we set the callback function to the mxDateTime Parser. The parse_date function takes a String of a date, which it passes to the mxDateTime Parsing function and a String for a frequency, which it converts into an int (to be stored internally). The Parsing function for the date returns a Python DateTime object (which, for my use, is basically a tuple filled with (year, month, day, hour, minute, second, etc.). I can extract this and pass all of these into a master function to calculate the date. The frequency is taken from the second String (and proper error messages are awarded in the event of a bizzare frequency) and stored internally (as an int for now).

A few things will need to be changed, though...

The frequency will need to be parsed, too, though, since Travis needs to support "custom" frequencies (read more about them here). Perhaps this will call for a second Python callback function, as parsing Strings with regular expressions is relatively easy in Python and difficult at best in C.

The mxDateTime parser returns the Python datetime object, which only supports time units up to the nanosecond (if I recall correctly...). The NumPy DateTime module must support units as high as femtoseconds. Hopefully this will be doable with just a couple of lines of code added to the parser.

Parsed!

The callback worked! The code previously posted only had to be slightly modified.

parsedates.set_callback(timeseries_parse.DateTimeFromString)

I have the mxDateTime Parser modified from the Scikits Timeseries imported here as timeseries_parse. In that program is a magical date parsing function called DateTimeFromString (and a similar DateFromString) which takes a string and returns a Python datetime object filled with the correct date amounts.

parsedates.parse_date("01/01/2001")
datetime.datetime(2001, 1, 1, 0, 0)

So here we see a datetime object with (Year, Month, Day, Hour, Minute, Second). Turning that into a long number is very easy and it all depends on the frequency metadata. If our frequency is in years, then our number is (Year - 1970) . We can convert this data into a long value very easily.

Callbacks

Sometimes you need to run a Python code segment from C in your module. There's a lot of good reasons to do this. C doesn't have much support for writing regular expressions, while Python is pretty robust. You can take a PyObject with a C string and send it to a Python parsing function. When the Python code is done manipulating the PyObject, you have it back safe and sound in C.

Writing callback functions is pretty easy.

static PyObject *callback = NULL;

static PyObject *
set_callback(PyObject *dummy, PyObject *args)
{
PyObject *result = NULL;
PyObject *temp;

if (PyArg_ParseTuple(args, "O:set_callback", &temp))
{
if (!PyCallable_Check(temp))
{
PyErr_SetString(PyExc_TypeError, "parameter must be callable");
return NULL;
}
// Reference to new callback
Py_XINCREF(temp);
// Dispose of previous callback
Py_XDECREF(callback);
// Remember new callback
callback = temp;
// Boilerplate to return "None"
Py_INCREF(Py_None);
result = Py_None;
}
return result;
}

This function takes a dummy object and some arguments. It stores a callable function into a global PyObject named callback. You can later use this global PyObject with the callback function stored in it like this:

result = PyEval_CallObject(callback, arglist);

Result is a PyObject (pointer) with the result of the callback function stored in it. Here's a pretty simple example:

def add_ftn(a,b):
return a + b

parsedates.set_callback(add_ftn)

We set the callback PyObject to store the callable Python function add_ftn. We can test the callback by running the code above with the PyEval_CallObject(callback, arglist). This C method will take arguments (for add_ftn, we need two) and send them to the callback function stored in the global PyObject variable callback.

Enthought and Other Developments

In the words of my Mentor, Pierre,

Enthought is a private company based in Austin, TX, founded by Eric Jones, a long-time Pythonista. Enthought's main source of revenue is the programming of specific scientific application.

Enthought recently had a client request a datetime type exactly like the type I've been working on. Enthought is a prominent contributer to NumPy. Travis Oliphant will be working on the new datetime dtype, himself. This is the guy who literally wrote the book on NumPy. And I get the privelage of assisting him.

This is really a godsend, since my knowledge of low level NumPy is quite sparse. This is the nature of open source, it would seem: collaboration between the ignorant and learning (me) and the experienced brilliance which created the foundations and core of these massive projects.

I've been commissioned to write two sets of code. The first set is to get and set datetime members of the narrays. The second will be incorporating the mxDateTime Parsing module for strings to datetime conversions.

More on those later.