Parsed!

The callback worked! The code previously posted only had to be slightly modified.

parsedates.set_callback(timeseries_parse.DateTimeFromString)

I have the mxDateTime Parser modified from the Scikits Timeseries imported here as timeseries_parse. In that program is a magical date parsing function called DateTimeFromString (and a similar DateFromString) which takes a string and returns a Python datetime object filled with the correct date amounts.

parsedates.parse_date("01/01/2001")
datetime.datetime(2001, 1, 1, 0, 0)

So here we see a datetime object with (Year, Month, Day, Hour, Minute, Second). Turning that into a long number is very easy and it all depends on the frequency metadata. If our frequency is in years, then our number is (Year - 1970) . We can convert this data into a long value very easily.

Callbacks

Sometimes you need to run a Python code segment from C in your module. There's a lot of good reasons to do this. C doesn't have much support for writing regular expressions, while Python is pretty robust. You can take a PyObject with a C string and send it to a Python parsing function. When the Python code is done manipulating the PyObject, you have it back safe and sound in C.

Writing callback functions is pretty easy.

static PyObject *callback = NULL;

static PyObject *
set_callback(PyObject *dummy, PyObject *args)
{
PyObject *result = NULL;
PyObject *temp;

if (PyArg_ParseTuple(args, "O:set_callback", &temp))
{
if (!PyCallable_Check(temp))
{
PyErr_SetString(PyExc_TypeError, "parameter must be callable");
return NULL;
}
// Reference to new callback
Py_XINCREF(temp);
// Dispose of previous callback
Py_XDECREF(callback);
// Remember new callback
callback = temp;
// Boilerplate to return "None"
Py_INCREF(Py_None);
result = Py_None;
}
return result;
}

This function takes a dummy object and some arguments. It stores a callable function into a global PyObject named callback. You can later use this global PyObject with the callback function stored in it like this:

result = PyEval_CallObject(callback, arglist);

Result is a PyObject (pointer) with the result of the callback function stored in it. Here's a pretty simple example:

def add_ftn(a,b):
return a + b

parsedates.set_callback(add_ftn)

We set the callback PyObject to store the callable Python function add_ftn. We can test the callback by running the code above with the PyEval_CallObject(callback, arglist). This C method will take arguments (for add_ftn, we need two) and send them to the callback function stored in the global PyObject variable callback.

Enthought and Other Developments

In the words of my Mentor, Pierre,

Enthought is a private company based in Austin, TX, founded by Eric Jones, a long-time Pythonista. Enthought's main source of revenue is the programming of specific scientific application.

Enthought recently had a client request a datetime type exactly like the type I've been working on. Enthought is a prominent contributer to NumPy. Travis Oliphant will be working on the new datetime dtype, himself. This is the guy who literally wrote the book on NumPy. And I get the privelage of assisting him.

This is really a godsend, since my knowledge of low level NumPy is quite sparse. This is the nature of open source, it would seem: collaboration between the ignorant and learning (me) and the experienced brilliance which created the foundations and core of these massive projects.

I've been commissioned to write two sets of code. The first set is to get and set datetime members of the narrays. The second will be incorporating the mxDateTime Parsing module for strings to datetime conversions.

More on those later.

A Sparse Documentation

A bit of a frustrating last couple of days. I've been all over the place with my research, which means I kept getting lost and confused. First I tried to figure out how to incorporate a scalar datatype into a NumPy dtype. Then I got lost trying to learn some more advanced Python C-API on object handling. I couldn't figure out quite how to make my datetime object play nice with frequencies. So, off I went venturing into the land of the NumPy C-API.

The documenation is very well written, but not exactly geared towards my project. I read, "The best way to truly understand the C-API is to read the source code." So I ran right to the source. The last few days have been a marathon of running over code and trying to understand how everything fits together.

Communication is key. Yeah, we say that for relationships and other insignificant things, but I'm talking about source code. The NumPy source code has a significant amount of C code, which can be overwhelming at times. I've decided the only way to understand what's going is to document it for myself. I've been literally running through each significant file and using pen and paper to pin down exactly what's going on. I'm interested in anything related to the Array Scalar Types, specifically the LongLong type. Both the datetime and timedelta types will very similar to the LongLong type. There are key differences, which I guess I should talk about. But I'll save that for tomorrow. More on the NumPy source code:

There are some fancy repetition techniques employed in the definitions of these scalar types. Commented out above each "generic" method is a comma separated list of each scalar type which. The "generic" methods have a variable (macro?) to replace the name and repeat for each scalar type in the comma separated values. I know how it works, but I don't know why.

There's been a very important update on my project. More on that later this week.

Scalar Objects VS dtype

Just a clarification:

When I store an object from a narray with dtype (for example) float64, this means the stored variable is of scalar object of float64's .type attribute.

>>> somearray = numpy.array([1,2,3,4,5], dtype='float64')
>>> somefloat = somearray[0]
>>> type(somefloat)
<type 'numpy.float64'>



So numpy.float64 is a scalar type. The dtype is only there to tell the numpy array how to handle the data in the array.

dtypes

A NumPy dtype is not a normal Python object. Dtypes tell the narray how to interpret the array. A dtype is a way to specify exactly what every member in the narray is.

>>> numpy.array([1,2,3,4,5], dtype='int32')
array([1, 2, 3, 4, 5], dtype=int32)

The NumPy array is able to take a list of data [1,2,3,4,5] and a dtype to refer to that data (dtype='int32'). When I create this narray, the dtype='int32' makes the list of data be interpretted as a list of 32 bit integers. See what happens when I change the dtype to a float:

>>> numpy.array([1,2,3,4,5], dtype='float64')
array([ 1., 2., 3., 4., 5.])

The data inside of the narray is now interpreted as an array of 64 bit floats. My goal is to make a new one of these dtypes.

>>> numpy.array(["12-3-2009"], dtype='datetime64[D]')
array([12-3-2009])

I've been working on creating some kind of separate module with datetime64 as a scalar object type. This is not the project goal. I need to create a numpy array dtype datetime64 and timedelta64 for use in the narrays. I've been sifting through NumPy's core code all weekend and can't seem to find the file(s) where dtypes are referred to. My current plan is to take these already created dtypes' chunk of code and copy paste so I can start with something barebones and work my way up

Parsing

Yuck. I've hit a wall and it hurts.


if (! PyArg_ParseTupleAndKeywords(args, kwds, "OO", kwlist, &obj_time, &obj_freq))


This line of code should take the args sent to the function, use whatever kwds were supplied to identify those arguments, and parse them as two PyObjects. The kwlist is used to identify which kwds refer to which recipients of the PyObject variables.


static char* kwlist[] = {"time", "freq", NULL};


Let's give it this input.


>>> print d.datetime64(time='1', freq='1')


We've created a datetime64 object and (allegedly) given it values 1 for both time and freq. This should parse so that we create two PyObjects with values '1' and the different kwds ("time" and "freq"). I should be able to extract those values by simply checking the appropriate PyObjects for their respective keywords. But, alas, if life were easy, it would be boring.


if (PyObject_HasAttrString(obj_time, "time"))


This resolves to false. Why? This is the exact same implementation as the TimeSeries Date type. I mean, I almost copied this. I don't understand why the arguments are being completely discraded. I can very easily parse to something else, like longs or ints, but that wouldn't that just be a workaround? Maybe not...

I'll be trying to parse:


if (! PyArg_ParseTupleAndKeywords(args, kwds, "iL", kwlist, freq, time))


Where do the keywords go, now? I assumed they were placed into some kind of magical PyObject slot. But since I'm parsing to an int and a long long int ("iL"), I wonder what happens to the keywords?

Yes, I've been neglecting writing my Unit Tests, I know. But this is just so darned frustrating. Whether or not the Unit Tests even exist, I need to be able to create datetime64 types with appropriate values. I need to be able to comfortably be able to make datetime64 types with different values before doing anything fancy.

This is important, I promise. Now, off to parse.