Question - Plans to implement algorithm to search for consensus motifs

[refactoriel]

I'm a PhD candidate in Neuroscience and would like to use this toolbox on behavioural data.

I have velocity time-series of several hundred trials in which a subject moves to one of two targets, dependent on varying cues presented throughout the trial. My goal is to find repeated motifs (not necessarily present in all trials) and see if these motifs correlate with simultaneously recorded brain activity patterns.

One idea was to find consensus motifs like in this paper. As far as I can see there is yet no way to find consensus motifs using stumpy. Do you plan on adding something like that?

I'm not sure if the search for a consensus motif across trials would be the best approach for my data, as I don't expect to find a common motif in every single trial. What I need is some kind of ranking for motifs from most common to less common. I guess I would obtain that by computing the matrix profile and iteratively "ascending" the distances (leaving out the trivial matches close to the local minima). Does that make sense?

Many thanks for this great toolbox, I think it will be really useful for my research!

[seanlaw]

@DanBenHa Thank you for your question and welcome to the STUMPY community! In short, we do not currently have any implementation for finding time series consensus motifs but I have added this feature request as I think it is certainly worth having. Perhaps, this is something that we can work together on and that you might consider contributing to STUMPY (I can definitely assist you where you may need help)?

I'm not sure if the search for a consensus motif across trials would be the best approach for my data, as I don't expect to find a common motif in every single trial. What I need is some kind of ranking for motifs from most common to less common. I guess I would obtain that by computing the matrix profile and iteratively "ascending" the distances (leaving out the trivial matches close to the local minima). Does that make sense?

Yes, this definitely makes sense. If I could make a suggestion, maybe you can:

1. Append a np.nan value to the end of each of your time series (this will help to clearly denote the end of your time series - see note below)
2. Take all of your time series (with np.nan appended) and concatenate them all together into one long time series
3. Compute the matrix profile for this concatenated time series (this would be a self-join where you'd ignore the trivial matches as you pointed out) and sort the discovered motifs from smallest value to largest value
4. Then, for each motif, you'd perform a targeted search for that specific pattern by computing a "distance profile" for that pattern. You'll need to decide on a reasonable threshold for what constitutes as a valid "match" for your motif pattern and then you'd either count the total number of times the pattern is matched. A motif could potentially match multiple times within the same time series or across different time series so you'll have to do a little bookkeepping (to see if ) here but it shouldn't be too much work - but, depending on your skillset, please do not hesitate to ask if you need to bounce ideas off of somebody.

Since you aren't certain that a motif will necessarily reside in ALL time series then the above may be a "better" approach but you'll need to test and learn.

A note about appending np.nan: In STUMPY, any subsequence that contains a np.nan will not return any matrix profile value (for that particular subsequence) so, as a trick, if you concatenate independent time series and separate them with an np.nan then it will not produce any valid matrix profile value at those boundary locations (i.e., you don't have to worry about a "fake" subsequence that straddles two time series coming out as a motif).

Let me know if you have any questions or comments and I'd be happy to clarify!

[seanlaw]

Okay, I see that I've basically restated some of their "brute force" approach in their paper (not exactly) and they have proposed something called "ostinato". My guess is that it shouldn't be too difficult to implement Table 2 in their paper.

[seanlaw]

From the ostinato explanation for Table 2, it isn't clear to me what is meant by and line 5 produces an index map of Tj in order to increase radius lower bound. I can't seem to decipher what is meant by an "index map" and what SI ← sortAndIndex(MP) really means.

However, according to their Matlab code, it appears that this is equivalent to performing an np.argsort in ascending order like:

SI = np.argsort(MP)
for q in SI:
    radius = MP[q]

[refactoriel]

@seanlaw Thanks a lot for your replies and the warm welcome!

What you suggest sounds reasonable. I'm also reading #149 to get inspired - a different domain but probably similar goals. Will let you know if I have new ideas/questions.

I'd certainly be interested in contributing code. At first glance the Ostinato algorithm doesn't look that complicated but I'll have a closer look (that is, if you're not already working on an implementation yourself).

[seanlaw]

What you suggest sounds reasonable. I'm also reading #149 to get inspired - a different domain but probably similar goals. Will let you know if I have new ideas/questions.

Great and good luck!

I'd certainly be interested in contributing code. At first glance the Ostinato algorithm doesn't look that complicated but I'll have a closer look (that is, if you're not already working on an implementation yourself).

Sounds good and, no, I haven't worked on it yet. Let's track it in issue #277. Usually, we recommend contributing a code-only Jupyter notebook first that reproduces one of the primary figures from the paper that you pointed to (no need for text annotations). This way, we can start with a rough prototype with the Ostinato code and build enough confidence that the code reproduces the paper. Let me know how that sounds.

[refactoriel]

Let's track it in issue #277. Usually, we recommend contributing a code-only Jupyter notebook first that reproduces one of the primary figures from the paper that you pointed to (no need for text annotations). This way, we can start with a rough prototype with the Ostinato code and build enough confidence that the code reproduces the paper. Let me know how that sounds.

Sounds good. I'll get started on a notebook and continue the feature-related discussion in #277. Where would I best put the notebook? In my forked repo or somewhere else?

[seanlaw]

I'll get started on a notebook and continue the feature-related discussion in #277. Where would I best put the notebook? In my forked repo or somewhere else?

Good questions! Okay, I think we should be a little more pragmatic and scope it so that it is slightly more manageable. I'd like to separate the "notebook reproducer" (now tracked in #278) from the feature enhancement (#277). First, let's get the "notebook reproducer" completed and iron out all of the kinks and, once that is complete, the feature enhancement should be straightforward (i.e., we'll extract the ostinato function from the notebook, add it to STUMPY, and replace the notebook function with a call to STUMPY).

Please take a look at #278 as I've outlined what is roughly needed for a notebook reproducer. Let me know if you have any questions, comments, or concerns.

[seanlaw]

Closed in #279