Data normalisation

[Tilanthi]

How would you recommend dealing with time series where the overall amplitude is increasing, even though the underlying pattern is the same please. Taking for example the Steamgen dataset and trying to identify a repeating pattern, the underlying code works well with your data set, but if I slowly multiply the elements in the data factor so that the amplitude is slowly growing, whilst retaining the pattern, the code fails to detect the repeated part of the curve.

Can you suggest whether there is a normalisation procedure that might be used to help with this if one wanted to identify similar features, but the overall amplitude of the pattern had changed? Many thanks

[seanlaw]

@Tilanthi Would you mind sharing the code that you used to slowly generate the upper plot? Technically, if an entire pattern is vertically shifted then this wouldn't be an issue but I suspect that, in your example, it isn't a simple vertical shift.

[Tilanthi]

Hi Sean - thanks for replying - I simply took the data file you downloaded from the web and multiplied the i'th steam column by (1+i/960)) - where 960 was just an arbitrary scaling number that I used. The modified data file is attached, and I read it into a Jupyter notebook with: steam_df = pd.read_csv("modified1.txt",delimiter="\t")

modified1.txt

Many thanks for your advice

Cheers

[seanlaw]

@Tilanthi Let me take a look and see what I can find. Essentially, you've added an increasing trend to the data.

[seanlaw]

@Tilanthi I started with this tutorial. After the steamgen data is retrieved and read, I added the trend factor as you had proposed via:

i = steam_df.reset_index()["index"]
steam_df["steam flow"] = steam_df["steam flow"] * (1 + i/960)

And then just followed the rest of the tutorial (I only added the two lines of code above and that was it). As you can see, STUMPY was still able to detect the location of the correct motif (even though they were scaled differently):

[Tilanthi]

Thanks so much - I now reproduce you results, I'm not sure why it failed when I loaded the data from a csv file, but I will check this out. Many thanks again, you code works superbly for this application I am interested in.

[seanlaw]

Awesome! Glad to hear that it is working.

Technically, the z-normalized Euclidean distance takes care of the differences in trend so you should be good to go. I'm going to close this issue for now but feel free to reopen (or file another issue) if you have any more questions!