如何平滑仅在某些部分具有大噪声的曲线？

Question

我想平滑下图的散点图（点很密集），数据是here。

曲线中间噪声较大，想平滑曲线，y值也单调递增。

由于有很多这样的曲线，所以很难知道曲线中的噪声在哪里。

我试过scipy.signal.savgol_filter，但没用。

我使用的代码是：

from scipy.signal import savgol_filter
from scipy import interpolate
import numpy as np
import matplotlib.pyplot as plt

s = np.loadtxt('data.csv', delimiter=',')
x = s[:, 0]
y = s[:, 1]
yhat = savgol_filter(y, 551, 3)

plt.plot(x, y, 'r')
plt.plot(x, yhat, 'b')
plt.show()

非常感谢您的建议。谢谢！

--------------------更新---------------- ----------

按照科林的方法，我得到了我想要的结果。这是代码：

from scipy.signal import savgol_filter
from scipy import interpolate
import numpy as np
import matplotlib.pyplot as plt

s = np.loadtxt('data.csv', delimiter=',')
x = s[:, 0]
y = s[:, 1]
yhat = savgol_filter(y, 551, 3)

tolerance = 0.2
increased_span = 150
filter_size = 11

first_pass = medfilt(y,filter_size)
diff = (y-first_pass)**2
first = np.argmax(diff>tolerance) - increased_span
last = len(y) - np.argmax(diff[::-1]>tolerance) + increased_span
print (first, last)
#interpolate between increased span
yhat[first:last] = np.interp(x[first:last], [x[first], x[last]],  [y[first], y[last]])


f = interpolate.interp1d(x, yhat, kind='slinear')
x_inter = np.linspace(x[0], x[-1], 1000)
y_inter = f(x_inter)
y_inter = savgol_filter(y_inter, 41, 3)

plt.plot(x, y, 'r')
plt.plot(x, yhat, 'b')
plt.show()

Answer 1

如果我们首先隔离问题区域，有很多方法可以将其删除。这是一个例子：

tolerance = 0.2
increased_span = 150
filter_size = 11

#find noise
first_pass = medfilt(y,filter_size)
diff = (yhat-first_pass)**2
first = np.argmax(diff>tolerance) - increased_span
last = len(y) - np.argmax(diff[::-1]>tolerance) + increased_span

#interpolate between increased span
yhat[first:last] = np.interp(x[first:last], [x[first], x[last]],  [y[first], y[last]])

如何平滑仅在某些部分具有大噪声的曲线？

How to smooth a curve with large noise which is only in certain part?

python

numpy

data-processing

scipy

smoothing