Tensorflow Estimator:使用加权分布(概率)的样本
Tensorflow Estimator: Samples using weighted distribution (probability)
我想使用加权分布(概率)对数据进行抽样
例子如下:
class分布:
doc_distribution = {0: 40, 1: 18, 2: 8, 3: 598, ... , 9: 177}
我将按 class 的等概率生成数据集批次。
total_dataset = 0
init_dist = []
for value in doc_distribution.values():
total_dataset += value
for value in doc_distribution.values():
init_dist.append(value / total_dataset)
target_dist = []
for value in doc_distribution.values():
target_dist.append(1 / len(doc_distribution))
然后,我制作 tf.estimator
的 input_fn
来导出模型,
def input_fn(ngram_words, labels, opts):
dataset = tf.data.Dataset.from_tensor_slices((ngram_words, labels))
rej = tf.data.experimental.rejection_resample(class_func = lambda _, c : c, \
target_dist = target_dist, initial_dist = init_dist, seed = opts.seed)
dataset = dataset.shuffle(buffer_size = len(ngram_words) * 2, seed = opts.seed)
return dataset.batch(20)
最后,我可以得到 rejection_resample
的结果如下:
for next_elem in a:
k = next_elem[1]
break
dist = {}
for val in np.array(k):
if val in dist:
dist[val] += 1
else:
dist[val] = 1
print(dist)
结果是:{3: 33, 8: 14, 4: 17, 7: 5, 5: 10, 9: 12, 0: 6, 6: 3}
不知道为什么rejection_resample
效果不好,我只是想平均提取样本。
我该如何解决?
在tf.estimator
的input_fn
中有什么方法可以平均采样吗?
我们可以用tf.data.experimental.sample_from_datasets
代替rejection_resample
。
unbatched_dataset = [(dataset.filter(lambda _, label: label == i)) for i in range(0, classify_num)]
weights = [1 / classify_num] * classify_num
balanced_ds = tf.data.experimental.sample_from_datasets(unbatched_dataset, weights, seed=opts.seed)
dataset = balanced_ds.shuffle(buffer_size = 1000, seed = opts.seed).repeat(opts.epochs)
我想使用加权分布(概率)对数据进行抽样
例子如下:
class分布:
doc_distribution = {0: 40, 1: 18, 2: 8, 3: 598, ... , 9: 177}
我将按 class 的等概率生成数据集批次。
total_dataset = 0
init_dist = []
for value in doc_distribution.values():
total_dataset += value
for value in doc_distribution.values():
init_dist.append(value / total_dataset)
target_dist = []
for value in doc_distribution.values():
target_dist.append(1 / len(doc_distribution))
然后,我制作 tf.estimator
的 input_fn
来导出模型,
def input_fn(ngram_words, labels, opts):
dataset = tf.data.Dataset.from_tensor_slices((ngram_words, labels))
rej = tf.data.experimental.rejection_resample(class_func = lambda _, c : c, \
target_dist = target_dist, initial_dist = init_dist, seed = opts.seed)
dataset = dataset.shuffle(buffer_size = len(ngram_words) * 2, seed = opts.seed)
return dataset.batch(20)
最后,我可以得到 rejection_resample
的结果如下:
for next_elem in a:
k = next_elem[1]
break
dist = {}
for val in np.array(k):
if val in dist:
dist[val] += 1
else:
dist[val] = 1
print(dist)
结果是:{3: 33, 8: 14, 4: 17, 7: 5, 5: 10, 9: 12, 0: 6, 6: 3}
不知道为什么rejection_resample
效果不好,我只是想平均提取样本。
我该如何解决?
在tf.estimator
的input_fn
中有什么方法可以平均采样吗?
我们可以用tf.data.experimental.sample_from_datasets
代替rejection_resample
。
unbatched_dataset = [(dataset.filter(lambda _, label: label == i)) for i in range(0, classify_num)]
weights = [1 / classify_num] * classify_num
balanced_ds = tf.data.experimental.sample_from_datasets(unbatched_dataset, weights, seed=opts.seed)
dataset = balanced_ds.shuffle(buffer_size = 1000, seed = opts.seed).repeat(opts.epochs)