Also: how's this for a problem/algorithm: Given a data set that is contaminated in one direction by bad values, determine the best mean level if the contamination were not there. First choice is probably going to be a median, but if the contamination is sufficiently prevalent, you're likely to end up biased in the direction of the contamination. Alternatively, you can take the assumption that the noise in the data set is normally distributed, and therefore is symmetric. This leads to the suggestion that any asymmetry in the observed distribution function of the data is due to the contamination.
Ok, so how to symmeterize the data (yeah, that's probably not a word)? Construct the median subtracted CDF, and then construct the a CDF (reflected around x = 0 and y = 0.5). CDF1 tells you what the data says the distribution is, and CDF2 tells you what the opposite side of the median looks like, telling you how asymmetric the observed distribution is. So you can just reweight the values on the contaminated side by the ratio of the CDF values at that point CDF1 / CDF2 (properly accounting for CDF2 == 0.0).
However, this will basically make things left of the median look like the right of the median, resulting in a median that's the median again. So, let's switch the median to a weighted mean. This should give a result closer to the unbiased distribution peak.
One concern with this is that you can't iterate it. Iterating will just increase the fraction of low weight values, eventually pushing the "mean" to the maximum value. This is a bit of a concern. Maybe I should run further with the assumption of normality and come up with something that's erf based?
 |
| Hey look! Bears! |
 |
| That's totally a super villain lair. |
- Wait, you can put mac&cheese on a hot dog?
- I thought the same thing when I first saw the NASA mosaic. I wonder if they don't use hugin like I do?
No comments:
Post a Comment