Wednesday: Even less productive than yesterday

I discovered my "fast" algorithm is also "only about 70% accurate", which means tomorrow I debug that mess.  However, it was nice to realize that all the weird stuff I've found matches the weird stuff other people found too.  So that's nice.  Lots of different support is better than one possibly incorrect due to the aforementioned 70% accurate thing.

I got to the "more weird scifi stuff" part of the game.

I mean.

This isn't historically accurate.

And then I fought a villain, and won, but due to this stupid block:

Right here, in the background behind her corpse.

I wasn't able to put her body with her dead daughter's body in the sci-fi room.  Why is that block there, game?  I could be more pleasant to this lady that I know very little about, and probably don't have enough to be so angry with her that I leave her body in the desert which I guess is what you wanted me to do, isn't it game?

Anyway, the pyramids look cool.

• I knew Gert wasn't going to be evil, despite what the cover wanted us to believe.
• Another cliffhanger?  I guess I'll just have to pre-order the third book in the series too.
• Decans.  How ancient Egyptian constellations were arranged.  The game may not be historically accurate, but that doesn't mean I can't look up accurate details.

Tuesday: Less done than expected.

Mostly because none of the big math/science packages actually agree with each other, so you have to futz with a bunch of stuff.  The good news is I updated the slow algorithm by a factor of about 100.

Being respectful.

That's pretty ominous.

But I guess the lights are just constellations, so I have to come back once I finish the "do the constellations" quest.

• What?  You need taxes to pay for roads?  WHAT?
• Duh.  I don't understand why someone who actually has plans for how to do things has to debate with people who are actively avoiding giving details because people won't like them.
• That's cool.
• Waria.

Monday: Tomorrow I need to math at something to prove that I'm right.

Certainly hope I'm right, I guess.

Finally completed the Celebi questline.

• Why art is cool.
• Crisis on Infinite Earths keeps getting cooler.
• I was thinking about this problem last week, and wasn't totally sure it was Hilbert's.  It is.

Monday: I'm super sleepy.

I'm still kind of baffled by this:

This shouldn't be a straight line.

• Bon Appétit Cinematic Universe.
• They insist of destroying everything they bought, don't they.
• Elephants.
• REally what should happen to Lex in any universe.

Monday: Productive days.

I figured out most of the problems I ran into last week, so that's good.  I might end up having to do more telecons, but I guess that's just how things work.

My switch ran out of power, so I had to play other games yesterday.

• WTF.
• I wonder why.  It's not like people have less stable jobs, and get paid far less than they need to cover student loans and everyday living, right?
• Oh, that's cool.  I've seen a lot of articles about "new multiplication method", some of which mention that the new one is n log n.  None of them had any reasonable details, but I finally tracked this one down, which presents the previous best algorithm.  The trick is to treat each digit of the numbers as elements of a sequence, and then convolve the two sequences in Fourier space.  Cool.

Friday: And now it's cold.

At least one of the math things I thought through over break works the way I wanted it to.  These plots are the distributions of 100k samples drawn from Poisson and Gaussian distributions with mean = variance = 8.5 * 10**i

So mean of 8.5.

Mean of 85.

Mean of 85000.

The Poissonian distribution turns into a Gaussian, and a +2-sigma deviation comes up ~2% of the time as you'd expect.  If this distribution is the number of times a thing happens a day, there should be like six of those kind of days a year.

• Peanut butter isn't on this list.
• I like Stardew humor.
• She-Hulk, from when Byrne wasn't insane.
• "Skurch."

Tuesday: New rule: no playing Stardew until 6am.

Which I wouldn't have had to do if Marnie would have just gone to work one dang day and sold me a stupid chicken.  But nope!  She decided to take two days off, and then go to the Flower Dance, and not sell me anything to put in my stupid empty coop THAT I BUILT LIKE FOUR DAYS AGO, MARNIE.

But yeah, I probably shouldn't play for that long again.

The problem with getting a book delivered is that I now have to decide if I want to read it all right now, or save it for my travel book for when I visit Austin.

• And I finished my Crunchyroll show, and WTF, Bell's grandfather was actually Zeus in disguise? WTAF?
• Ellsworth Kelly stamps.  They didn't have any at the post office when I went there to pick up a package today.
• Some neural nets can be thought of as differential equations, and those can be thought of as classical mechanics.
• Geometry.

Thursday: This is why documentation is super important.

Because with a full set, it wouldn't have taken me so long to make this:

So if I have time tomorrow, I'll look into adding the overlay stuff to the widget code I was working with.  And then there'll be a telecon before we head home.  And then home.

And then serious "my own bed" ptfo.

Monday: Meetings, outlining, discovering my last merge had bugs that no one caught. You know, a fun-day.

Ugh.  At least the bugs are easy to fix and cosmetic at worst, although ugh.  Also double ugh.

The outline worked well, though, and I was able to get parts A and B to output to the same format, which I'm calling a win.

Then I tried to find tiny cans of pop, and ended up going to Target because no place else sells them, I guess.  Then home, dinner, and another episode of CardCaptor Sakura.  Which led to this:

A relationship graph for all important human characters on the show.

Blue is for friendship, green is for "like-like", and red is for "hates."  I'm really unhappy that season three reworked some of the relationship things (redirecting the Sakura-Yukito-Xiaolong triangle; not redirecting the Rika-Terada-san creepfest).

But it gave me an excuse to play with a directed graph library, so that's always fun for the future.  It's a bit unstable to initial ordering/edges, but that's fine.  Snippet:

#/usr/bin/env python3

import pygraphviz as pgv

g = pgv.AGraph(strict=False,directed=True, concentrate=True)

char =  ['sakura', 'toya', 'yukito', 'xiaolong',
         'tomoyo', 'sonomi',
         'nadeshiko', 'fujitaka',
         'meiling',
         'naoko', 'chiharu', 'takashi', 'rika', 'terada-san']
for n in char:
        g.add_node(n)

g.edge_attr['color'] = 'green'

g.add_edge('sakura', 'tomoyo', dir='both', color='blue:green')
g.add_edge('toya', 'sakura')

g.add_edge('toya', 'xiaolong', color='red')

# etc...

g.draw("/tmp/ccs.png", prog='dot')

Finally, more Bloodstained, where I discovered I need another secret item to upgrade my sword again, which is a pain.  I also broke the game and fell out of the map, and then was greeted by "Party Legs" when I got back into the map:

• I am surprised.
• AO3 is better than paid things.
• Polygon makes good game stuff.

Friday: A good day.

I got a talk proposal submitted (thanks, Julie!), I was able to confirm that the stuff I spent the week writing works well, I found a pile of hidden bugs in the code that's supposed to test, and I finally decoded the "how to get medicine you really need" puzzle, and so will have medicine delivered tomorrow.  A three month supply, all for the high price of $5.  In Hawaii, this was running $200/month.  I think that right there basically covers the pay change, and that skips the fact that I pay less for my insurance (probably why it was so hard to sort things out), and that I pay less for groceries and things.

So that's all good.

This is less good.  It suggests the new code isn't calculating things the same way, and is therefore getting errors on ~1%, which over ~25\AA, works out to 250m\AA.  That's a huge error, and reasonably consistent with the matched data I was looking at earlier this week.  Guess I know what's on the list of things to sort out this weekend.

• The cold war.
• This will be awful.
• Isn't Stouffer's the leading meat lasagna?  (Yes, I'm complaining about an ad.)
• I see.  So those garlic noodles are a west coast thing.

Monday: Today has been nothing but fighting against indices.

Stupid indices.

Also snow.  I had to clean my car off this morning.

But mostly indices.  It's possible I should just redo this with the way I know works first, and then try to make it snazzy.

• Well that's fun.
• So it costs more, helps fewer people, and directly goes against what people voted for.  Good job.
• I had not heard of Mansa Musa.
• A lot of these schools are nearby.  Will tomorrow be a snow day?

Tuesday: I need to actually think about this systematically, I guess.

Technically it's not my problem, but I have enough thoughts that it never hurts to give things a look.

• These people are not clever.
• The TMT probably will be built.
• I just assumed you did, and that shrimp was the only thing to not get soy sauce.

Wednesday: Ugh, that just convolves the two

I know I've made this work before, but I can't seem to sort out what I'm doing wrong this time.

Maybe when I did this before, one of the two distributions was sufficiently close to flat that it didn't matter?

Ugh.  I guess interpolating the CDFs and working out ratios is the way to go.

• It's supposed to be a peaceful match.
• So the answer is yes, Republicans have based basically everything on a policy of racism.
• Red pandas are cute.

Thursday: Just before I went home, I realized that the hygenist did more actual work than the dentist.

Like, why not just have her do everything?  The dentist was kind of a jerk, and the only real "work" he did was stab me with the scrape-y thing and floss.  "Did she do the floss?" So she sometimes does that too?

And this was pretty much how work went.  Minus the tigers.  I added the tigers for fun.

I was also surprised at how much the Pobs value changes with Pexpect, but looking at the plot, it makes sense.  You have Nobs, and that's fixed, so if Pexpect goes up, that pulls the distribution upward, and the Pobs plummets.

I need to send more postcards.

• This is the reason I have most of my student loans paid off.  Republicans are garbage people.
• "So what do the dragons stand for?"
• I liked this.
• Being grownup.  Also, that is a sweet shark lunchbox.
• Ok, I'm a big fan of pancakes, and I'm a fan of sweet and savory, but I'm pretty sure "chili pancakes" are messed up, 1936.
• I always try to tell the birds which way I'm going, so they can run the other way.  It is not often successful.
• They are secret friends, and that is great.

Wednesday: No, it's fun to discover a massive problem with a database just when you want to go home.

But I think I was able to fix it with a redo, and changing all the "INSERT"s to "INSERT IGNORE"s.

And I wanted to make a proper version of the garbage scribble at the top of this post.

I really like the way R plots things.

 It's reasonably simple:

b = data.frame(N = seq(0,177))
b$P = choose(177,b$N) * 0.24**b$N * (1 - 0.24)**(177 - b$N)

L = rbind(c(0,0), subset(b, b$N <= 36),  c(36,0))
U = rbind(c(44,0), subset(b, b$N > 43), c(177,0))

ggplot(b,aes(x = N, y = P, color="P_expect = 0.24")) + \
  geom_polygon(data=L,aes(x = N, y = P, fill="Pobs")) + \
  geom_polygon(data=U,aes(x = N, y = P, fill="Pover")) + \
  geom_line() + geom_point()  + \
  scale_colour_manual(values=c("black"),name="",
                      guide=guide_legend(override.aes=aes(fill=NA))) + \
  scale_fill_manual(values=c("red","blue"),name="")

ggsave("/tmp/2017b.png")

The only major issues are that doing the shading requires constructing a polygon object, and that needs to have endpoints set correctly (or it shades the wrong way).  Getting all the colors and legend set was also not super obvious, and that "override.aes" thing is just nonsense.

"But it's not snek."  So then it was a challenge to see how to do this in snek, too.

Snek is simpler, but their documentation is far worse.  Examples should start slow, not alphabetically with "animation."  Why would you do that?

#!/usr/bin/env python3                                                                                                       
import matplotlib.pyplot as plt
import scipy.special
import numpy as np

P_expect = 0.24
N_talks  = 177
N_obs    = 36
N_expect = 44

N = np.arange(0,N_talks,1)
P = scipy.special.binom(N_talks,N) * P_expect**N * (1 - P_expect)**(N_talks - N)

# Lower portion
Nl = np.arange(0,N_obs,1)
Pl = scipy.special.binom(N_talks,Nl) * P_expect**Nl * (1 - P_expect)**(N_talks - Nl)
Zl = Nl * 0.0

# Upper portion
Nu = np.arange(N_expect,N_talks,1)
Pu = scipy.special.binom(N_talks,Nu) * P_expect**Nu * (1 - P_expect)**(N_talks - Nu)
Zu = Nu * 0.0

plt.grid()
label_text = "P_expect = %.2f" % P_expect
plt.fill_between(Nl,y1=Zl,y2=Pl,color="red",label="P_obs")
plt.fill_between(Nu,y1=Zu,y2=Pu,color="blue",label="P_over")
plt.plot(N,P,color="black",label=label_text)
plt.scatter(N,P,s=5,color="black")
plt.legend()
plt.savefig("mpl.png")

Having an explicit fill_between() function saved a lot of time.  The legend() function was also helpful for making it just work.

I'm still behind on my RSS stuff.

• "Oh no!  This new Spider-Man movie confuses the timeline!"  I complain about stupid stuff, but this is too far.  Comic book timelines have been insane forever.  I mean, look at Squirrel Girl.  Doreen was 14 when she defeated Doom with Tony, then she did stuff for a few years with the GLA, then she was kind in the Avengers, then she babysat Luke Cage/Jessica Jones' daughter, and now she's in college.  How old is she?  Why isn't she 39 if the comics follow regular time?  Doesn't matter, she's doing college now, enjoy your wonderful stories.  Comic book time is meaningless.
• Also they cast Doreen, and this seems like it'll work.
• Wonder Woman.
• Whoops.
• Best Spider-Man.
• Buffalo.

Tuesday: My back hurts.

And I didn't have a chance to read all my internet today.  I did decide that since the company I use for domain name crap seems broken, I should just buy a domain name and use a proper service.  Googling that problem had Google tell me that Google does that.  Fine.  Done and done.  It doesn't look like the DNS update has propagated, but Googling that suggest it can take up to 24 hours.  Then I'll be able to easily read my internet while waiting for lunch again.  Woo.

Today's picture is the binomial distribution stuff that I took a picture of with my Google powered phone, from a company that was owned by Google when I bought it, sent via Google chat, and then uploaded to Google photos so I could use it on my Google blog.

The point is that a lot of the time, I just want crap to work, and Team G is pretty good at doing that.

Thursday: No seriously, this week is all messed up for me now.

I spent part of the day thinking it was Monday, because we had a meeting, and those are usually on Monday.  But it's not, so tomorrow I have to try and sort out things that I want to finish this week.

Also, I learned that the answer to "I have all this crap in a data frame, and I want answers for each subset, but I don't want to have to do any work" is ddply.  Originally I did this with work data, but using the NSF PhD statistics works as well.

library(plyr)

nsf = data.frame(read.table("./matched.dat",sep='\t',header=TRUE))
nsf$R = nsf$female / (nsf$male + nsf$female)

q = ddply(nsf, .(broad_field), function(x) c(m = mean(x$R),s = sd(x$R), quantile(x$R, c(0.0, 0.5, 0.75, 0.90, 0.98, 1.0))))

                           broad_field         m           s

1                            Education 0.6780287 0.012619293

2                          Engineering 0.2169928 0.014537864

3                  Humanities and arts 0.5035801 0.007278568

4                        Life sciences 0.5384433 0.017871779

5    Mathematics and computer sciences 0.2479207 0.009173072

6                               Otherb 0.5059506 0.011422802

7 Physical sciences and earth sciences 0.3117802 0.018656152

8       Psychology and social sciences 0.5832883 0.011162981

        50%       75%       90%       98%      100%

1 0.6816616 0.6865097 0.6927421 0.6930276 0.6930990

2 0.2220837 0.2279097 0.2304950 0.2320948 0.2324947

3 0.5058046 0.5084105 0.5096879 0.5120727 0.5126689

4 0.5453048 0.5518069 0.5544373 0.5564318 0.5569305

5 0.2466649 0.2529897 0.2614564 0.2629909 0.2633745

6 0.5105890 0.5143730 0.5165125 0.5207457 0.5218040

7 0.3140528 0.3249957 0.3339903 0.3353106 0.3356407

8 0.5849175 0.5890918 0.5946965 0.5973446 0.5980066

Ok, so the quantile data isn't super useful with the NSF data, but still.  Means and sigmas for each factor, and for the work data, pulling the quantile stuff was useful.  Then, hubris took hold, and I wondered if I use this to do linear fits to each factor as well.  The answer is no, not with ddply, because that reads and writes a data frame, and lm outputs a model object.  So you have to use dlply to save those models in a list, and then make a data frame from the stuff you care about in the list:

f = dlply(nsf, .(broad_field), lm, formula = R ~ year)

coeffs = ldply(f,coef)

coeffs$x2010 = (coeffs[,2] + 2010 * coeffs[,3]) * 100

coeffs$m = coeffs[,3] * 100

                           broad_field (Intercept)          year    x2010           m

1                            Education   -5.318890  0.0029835418 67.80287  0.29835418 

2                          Engineering   -7.627153  0.0039025598 21.69928  0.39025598 

3                  Humanities and arts   -1.145504  0.0008204401 50.35801  0.08204401 

4                        Life sciences   -9.438112  0.0049634604 53.84433  0.49634604 

5    Mathematics and computer sciences    1.055520 -0.0004017907 24.79207 -0.04017907 

6                               Otherb   -3.713211  0.0020990852 50.59506  0.20990852 

7 Physical sciences and earth sciences  -10.098766  0.0051793762 31.17802  0.51793762 

8       Psychology and social sciences   -4.194659  0.0023770883 58.32883  0.23770883 

And yes, x2010 is the same as the mean above, and m (percent change per year) is approximately 1/5 of the sigma above.  Taking a quick average of just the math, engineering, and physical sciences data for 2017 gives something like 28%.  The lack of improvement in math is a problem, and engineering starts at a deficit.  The depressing thing is that even with the highest improvement rate for the physical sciences, it's still something like 40 years until parity.

Also a plot:

library(ggplot2)

ggplot(nsf,aes(x=year,y=R,color=broad_field)) + geom_point() + geom_line() + geom_smooth(method=lm)

ggsave("/tmp/with_fits.png")

Yes, I'm sure there's a line thickness parameter I could have changed.  Whatever.

• More ggplot information.
• Hrm.
• That sandwich thing popped up again.
• Sits.
• I like this, because even though I have no clue what fandom this is from, I can pick up the entire story just from the pictures and the caption.
• Just some Wonder Woman.

Monday: I should have just taken the four day weekend. I would have been about as productive.

However, last night, I was able to plot up and fit how the height a  Magikarp jumps scales with JP:

JP/1000, obviously.  But my "this feels like it's a square root" idea seems to be supported.  This is just one pass through the league I'm currently battling.

And today I finally decided that I wasn't going to find an easy way to reshape this table of PhD recipients, so I just wrote a perl script to do it.  Once it was in nice "field/year/numbers" format, I was able to get R to make a pretty plot:

So I guess the 24% value probably isn't too far off.

f = data.frame(read.table("./matched.dat",sep='\t',header=TRUE))

ggplot(data=f,aes(x=year,y=(female / (female + male)), color=field)) + geom_line(aes(group=field)) + geom_point()

• A good story.
• "Millions of identical human men".
• Cocoa.
• I am glad to see that Squirrel Girl and Koi Boy being friends is backed up by nature.

Thursday: I would have been productive today.

But everything conspired to not work.  I was able to get one thing running, and tomorrow I'll probably spend a large chunk of the day aborting and restarting shit that should have run today, but things decided not to work.  I'm also increasingly concerned that my database ingest is taking far longer than it should be, and I'm not sure how to fix it.  One thought is to pull out the select statement stuff, and do that in one pass, and just insert directly.

And although I still need to do a time breakdown, this shows that my classification thing is reasonably consistent.  This shows the time average over existing entries.  Something like 98% have classification deviations are less than ~5%.  There is some evidence of class switching, but I haven't sorted out how I want to break that out.

I have caught up on RSS, but I have a massive backlog.

• Spaghetti goblin.
• Pokemon.
• The best inspiration.

Wednesday: My Cloyster is a wonder.

I wasn't expecting him to last so long.

Not that I'm complaining.

I've also decided that I like R, and need to know more about it.

It has nothing to do with how easy it was to make this.

Or this.  Nothing at all to do with how quick it is to turn a giant data file into useful answers.

"But snek," Julie said.

With this wonderful gif.

The thing is, I don't like snek, and although R has some odd syntax (specifically the <- a="" and="" are="" assignment="" automatically.="" back="" boxplot.png="" can="" closer="" dev.copy="" dev.off="" emicolons="" far="" good.="" have="" i="" into="" is="" it="" like.="" macro="" nbsp="" operator="" p="" plot="" png="" revert="" save="" stick="" that="" the="" things="" to="" what="" window="" work="" x="">
Before I start the links, the first one has a background painting, and I wanted to know what it was.  So I opened the gimp:

And titled the quickly made cut out "will_this_work.jpg".

The answer is yes, google images is perfectly happy telling me that this is Perugino's Resurrection.

• A happy pope.
• Interesting article on NN bias.
• Yeah, this seemed like a significant part of the timing.
• Sane people would realize they've chosen poorly.
• Sneaky animals.
• The math breakdown of why this full of shit.
• Art.
• This dog looks done.  With everything.