Data Is Weird

Data Is Weird

When using data that fluctuates, it can be very confusing. I recently ran into a unique situation where the data changed drastically within a minute. So I thought I would share. Below is the result of what happened.

Noticing Something is Wrong

We work with farmers creating an app to help manage processes throughout their farms. In this app it follows the markets of commodities (corn/soybeans/wheat/etc…)

One day ago I finished building some graphs that illustrated sales. The three labels are Sold, Unsold Insured, and Unsold Uninsured. This data is coming from a calculation that is happening elsewhere. After I finished the graph everything looked fine. This was the result.

The next morning I come in and I need to make some color scheme changes. First thing I notice is that the graph is no longer the same. It is something similar to this.

Didn’t really think anything of it at first. Then I switch to a different farm and I can see this farm should have bars filling both Unsold Insured and Unsold Uninsured sections, but the bar is just filling up with green instead. Looks something like this.

Finding the Problem

Time to go back and look at the raw data and the calculation for the graph. I can see the raw data has negative values! The graph does not take into consideration if the commodity is losing money.

So lets say a farmer has sold 100,000 bushels of corn and is waiting to sell the other 50,000 bushels (both insured and uninsured.) Now lets say the market for corn is $4.00. Then it drops down to $3.68. That can be a significant change. As a result of this, the farmer could potentially be losing quite a bit of money.

Because we wanted to keep track of things like this, we set up an “Unsold Position” section which monitors the current market’s value for that commodity. So when something like this changes the farmer gets notified.

Revising the Graph

Now that I know the graph isn’t considering if the commodity is losing money, I need to change how the calculations work. Little change here, little change there. I get everything working properly.

Because of the negative value I wanted to illustrate that with a red color to draw attention to it. Now the style looks all sorts of janky. I couldn’t tell the difference from where the Unsold Insured started and the Unsold Uninsured started. Looks similar to this.

Time to bring in the designer. I send him a possible solution and he agrees with the concept and send back another possible solution. So I go ahead and code up both solutions to see which one we like better.

As soon as I do save it, (time is 8:43am) I ask him to come take a look. I refresh and we take a look at the first version. We see something like this.

Then I make the other change and go to refresh the page again (time is 8:45am) Nothing looks the same at all! This is what we see.

Now I was extremely confused. Time to look at the raw data again. I can see that it has all changed! Now I am getting positive numbers. Having forgot that in the background our data automatically updates with the markets. So after searching around for a few minutes I start laughing and tell him that the markets probably have changed which is why we are no longer getting negative values.

He doesn’t believe me and looks up the market values here: agweb.com/markets/futures. Then he says it updated at 8:45am.

In conclusion, data is weird. It was pretty bizarre how this situation panned out considering right when I bring the designer in for the final say, the data has completely changed and we couldn’t recreate the issue.