Jekyll2020-10-08T05:21:00+00:00matthewcrews.com/feed.xmlMatthew CrewsA blog on the intersection of Mathematical Optimization and Machine LearningMatthew CrewsWhat I Look for in a Machine Learning Engineer2020-10-07T00:00:00+00:002020-10-07T00:00:00+00:00matthewcrews.com/what-i-look-for-in-a-machine-learning-engineer<p>I was recently asked by someone, “What do I need to do to get into Machine Learning and Finance industry?” I told them that I would think about it and get back to them. I have been in many interviews for Machine Learning Engineers and I do have a set of questions that I frequently use to get a feeling for where someone is in their career and where they are hoping to go. We are looking for people at various phases in their growth, so it is less of a concern about whether someone is some elite developer and more about whether they are inquisitive and eager to learn. I will not share the questions I use in interviews, but I will happily share the attributes I am looking for.</p>
<h2 id="sound-engineering">Sound Engineering</h2>
<p>Before anything else, I am looking for someone’s engineering acumen. I went to college for Chemical Engineering and I believe on our first day the professor defined engineering as problem solving under constraints. The key thing that we learned was how to take an abstract problem statement, break it down into manageable pieces, and then solve the smaller subproblems. I believe that most of engineering discplines could be boiled down to this, they just deal with different problem domains.</p>
<p>I believe a key differentiator between more and less experienced developers is in understanding where to break the problem down. Problems have an inherent amount of complexity. Good design can mitigate that complexity keep it from growing out of control. It is the same concept of algorithmic complexity. We analyze an algorithm and see how the runtime grows as the size of the problem increases. In the same way, we can look at an architecture and see how the complexity grows as we increase functionality and services.</p>
<p>The domain of Machine Learning is still relatively fresh, and the industry is still churning with new Python libraries popping out every other week. Therefore, the solutions we build for delivering Machine Learning Models must be well designed because they will likely need to evolve. Building a system that can cleanly evolve over time is a difficult engineering challenge. I am looking for people who understand the implications of decisions in the near term but also how it will affect the evolveability of the system going forward.</p>
<h2 id="strong-machine-learning-fundamentals">Strong Machine Learning Fundamentals</h2>
<p>Though this may seem obvious, there are some nuances I wanted to highlight. While I am fascinated by the advances we are seeing with Neural Networks and Reenforcement Learning, they should not be the first tool you reach for. I am going to ask someone about a hypothetical problem and get their feedback on how they would approach it. If the first thing they suggest is, “Oh, use a Neural Network!” red flags are going to go off. Neural Networks are powerful, but they should not be the first tool you reach for in your toolbox.</p>
<p>It is impressive just how far you can get with Linear Regression and Logistic Regression. These algorithms are robust, and their behavior is easily explained. It is enourmously valuable to be able to explain the behavior of a model to your non-ML colleagues in terms they can understand. You can then take things a step further and do some data exploration and try subsetting your population and having different models for the different populations. Think of it as a brute force Ensemble technique. Now, if you reach the end of Linear and Logistic Regression, I am all for trying more sophisticated techniques. You just should not start with the fanciest tool in your toolbox.</p>
<p>I also want someone to understand the failure modes of the model and how they impact the outcome. It is easy to get caught up in trying to tune a model for a particular metric but is that really the best thing for how it will be used in production? Machine Learning models really took off when they were applied to the field of advertising where the upside of being correct was large but the downside of being wrong was low. This was a perfect application of these tools. What happens when the downside of a wrong prediction is large? What if a wrong prediction leads to someone dying? What metric are you going to tune the model for in that scenario? I want someone who is going to ask the question, “Why do you want to predict that? How do you plan to use the prediction?” because only when you know that can you really discern the best model.</p>
<h2 id="team-skills">Team Skills</h2>
<p>The industry often calls these “Soft Skills”. I think that undervalues how critical they are. The problems we are facing now are so large and complex they cannot be completed by a single individual. If someone does not have the ability to operate well within a team, I am not interested. I do not care how much of a wizard you are. If you are one of the fabled 10X developers but you apply a negative multiplier to the rest of the team, it is going to be a net loss for us.</p>
<p>The first and most critical Team Skill is the ability to communicate in writing. This includes the clarity of your code, emails, chat, diagrams, all of it. If you do not have the ability to put your thoughts down into a written form, it is going to be difficult for you to pass your ideas around the team. So much of our communication now occurs through text whether it be email, code, pull requests, and chat messages. I will not automatically disqualify someone if there writing is not amazing, but it is a clear differentiator. If you have a blog where I can see your work and how you explain your thoughts, you are automatically in the top 5% of people I interview.</p>
<p>Next, I want to know if you can explain complex ideas. Throughout the interview I am going to be listening for what you are passionate about or have a deep understanding of. I am then going to ask you to explain that concept to me. I may already be familiar with the subject, but I want to know if you can explain it to someone who is not a domain expert. Developers who can break complex problems down into explainable pieces are a rare breed.</p>
<h2 id="technical-skills">Technical Skills</h2>
<p>There is a reason this section comes last. While it is important, it is the the that least differentiates people. I care little about which stack or set of technologies you have worked with in the past. If you happen to be familiar with what we are using it is a small bonus, but it is nowhere near the most important thing. I will want to know if you have worked with the concepts though. I want to know if you have worked with Object-Oriented and Functional languages. I want to know if you have worked with Relational Databases or streaming systems. The reason for this is more to guage what training someone would need should they come on board. Finding someone who spends time learning on their own is a huge bonus because it shows curiousity and initiative, not because I expect them to work overtime to get up to speed.</p>
<h2 id="academics">Academics</h2>
<p>I don’t care. I honestly don’t. I used to be really intimidated by people with degrees from top end schools and honestly, the above areas matter more in the long run. I work with people from wildly different backgrounds. Do not count yourself out if you do not have a Computer Science degree. I cannot remember where anyone on my team went to school or whether they have degrees.</p>
<h2 id="final-thoughts">Final Thoughts</h2>
<p>So, what do I say to my friend who asks me, “How do I get into Machine Learning and Finance?” I say, practice and show your work. Start a blog. Talk about what you are learning each week. Learn to turn complex problems into clear prose. Find a problem that you are intersted in and create a GitHub repo where you document your progress. Find some interesting data set and perform some analysis on it. Document what you find. Create a model for predicting an outcome and explain which metrics you used to tune the model and why you chose those metrics. Do it using Pull Requests with tagging. Create a simple ML Model that you deploy as a Nuget package and setup an automated CI/CD pipeline to deliver new versions of the model when you merge to the main branch. These are the daily things we do as being developers.</p>
<p>Finally, and perhaps most importantly, get involved in the community. Just find a developer tribe you jive with and connect with some people. I know there are some hostile communities out there but there is also an abundance of generous developers. I know Python has tons of resources. I love F# and it is the most generous community I have been a part of. R also has great people and meetups. Connecting with people and learning from them will create more opprotunities than anything else.</p>Matthew CrewsI was recently asked by someone, “What do I need to do to get into Machine Learning and Finance industry?” I told them that I would think about it and get back to them. I have been in many interviews for Machine Learning Engineers and I do have a set of questions that I frequently use to get a feeling for where someone is in their career and where they are hoping to go. We are looking for people at various phases in their growth, so it is less of a concern about whether someone is some elite developer and more about whether they are inquisitive and eager to learn. I will not share the questions I use in interviews, but I will happily share the attributes I am looking for.The Anatomy of an Optimization Model2019-11-11T00:00:00+00:002019-11-11T00:00:00+00:00matthewcrews.com/the-anatomy-of-an-optimization-model<p>I am on a bit of a quest to bring Mathematical Optimization to the masses, or at least to Software Developers. I often come across problems where people are wanting to find the best plan for their problem but they lack to tools to express the problem. Typically the way this is “solved” is by some domain expert coming up with a laborious heuristic in Excel which involves outrageous amount of copying and pasting. I have seen this is take place in tiny companies all the way up to multi-billion dollar enterprises. What really breaks my heart when I see this is that I know there is a better way, but people are just are not aware of it. This is why I am pushing for more training on Mathematical Optimization at my company and why I am starting a blog series on Mathematical Modeling.</p>
<h2 id="the-goal">The Goal</h2>
<p>My hope with these blog posts is to start from a barebones introduction to the concepts which undergird mathematical modeling and slowly introduce newer and more advanced modeling techniques. I don’t think I will ever finish because I am always across new and interesting problems. What I will emphasize is the beautiful interplay between Machine Learning and Mathematical Optimization. I have been blessed to work on several projects where we were able to marry these tools to great effect. If you have any problems that you would like me to look at and write a blog post on, I would be happy to. My hope is to give examples which help people with their own work.</p>
<h2 id="ingredients-for-an-optimization-problem">Ingredients for an Optimization Problem</h2>
<p>There are three things you need to have before you are reading to apply Mathematical Optimization. You need 1) A Quantifiable Objective 2) Decisions you control and 3) Rules to follow. When you have these three things, you have a problem ripe for Mathematical Optimization. Let me unpack them a little to make sure you understand what I am talking about as we move forward.</p>
<h3 id="a-quantifiable-objective">A Quantifiable Objective</h3>
<p>For you to apply Mathematical Optimization to a problem you need a way to measure how successful you are. Without this, a Solver will not have a way to check if the solutions that it is finding are actually an improvement or not. You are typically trying to maximize some value or minimize it. Common Objectives are Maximize Revenue, Minimize Waste, Maximize User Engagement, Minimize Energy Use, or Minimize Cost. What you are trying to achieve could really be anything, the key is that you have the ability to quantify it.</p>
<h3 id="decisions-you-control">Decisions You Control</h3>
<p>When you create a Mathematical Model of your problem you are going to define the Decisions which need to be made. This is often something like how many people you assign to a task, how much water to put where, where to place a warehouse, which ad to show where. These are all examples of decisions you will be making in your business. The important thing is that you actually have control over them. When you solve the Optimization Model, you will get answers for what values you should use for your Decisions. It will tell you how many people to assign to a task or where to put the water or which ad to show where.</p>
<h3 id="rules-to-follow">Rules to Follow</h3>
<p>Strictly speaking, it is possible to have an Optimization Problem without a set of rules to follow but it is rare. Most real world problems will have some kind of constraint on what you are allowed to do. Typically there are a limited number of people or only so many locations where you can place a warehouse or only so much power available. It is often the rules to follow which make a Optimization Problem interesting. Most often someone is trying to find the best plan given a set of restrictions that they need to follow. These restrictions are what make finding the best answer difficult.</p>
<h3 id="where-we-go-from-here">Where we go from here</h3>
<p>Let’s say that we have checked all the boxes and it looks like you have a problem which is a good candidate for Mathematical Optimization. What do we do now? We formulate a model. How we do that is what I am looking forward to opening up to you in this and future blog posts. Once we have a Model of our problem we are able to hand it to a piece of software called a Solver which knows how to take the model and search for the best possible solution.</p>
<p>My plan for this series is to follow a simple pattern. First, present a real world problem which will help us ground the concepts. Second, develop the mathematical model and walk through how it works. There will be some math notation but I’ll walk through it slowly so you don’t get lost. Thirdly, translate the model into code. I will be using Python and the PuLP library for my examples. Python is ubiquitous and the PuLP library is open source and easy to install.</p>
<p>In the rest of this post I will walk through a toy problem for the purpose of introducing the vocabulary of Mathematical Optimization Modeling. In future posts I will work more complex problems which will have interesting characteristics.</p>
<h2 id="the-food-truck-problem">The Food Truck Problem</h2>
<p>One of the example problems I like to use is that of a Food Truck. I am from Portland, OR USA originally and we had food trucks everywhere. In this example we are running a food truck and we have to decide what items to pack for the day. We sell Hamburgers and Burritos. Hamburgers sell for \$5.00 and Burritos sell for \$7.50 (they are big Burritos). Each Hamburger requires us to carry 1.0 Lb. of ingredients on the truck and each Burrito requires 1.5 Lbs. (I told you they are big). We have a small Food Truck so we can only carry up to 650 Lbs. of ingredients. We also forgot to go to stock up the day before so we only have 200 Hamburger buns on hand and only 300 tortillas for Burritos. Since we run an awesome Food Truck, we always sell out of everything we bring. The question now becomes, how much Hamburgers do we pack for and how many Burritos so that we maximize our profit?</p>
<blockquote>
<p><strong>Note:</strong> This example problem is meant to be simple. I am mostly concerned with introducing the vocabulary of Optimization Modeling. Future problems will be more complex.</p>
</blockquote>
<p>This problem is a clear example of a Mathematical Optimization Problem. It has a Quantifiable Objective, Maximize Revenue. It has Decisions which we can control: the number of Hamburgers and Burritos we will pack for. Finally it has rules we must follow, the Food Truck can only carry 650 Lbs, we only have 200 Hamburger Buns and we only have 300 tortillas for Burritos. Now, I am going to show you how we formulate this as an Optimization Model and then I will walk through each piece so that it makes sense. For this model I am going to use the variable $x_{1}$ to represent the number of Hamburgers we are going to pack for and $x_{2}$ to represent the number of Burritos.</p>
\[Maximize: 5.00 x_{1} + 7.50 x_{2} \\
Subject\ to: \\
x_{1} \leq 200 \\
x_{2} \leq 300 \\
1.0x_{1} + 1.5x_{2} \leq 650 \\
x_{1}, x_{2} \geq 0\]
<p>Let’s unpack this. The first line of any Mathematical Optimization Model is going to be the Objective Function. This is the function which is used to quantify success. It will start with whether we are trying to Maximize the value of the Objective Function or Minimize it. In this case we are trying to Maximize. The formula that you see is the calculation for Revenue. Remember, $x_{1}$ is the number of Hamburgers and $x_{2}$ is the number of Burritos. For every Hamburger we will earn \$5.00 and for each Burrito we will earn \$7.50. This means to calculate the total revenue we multiply the number of Hamburgers by the revenue per Hamburger and the number of Burritos by the revenue per Burrito: $5.00x_{1} + 7.50x_{2}$.</p>
<p>After the Objective Function we get to a section referred to as the Constraints. This section typically begins with either a $Subject\ to$ or just $S.t.$ as a shorthand. This section is describing the rules that we need to follow. The first constraint is our limitation on the number of Hamburgers due to the number of buns that we have. We only have 200 buns available which means that $x_{1}$ must be less than or equal to 200. We write that as a constraint in this way: $x_{2} \leq 200$. The next constraint is describing our limit on the number of Burritos we could pack since we only have 300 tortillas. $x_{2}$ represents the number of Burritos we plan to pack and it must be less than 300 therefore we add this constraint: $x_{2} \leq 300$.</p>
<p>The third constraint represents the weight limit of our Food Truck. We can only carry 650 Lbs. so the combination of the number of Hamburgers and the number of Burritos must be less than this. We multiply the number of Hamburgers by the lbs per Hamburger and the number of Burritos by the lbs per Burrito and add them together to get the total weight. That total must be less than the capacity of the Food Truck. This gives us this constraint: $1.0x_{1} + 1.5x_{2} \leq 650$.</p>
<p>The final line of the model states the number of Hamburgers and Burritos cannot be less than zero. This is implicitly obvious to us as people but a Solver won’t have our context so we have to tell it that those numbers cannot be negative. This is where we are brining our added context to the problem.</p>
<h2 id="to-the-code">To the Code</h2>
<p>Now that we have walked through this small problem, let’s see how it translates to code. I am using Python 3.8 and PuLP 1.6.8. The first thing we do is import <code class="language-plaintext highlighter-rouge">PuLP</code> and create a new instance of a <code class="language-plaintext highlighter-rouge">problem</code> object.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Import the PuLP Library
</span><span class="kn">from</span> <span class="nn">pulp</span> <span class="kn">import</span> <span class="o">*</span>
<span class="c1"># Create an instance of a Problem object to populate
</span><span class="n">problem</span> <span class="o">=</span> <span class="n">LpProblem</span><span class="p">(</span><span class="s">"Food Truck"</span><span class="p">,</span> <span class="n">LpMaximize</span><span class="p">)</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">PuLP</code> provides us the tools we will need to create the Optimization Model and then solve it. Out of the box the <code class="language-plaintext highlighter-rouge">PuLP</code> library comes with some open source solvers so you can build and solve models without having to purchase a solver license.</p>
<p>The first argument of the <code class="language-plaintext highlighter-rouge">LpProblem</code> function is the name of our problem. The second is the type of optimization we want to perform, Maximization or Minimization. In this case we are wanting to maximize revenue so we use the argument value of <code class="language-plaintext highlighter-rouge">LpMaximize</code>. If we wanted to minimize we could have used the <code class="language-plaintext highlighter-rouge">LpMinimize</code> value.</p>
<p>Now let’s create some decision variables to represent how many burgers and tacos we are going to bring on our food truck. We do this by using the <code class="language-plaintext highlighter-rouge">LpVariable</code> function.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">x1</span> <span class="o">=</span> <span class="n">LpVariable</span><span class="p">(</span><span class="s">"burgers"</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
<span class="n">x2</span> <span class="o">=</span> <span class="n">LpVariable</span><span class="p">(</span><span class="s">"tacos"</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
</code></pre></div></div>
<p>The arguments for <code class="language-plaintext highlighter-rouge">LpVariable</code> are the name of the variable and the lower bound on the possible value. In some problems, the decision variables can take on negative numbers. In this case, having negative tacos or negative hamburgers does not make any sense so we specifiy that the lower bound is 0.</p>
<p>We now have the Decision Variables for the problems so we can now add the Objective Function and the Constraints. Let’s start with adding the Objective Function. Whenever we want to add something to a <code class="language-plaintext highlighter-rouge">problem</code> we use the <code class="language-plaintext highlighter-rouge">+=</code> operator. The <code class="language-plaintext highlighter-rouge">PuLP</code> library will infer whether we are adding an Objective Function or a Constraint based on the right hand side argument. All we need to do for the Objective Function is to provide the equation.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Add the Objective Function
</span><span class="n">problem</span> <span class="o">+=</span> <span class="mf">5.00</span><span class="o">*</span><span class="n">x1</span> <span class="o">+</span> <span class="mf">7.50</span><span class="o">*</span><span class="n">x2</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">problem</code> object now has an Objective Function. Now let’s go about adding the constraints. The first constraint is the <strong>Max Burgers</strong> constraint. To do this we use the <code class="language-plaintext highlighter-rouge">+=</code> operator to add constraints to our <code class="language-plaintext highlighter-rouge">problem</code> object. We then give the equation for the constraint and the name of the constraint.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Add Max Burgers constraint
</span><span class="n">problem</span> <span class="o">+=</span> <span class="n">x1</span> <span class="o"><=</span> <span class="mi">200</span><span class="p">,</span> <span class="s">"Max Burgers"</span>
</code></pre></div></div>
<p>We then need to add the <strong>Max Burritos</strong> constraint.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Add Max Burritos Constraint
</span><span class="n">problem</span> <span class="o">+=</span> <span class="n">x2</span> <span class="o"><=</span> <span class="mi">300</span><span class="p">,</span> <span class="s">"Max Burritos"</span>
</code></pre></div></div>
<p>Finally we need the <strong>Max Weight</strong> constraint.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Add Max Weight Constraint
</span><span class="n">problem</span> <span class="o">+=</span> <span class="mf">1.0</span><span class="o">*</span><span class="n">x1</span> <span class="o">+</span> <span class="mf">1.5</span><span class="o">*</span><span class="n">x2</span> <span class="o"><=</span> <span class="mi">650</span><span class="p">,</span> <span class="s">"Max Weight"</span>
</code></pre></div></div>
<p>We now have a fully populated problem. To solve it, all we need to do is call the <code class="language-plaintext highlighter-rouge">solve()</code> method.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Solve the problem
</span><span class="n">problem</span><span class="p">.</span><span class="n">solve</span><span class="p">()</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1
</code></pre></div></div>
<p>We get a numeric response back but it will not mean much until we translate it to something we can understand. Fortunately, the <code class="language-plaintext highlighter-rouge">PuLP</code> library has a dictionary which stores the mapping from the numeric status of the problem to a human readable string. This dictionary is the <code class="language-plaintext highlighter-rouge">LpStatus</code> dictionary. Let’s use this to print out the string representation of the problem status.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Print the problem status
</span><span class="k">print</span><span class="p">(</span><span class="n">LpStatus</span><span class="p">[</span><span class="n">problem</span><span class="p">.</span><span class="n">status</span><span class="p">])</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Optimal
</code></pre></div></div>
<p>We should see the string <code class="language-plaintext highlighter-rouge">optimal</code>. This means that the solver was able to find the optimal answer. In the future we will go over the other possible statuses and what they mean.</p>
<p>Now, let’s look at what values for the Decision Variables the Solver chose.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Loop through each of the Decision Variables in the problem
</span><span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">problem</span><span class="p">.</span><span class="n">variables</span><span class="p">():</span>
<span class="c1"># Print the name of the Variable and the Value the Solver chose
</span> <span class="k">print</span><span class="p">(</span><span class="n">v</span><span class="p">.</span><span class="n">name</span><span class="p">,</span><span class="s">'='</span><span class="p">,</span><span class="n">v</span><span class="p">.</span><span class="n">varValue</span><span class="p">)</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>burgers = 200.0
tacos = 300.0
</code></pre></div></div>
<p>Let’s see what kind of Revenue we should expect if we follow this plan.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Get the expected Revenue
</span><span class="n">revenue</span> <span class="o">=</span> <span class="n">value</span><span class="p">(</span><span class="n">problem</span><span class="p">.</span><span class="n">objective</span><span class="p">)</span>
<span class="c1"># Print the expected result
</span><span class="k">print</span><span class="p">(</span><span class="s">f"$</span><span class="si">{</span><span class="n">revenue</span><span class="p">:,.</span><span class="mi">2</span><span class="n">f</span><span class="si">}</span><span class="s">"</span><span class="p">)</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$3,250.00
</code></pre></div></div>
<p>There you have it. A tiny Mathematical Optimization problem. Granted, this was completely overkill for such a simple problem. My goal was to introduce these concepts and start growing our vocabulary around Optimization. From here we will adjust how we are managing the data and Decision Variables for our problem. This approach will not scale. In the next post we will look at adding the concept of $Sets$ and $Parameters$. This will make it easy for us to create large optimization problems involving hundreds of thousands, if not millions of Decision Variables and Constraints.</p>Matthew CrewsI am on a bit of a quest to bring Mathematical Optimization to the masses, or at least to Software Developers. I often come across problems where people are wanting to find the best plan for their problem but they lack to tools to express the problem. Typically the way this is “solved” is by some domain expert coming up with a laborious heuristic in Excel which involves outrageous amount of copying and pasting. I have seen this is take place in tiny companies all the way up to multi-billion dollar enterprises. What really breaks my heart when I see this is that I know there is a better way, but people are just are not aware of it. This is why I am pushing for more training on Mathematical Optimization at my company and why I am starting a blog series on Mathematical Modeling.Discrete Optimization with Cutting Planes2018-05-25T00:00:00+00:002018-05-25T00:00:00+00:00matthewcrews.com/discrete-optimization-with-cutting-planes<p>Previously I described how we can perform Discrete Optimization using the Branch and Bound technique. Today I want to describe another foundational technique for Discrete Optimization, Cutting Planes. Cutting Planes is like the Branch and Bound technique in that it uses a series of LP Relaxations to search for solutions. Where it is different from Branch and Bound is in how it refines the LP Formulations. Branch and Bound would subdivide the solution space by branching on a decision variable and creating two new subproblems. Instead, what Cutting Planes does is iteratively add constraints which eliminate the nonintegral solutions from the feasible space while not eliminating any feasible integral solutions. These constraints that we add are called “Cuts” because they are cutting off nonintegral solutions from the feasible space.</p>
<h2 id="the-cutting-planes-algorithm">The Cutting Planes Algorithm</h2>
<p>Conceptually the Cutting Plane Algorithm is rather simple. It is made up of the following three steps.</p>
<ul>
<li>Step 1: Solve the LP Relaxation of the Current Problem</li>
<li>Step 2: Check if the integrality requirements of the initial problem have been met. If so, terminate, an optimal solution has been found. If integrality has not been achieved proceed to Step 3.</li>
<li>Step 3: Add a constraint to the problem which removes the current optimal solution from the feasible space but does not eliminate any of the integer feasible solutions. Return to Step 1</li>
</ul>
<blockquote>
<p>Obviously you need to check for infeasibility and unboundedness as well. If either one of those conditions arise, terminate.</p>
</blockquote>
<p>That is really all there is to this algorithm. The art is in Step 3, generating cuts. A good cut will accelerate the convergence toward an integral solution. Part of the challenge is that there are frequently an infinite number of cuts which could work to reduce the size of the solution space while not eliminating integer feasible solutions. The trick then becomes finding good cuts quickly. Searching for the best cut is in of itself an optimization problem. We cannot afford to spend an infinite amount of time searching for the best cut though. There are recipes for being able to calculate good cuts quickly. We will go over some of them in a future post. For now, let’s walk through a graphical example of how Cutting Planes works.</p>
<h2 id="graphical-walkthrough">Graphical Walkthrough</h2>
<p>Let’s say that we have a Discrete Optimization problem with two constraints, Constraint A and Constraint B. These two constraints define a space in which the solution must lie. Our decision variables are $x_1$ and $x_2$. For this scenario they are integer decision variables. Here is a quick sketch of the example problem.</p>
<p><img src="../assets/2018-05-24-discrete-optimization-with-cutting-planes/2018-05-25-11-48-53.png" alt="Initial LP" /></p>
<p>This is just a conceptual walkthrough of Cutting Planes so don’t fret about exactly what the numbers are. The dotted grey lines are the boundaries of the constraints. The blue dots indicate the integer feasible solutions and the green arrow is the direction in which the objective function is pointing (the direction of greatest improvement). If we take the LP Relaxation of this problem the optimum would be at the intersection of Constraint A and Constraint B. In the following image this point is indicated by the green dot.</p>
<p><img src="../assets/2018-05-24-discrete-optimization-with-cutting-planes/2018-05-25-11-49-17.png" alt="Solution to Initial LP" /></p>
<p>We have labeled the solution to this initial LP $Z_{LP}^{0}$ . The superscript indicates which iteration this is the solution to and the $LP$ subscript indicates it is a solution to the LP Relaxation. We check if the integrality requirements of the original problem have been met. They have not since the optimal solution to the LP does not lie on integer values for $x_1$ and $x_2$. Now we need to generate a new constraint that we can add to the problem which removes $Z_{LP}^{0}$ from the feasible space but does not eliminate any of the integer feasible solutions (the blue dots).</p>
<p>Later we will go over some methods for calculating these constraints. For now let’s just use visual analysis. One of the easiest constraints that we can add is $x_2 \leq 4$. It removes $Z_{LP}^{0}$ from the feasible space but does not cut off any of our integer feasible solutions. Let’s add this constraint and redraw our problem. Let’s draw the new constraint as a yellow dotted line.</p>
<p><img src="../assets/2018-05-24-discrete-optimization-with-cutting-planes/2018-05-25-11-49-50.png" alt="Solution to LP^1" /></p>
<p>I have taken the liberty of marking the new solution $Z_{LP}^{1}$ on the diagram. The good news is that now $x_2$ has taken on an integer value but $x_1$ has not. $x_1$ lies between $2$ and $3$ so we need to add another constraint. Again we can just look at the problem and see we can add the constraint $x_1 + x_2 \leq 6$. This does not remove any integer feasible solutions but it will remove $Z_{LP}^{1}$ from the feasible space. Let’s add this constraint and see what we get.</p>
<p><img src="../assets/2018-05-24-discrete-optimization-with-cutting-planes/2018-05-25-11-54-09.png" alt="Solution to LP^2" /></p>
<p>We have added the constraint of $x_1 + x_2 \leq 6$ and found the new solution, $Z_{LP}^{2}$ . Our new solution $Z_{LP}^{2}$ lies on integral values of $x_1$ and $x_2$. We can now end our search since we have found a solution which meets the integrality requirements of the original problem. We have successfully used Cutting Planes to solve a Discrete Optimization problem!</p>
<h2 id="next-steps">Next Steps</h2>
<p>This walkthrough had some nice pictures but you should have this nagging question, “How do we generate these cuts?” In this problem it was easy to see which cuts were and were not feasible. We just chose some obvious ones based on what we could see. We need to be able to do this in much higher dimensionality though. Next time we will introduce some of the most common cuts and how we generate them. Today we just wanted to lay a conceptual foundation for how Cutting Planes worked. I hope you enjoyed the post and I always welcome feedback!</p>Matthew CrewsPreviously I described how we can perform Discrete Optimization using the Branch and Bound technique. Today I want to describe another foundational technique for Discrete Optimization, Cutting Planes. Cutting Planes is like the Branch and Bound technique in that it uses a series of LP Relaxations to search for solutions. Where it is different from Branch and Bound is in how it refines the LP Formulations. Branch and Bound would subdivide the solution space by branching on a decision variable and creating two new subproblems. Instead, what Cutting Planes does is iteratively add constraints which eliminate the nonintegral solutions from the feasible space while not eliminating any feasible integral solutions. These constraints that we add are called “Cuts” because they are cutting off nonintegral solutions from the feasible space.Discrete Optimization with Branch and Bound2018-05-22T00:00:00+00:002018-05-22T00:00:00+00:00matthewcrews.com/discrete-optimization-with-branch-and-bound<p>If you have spent any time with me you will know that I am passionate about Optimization. Now, you may pass this off as a bit of geekiness on my part but the reason I care about Optimization is that it has profound implications for how we care for people. When I get a moment to describe Optimization to someone the way I start off is by saying, “Optimization is the mathematics of caring for people.” If you care about making the world a better place for humanity, then you should care about Optimization.</p>
<p>The difficulty is that Optimization is often shrouded in mystery due to the math. My hope in this series is to clarify the mathematics of Optimization and make it more approachable. This will be the resource I wish I had when going through school. My desire is that by the end of the series you will have a firmer footing as you begin to scale the beautiful heights of Optimization and you come to enjoy it as art and as a tool for serving others.</p>
<blockquote>
<p>Note: I am not putting these posts out in a particular order. Right now I am just writing on what I am researching currently. When the series is complete I will recompile them into a more sensible order. The field is vast so I may jump around as I focus on different areas in my professional life.</p>
</blockquote>
<h2 id="integer-and-mixed-integer-programming">Integer and Mixed Integer Programming</h2>
<p>Integer Programming (IP) and Mixed Integer Programming (MIP) both fall under the umbrella of Discrete Optimization since some or all of their decision variables must take on discrete values. They are amazing tools for mathematically modeling problems. By adding decision variables which must take on integer values we can describe complex logic in ways that Linear Programming (LP) cannot. The downside to this is that the problems are far more difficult to solve. Thankfully, there are a host of algorithms which can solve even incredibly large problems in a reasonable amount of time. The performance of these algorithms is highly dependent on the quality of the implementation though.</p>
<p>One of the immediate challenges we face when trying to solve IP and MIP problems is that we cannot directly deal with the fact that some of the decision variables need to take on integral values. What most algorithms do is solve a Linear Programming relaxation of the original problem. The integral requirements of the decision variables is relaxed. This relaxed LP is solved and then constraints are added which force the integer decision variables to converge toward integral solutions. This means that we can really think of solving IP and MIP problems as recursively solving LPs where we add constraints at each recursive step. These first few posts will describe some of the algorithms in this family of solution techniques.</p>
<h2 id="branch-and-bound">Branch and Bound</h2>
<p>Branch and Bound is the most straightforward method of searching for IP/MIP solutions. We solve an initial LP Relaxation of the original problem. If we find a solution where some of the integer decision variables have taken on nonintegral values we select one to branch one, much like binary search. We end up creating two new subproblems, referred to as nodes in the search tree, which are copies of the original problem but each one has a new constraint which forces the variable we are branching on to take on an integral value.</p>
<p>For example, if we solved the LP Relaxation and the decision variable $x_1 = 1.5$ but it is supposed to be integral we can branch on $x_1$. We do this by creating two new instances of the original problem but in one of the subproblems, or node in the search tree, we add the constraint $x_1 \leq1$ and in the other subproblem , or node, we add the constraint that $x_1 \geq 2$. These new problems will no longer allow $x_1$ to take on the value of $1.5$. We now solve these new problems (nodes) and add new constraints to force other nonintegral integer variables toward integral values.</p>
<p>As we successively solve these subproblems (nodes) we may come across a solution where the integrality requirements are met. This is called the <em>incumbent solution</em>. This solution represents the best feasible solution to our original problem we have found thus far. As we continue to search we may find a better solution which also meets the integrality requirements. This improved solution becomes the new <em>incumbent solution</em>. Eventually we will prune all of the branches and the remaining <em>incumbent solution</em> is the optimal solution for the original problem. Let’s walk through a more formal description of the algorithm.</p>
<h2 id="algorithm-description">Algorithm Description</h2>
<p>Let’s put together a rough outline of how the Brand and Bound algorithm works. For that we will need some parameters.</p>
<p>$P =$ Our Initial Problem<br />
$P_{LP} =$ The LP Relaxation of our initial problem<br />
$z^\ast =$ The objective function value for $P_{LP}$<br />
$P^n =$ The $n^{th}$ subproblem of problem $P$<br />
$P^n_{LP} =$ The LP relaxation of the $P^n$ problem<br />
$z^n =$ The objective function value for the solution to $P^n$ <br />
$z_{UB}^n =$ The best upper bound on the objective function for node $n$<br />
$z_{LB}^n =$ The best lower bound on the objective function for node $n$<br />
$Z_{UB} =$ The best upper bound for the objective function observed so far<br />
$Z_{LB} =$ The best lower bound for the objective function so far</p>
<h3 id="step-0-initialize-the-problem">Step 0: Initialize the Problem</h3>
<p>Create a LP Relaxation of the original problem, $P$, and solve it. We refer to this relaxed problem as $P_{LP}$ and it will be the initial node in our tree of nodes that we search. We attempt to solve $P_{LP}$ and check for one of the following conditions.</p>
<h4 id="infeasible">Infeasible</h4>
<p>If the problem is infeasible at this stage we are done. If there is no solution to the $P_{LP}$ there is no solution to the original problem $P$.</p>
<h4 id="unbounded">Unbounded</h4>
<p>If $P_{LP}$ has no bounds then we are done. The problem is unconstrained and therefore no optimal solution exists.</p>
<h4 id="integer-solution">Integer Solution</h4>
<p>All of the integeral requirements of $P$ have been met. We are done since the solution to $P_{LP}$ and $P$ is the same.</p>
<h4 id="fractional-solution">Fractional Solution</h4>
<p>Some number of the integer variables have taken on nonintegral. We set $Z_{UB} = z^\ast$ where $z^\ast$ is the objective value for the initial problem $P_{LP}$ and $Z_{LB} = -\infty$. $Z_{LB} = -\infty$ is used to track what the best lower bound is for the original problem. We will use this value to prune nodes as we continue to search.</p>
<p>Select one of the nonintegral decision variables and branch. To branch we create two new nodes from the parent problem $P$. We make a copy of $P$ but we add a constraint to the child nodes which will force the nonintegral variable toward and integral value.</p>
<p>Let’s say that in $P$ $x_1$ is an integer decision variable. When we solve $P_{LP}$ we find that in the solution $x_1 = 1.5$. $x_1$ is supposed to take on an integral value so we decide to branch on this variable. We create a new child node $P^1$ which is the same as $P$ but with a new constraint $x_1 \leq 1$. The other child node we create is $P^2$ and is the same as $P$ but with the opposing constraint that $x_{1} \geq 2$. We add both of these child nodes to the Candidate List. We do not have an <em>incumbent solution</em> initially.</p>
<h3 id="step-1-select-a-node-from-the-candidate-list">Step 1: Select a Node from the Candidate List</h3>
<p>Now assuming that a feasible solution to $P$ was not found when we solved $P_{LP}$ then we need to choose a node to solve from the Candidate List. After Step 0 there will only be 2 nodes to the Candidate List but as we continue to iterate we will add more nodes in the Candidate List.</p>
<p>Which node we choose to solve is an algorithm design decision. We could choose the node with the best lower bound or continue down the children of the node we just solved. What some people do is do a depth first search to find a better $Z_{LB}$ to aid in pruning other nodes. The best choice is often problem dependent. Whatever strategy you employ, you will continue to evaluate nodes until the Candidate List has been emptied.</p>
<p>If at any point we arrive at Step 1 and find there are no nodes in the Candidate List yet have an <em>incumbent solution</em> we terminate the algorithm and declare the <em>incumbent solution</em> to be the optimal.</p>
<h3 id="step-2-solve-the-lp-relaxation-of-the-nth-node">Step 2: Solve the LP Relaxation of the $n^{th}$ node</h3>
<p>Based on whatever node selection rule we used in Step 1 we have chosen to solve the $P^n$ node. When you solve the LP Relaxation of the given node, $P_{LP}^n$, you will find $z^n$ which is the objective function value for $P_{LP}^n$. We then update $z_{UB}^n = z^n$. This represents the best possible objective function that could be achieved by the children of this node.</p>
<h3 id="step-3-check-for-infeasibility">Step 3: Check for Infeasibility</h3>
<p>If while solving $P_{LP}^n$ you find the solution is infeasible you can “prune” this branch. None of the children of this node will be feasible either so there is no point in continuing to search down this branch.</p>
<h3 id="step-4-check-against-z_lb">Step 4: Check against $Z_{LB}$</h3>
<p>If $z^n \leq Z_{LB}$ we can prune this branch. If we are on the first iteration of the problem though $Z_{LB} = -\infty$ so no node will be eliminated by this check. In Step 5 we update this value so eventually it will be an effective means of guiding our search down the tree.</p>
<p>Now, why can we prune based on $z^n \leq Z_{LB}$ you may ask. This is because we know that there is another branch which guarantees better solutions than the current branch. There is no point in us spending time searching down this branch because we already know we can do just as well if not better searching a somewhere else. If we have not pruned based on this test proceed to Step 5 or Step 6 depending on the condition of the solution.</p>
<h3 id="step-5-check-for-integrality">Step 5: Check for Integrality</h3>
<p>If the solution to $P_{LP}^n$ meets the integrality requirements of $P$ we have found a feasible solution. We store this new <em>incumbent solution</em> and update the value of $Z_{LB} = z^n$ if $Z_{LB} < z^n$. Again, $z^n$ is the value of the objective function for $P_{LP}^n$ which is the node we just solved. We prune this branch since it will not be possible to find a better solution. We then return to Step 1.</p>
<h3 id="step-6-branch-the-solution">Step 6: Branch the Solution</h3>
<p>If we have reached this step there are still nonintegral values for the integer decision variables so we must branch the current node $P^n$. From here we select a nonintegral decision variable to branch on and create two child nodes and add them to the Candidate List.</p>
<p>For example, let’s say that $x_2$ is an integer decision variable in problem $P$ but in the current solution, $P_{LP}^n$, we find $x_2=4.5$. We decide to branch on this decision variable since we need it to take on an integral value. We will create two new problems which are the same as our current problem $P^n$ but each has a new constraint forcing $x_2$ toward an integral value. One of the child nodes will have the constraint $x_2 \leq 4$ and the other node will have the constraint $x_2 \geq 5$. Both of these new nodes are added to the Candidate List. Loop back to Step 1 and continue.</p>
<h2 id="wrapping-up">Wrapping Up</h2>
<p>In this post I have given a quick overview of the Branch and Bound algorithm for solving IP and MIP problems. While there may be a lot of terminology the whole thing boils down to what is essentially a binary search with some rules for eliminating branches. Branch and Bound is a foundational technique for solving this class of problems. More advanced methods typically take the framework of Branch and Bound and add additional steps for speeding up convergence and strengthening bounds. In my next post I hope to provide some worked examples to illustrate how this technique works.</p>Matthew CrewsIf you have spent any time with me you will know that I am passionate about Optimization. Now, you may pass this off as a bit of geekiness on my part but the reason I care about Optimization is that it has profound implications for how we care for people. When I get a moment to describe Optimization to someone the way I start off is by saying, “Optimization is the mathematics of caring for people.” If you care about making the world a better place for humanity, then you should care about Optimization.Creating Summable Domain Types2018-05-13T00:00:00+00:002018-05-13T00:00:00+00:00matthewcrews.com/creating-summable-domain-types<p>One of the reasons that I love F# is that is makes it incredibly easy to model domains. By creating a Domain Model which represents the business domain it becomes relatively easy to create workflows and algorithms which streamline business processes. In this post I show how to create types for a domain which are summable, a feature I use frequently in my work.</p>
<h2 id="the-value-of-restricting-values">The Value of Restricting Values</h2>
<p>When I have to create a new Domain Model one of the first things that I do is define a single case Discriminated Union of <code class="language-plaintext highlighter-rouge">decimal</code> for the basic building blocks that I am going to work with (Costs, Items, Sales Rates, Days of Inventory, etc.). For example, when I am creating an algorithm to evaluate the financial viability of a product on marketplaces I have to calculate costs, I therefore create a <code class="language-plaintext highlighter-rouge">Cost</code> type. In my domain, a <code class="language-plaintext highlighter-rouge">Cost</code> is never negative therefore I can create a constructor which will enforce this behavior.</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">type</span> <span class="nc">Cost</span> <span class="p">=</span> <span class="nc">Cost</span> <span class="k">of</span> <span class="n">decimal</span> <span class="c1">// Define a single case DU 'Cost' for decimal</span>
<span class="k">module</span> <span class="nc">Cost</span> <span class="p">=</span>
<span class="k">let</span> <span class="n">create</span> <span class="n">c</span> <span class="p">=</span> <span class="c1">// Function for creating 'Cost' values</span>
<span class="k">if</span> <span class="n">c</span> <span class="o"><=</span> <span class="mi">0</span><span class="nc">M</span> <span class="k">then</span> <span class="c1">// Check that the value is greater than 0.0M</span>
<span class="nc">None</span> <span class="c1">// Return None if outside bounds</span>
<span class="k">else</span>
<span class="nc">Some</span><span class="p">(</span><span class="nc">Cost</span> <span class="n">c</span><span class="p">)</span> <span class="c1">// Return input wrapped in a 'Cost' value</span>
</code></pre></div></div>
<p>The beautiful thing about this is that when I am working with a <code class="language-plaintext highlighter-rouge">Cost</code> type I never have to worry about it being negative. This is a powerful thing when it comes to composing algorithms because I have eliminated a whole host of possible values that I would need to handle. It is amazing how easy it is for a negative numbers to sneak in and cause havoc. I force myself to deal with this bad data at the boundary of the domain instead of inside the algorithm performing the analysis.</p>
<h2 id="the-downside-where-did-addition-go">The Downside: Where Did Addition Go?</h2>
<p>There is a downside to doing this though, basic math operations will not work. At this point if I try to add two different <code class="language-plaintext highlighter-rouge">Cost</code> values I will get a compiler error.</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">totalCost</span> <span class="p">=</span> <span class="n">cost1</span> <span class="o">+</span> <span class="n">cost2</span> <span class="c1">// Error: The type 'Cost' does not support the '+' operator</span>
</code></pre></div></div>
<p>Fortunately this is easy to overcome. All we need to do is implement the <code class="language-plaintext highlighter-rouge">+</code> operator for the type. We do this by adding a <code class="language-plaintext highlighter-rouge">static member</code> to our type alias. We add the keyword <code class="language-plaintext highlighter-rouge">with</code> to the end of our previous type alias definition and provide the <code class="language-plaintext highlighter-rouge">+</code> static member.</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Updated definition of 'Cost'</span>
<span class="k">type</span> <span class="nc">Cost</span> <span class="p">=</span> <span class="nc">Cost</span> <span class="k">of</span> <span class="n">decimal</span> <span class="k">with</span>
<span class="k">static</span> <span class="k">member</span> <span class="o">(+)</span> <span class="p">(</span><span class="nc">Cost</span> <span class="n">c1</span><span class="p">,</span> <span class="nc">Cost</span> <span class="n">c2</span><span class="p">)</span> <span class="p">=</span>
<span class="nc">Cost</span> <span class="p">(</span><span class="n">c1</span> <span class="o">+</span> <span class="n">c2</span><span class="p">)</span>
</code></pre></div></div>
<p>The arguments for the <code class="language-plaintext highlighter-rouge">+</code> function may look a little odd so let me explain. By declaring the arguments of the function as <code class="language-plaintext highlighter-rouge">(Cost c1, Cost c2)</code> I am telling the compiler that I expect a <code class="language-plaintext highlighter-rouge">Cost</code> type as the input and I want you to unpack the value inside of <code class="language-plaintext highlighter-rouge">Cost</code> and put it in the <code class="language-plaintext highlighter-rouge">c1</code> and <code class="language-plaintext highlighter-rouge">c2</code> values respectively. This allows me to work with the <code class="language-plaintext highlighter-rouge">decimal</code> values inside of the <code class="language-plaintext highlighter-rouge">Cost</code> type. The function itself adds the two values together and then wraps the result in a <code class="language-plaintext highlighter-rouge">Cost</code>. Now when we go to add two <code class="language-plaintext highlighter-rouge">Cost</code> values we no longer get an error.</p>
<p>The beauty of this is that I have maintained control over the values that <code class="language-plaintext highlighter-rouge">Cost</code> can take on. I declared a <code class="language-plaintext highlighter-rouge">create</code> function which insures positive values. I only allow addition of <code class="language-plaintext highlighter-rouge">Cost</code> types which means that a <code class="language-plaintext highlighter-rouge">Cost</code> will only ever be positive. Some people may brush this off as trivial but as someone who has seen the damage that can happen from values going outside of the expected range, this extra work for reliability and peace of mind is worth it. For me, it is more efficient to ensure values cannot go outside their allowed bounds through controlling construction and operator definitions than to have value checks all over the place.</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">totalCost</span> <span class="p">=</span> <span class="n">cost1</span> <span class="o">+</span> <span class="n">cost2</span>
<span class="c1">// Result: val totalCost : Cost = Cost 15.0M</span>
</code></pre></div></div>
<h2 id="enabling-summation">Enabling Summation</h2>
<p>Well, that is great and all but what happens when we have a <code class="language-plaintext highlighter-rouge">List</code> of <code class="language-plaintext highlighter-rouge">Cost</code> values and we want to sum them. What happens then?</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">sumCosts</span> <span class="p">=</span>
<span class="p">[</span><span class="n">cost1</span><span class="p">;</span> <span class="n">cost2</span><span class="p">]</span>
<span class="p">|></span> <span class="nn">List</span><span class="p">.</span><span class="n">sum</span> <span class="c1">// Error: The type 'Cost' does not support the operator 'get_Zero'</span>
</code></pre></div></div>
<p>Now when I first came across this I was confused. I had no idea what this <code class="language-plaintext highlighter-rouge">get_Zero</code> operator meant. After digging around for a while I was able to find some examples of what it was referring to. The <code class="language-plaintext highlighter-rouge">sum</code> function wants a starting point for the summation and it gets that by calling the <code class="language-plaintext highlighter-rouge">Zero</code> function on the type. I don’t know why the compiler is saying <code class="language-plaintext highlighter-rouge">does not support the operator 'get_Zero'</code> instead of saying <code class="language-plaintext highlighter-rouge">the type does not have a function named 'Zero'</code>. Again, F# makes this easy to implement.</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Summable 'Cost' type</span>
<span class="k">type</span> <span class="nc">Cost</span> <span class="p">=</span> <span class="nc">Cost</span> <span class="k">of</span> <span class="n">decimal</span> <span class="k">with</span>
<span class="k">static</span> <span class="k">member</span> <span class="o">(+)</span> <span class="p">(</span><span class="nc">Cost</span> <span class="n">c1</span><span class="p">,</span> <span class="nc">Cost</span> <span class="n">c2</span><span class="p">)</span> <span class="p">=</span>
<span class="nc">Cost</span> <span class="p">(</span><span class="n">c1</span> <span class="o">+</span> <span class="n">c2</span><span class="p">)</span>
<span class="k">static</span> <span class="k">member</span> <span class="nc">Zero</span> <span class="p">=</span>
<span class="nc">Cost</span> <span class="mi">0</span><span class="p">.</span><span class="mi">0</span><span class="nc">M</span>
</code></pre></div></div>
<p>Now when we try to sum a list of <code class="language-plaintext highlighter-rouge">Cost</code> values we get the expected result.</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">sumCosts</span> <span class="p">=</span>
<span class="p">[</span><span class="n">cost1</span><span class="p">;</span> <span class="n">cost2</span><span class="p">]</span>
<span class="p">|></span> <span class="nn">List</span><span class="p">.</span><span class="n">sum</span>
<span class="c1">// Result: val sumCosts : Cost = Cost 15.0M</span>
</code></pre></div></div>
<h2 id="freedom-through-constraints">Freedom Through Constraints</h2>
<p>The more I dive into Domain Driven Design with F#, the more I love it. By ensuring values comply with expectations at the boundary of the domain, I am freed to reason about my algorithms without worrying about data going awry inside the domain. While it takes a few more keystrokes to define operations on these domain types, I hope that I showed you that it takes little effort in F# and can lead to more reliable and robust code. Keep calm and curry on!</p>Matthew CrewsOne of the reasons that I love F# is that is makes it incredibly easy to model domains. By creating a Domain Model which represents the business domain it becomes relatively easy to create workflows and algorithms which streamline business processes. In this post I show how to create types for a domain which are summable, a feature I use frequently in my work.F# for Optimization Modeling2018-05-04T00:00:00+00:002018-05-04T00:00:00+00:00matthewcrews.com/fsharp-for-optimization-modeling<p>I recently attended a training event hosted by Gurobi. For those who don’t know, Gurobi produces one of the best mathematical solvers in the industry. It was a great event and we were able to spend ample time with engineers and experts in the field.</p>
<p>Using a mathematical solver requires the ability to formulate models and at this time one of the easiest languages for doing that is Python. Python is a great language for many use cases. One is providing a quick and easy means of formulating models that can then be fed to a solver. I was able to spend some time with one of the engineers who implemented Gurobi’s Python library, <code class="language-plaintext highlighter-rouge">gurobipy</code>. He pointed to the formulation of the <code class="language-plaintext highlighter-rouge">netflow</code> problem as an example of how terse and concise Python could be for modeling.</p>
<p>Since I love F#, I naturally wanted to see if I could accomplish the same thing using F#. What started as a silly proof of concept is slowly turning into a more full fledged library for wrapping the Gurobi .NET library in a functional F# wrapper. Below I give an example of how the power of functions in F# allows us to nearly duplicate the functionality of Python. The library I am working on can be <a href="https://github.com/matthewcrews/fsharp-gurobi-test">found here</a>.</p>
<blockquote>
<p><strong><em>Note</em></strong> I am not saying one language is better than another. I merely like to challenge myself with formulating ideas in different languages. It forces me to translate across paradigms which I find a useful exercise for the mind.</p>
</blockquote>
<h2 id="netflow-example">Netflow Example</h2>
<p>The following shows an example of a network flow problem provided by Gurobi and modeled in Python. The full formulation can be <a href="http://www.gurobi.com/documentation/8.0/examples/netflow_py.html">found here</a>. In this example I am just comparing and contrasting the Python and F# constraint formulation methods.</p>
<blockquote>
<p><strong><em>Disclaimer</em></strong>: All Python code is copyrighted by Gurobi Optimization, LLC</p>
</blockquote>
<h3 id="creating-a-model">Creating a Model</h3>
<h4 id="python">Python</h4>
<p>In Python the creation of the model and decision variables is quite straightforward.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Copyright 2018, Gurobi Optimization, LLC
</span>
<span class="c1"># Create optimization model
</span><span class="n">m</span> <span class="o">=</span> <span class="n">Model</span><span class="p">(</span><span class="s">'netflow'</span><span class="p">)</span>
<span class="c1"># Create variables
</span><span class="n">flow</span> <span class="o">=</span> <span class="n">m</span><span class="p">.</span><span class="n">addVars</span><span class="p">(</span><span class="n">commodities</span><span class="p">,</span> <span class="n">arcs</span><span class="p">,</span> <span class="n">obj</span><span class="o">=</span><span class="n">cost</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">"flow"</span><span class="p">)</span>
</code></pre></div></div>
<h4 id="with-gurobifsharp">With Gurobi.Fsharp</h4>
<p>In F# we have a similar syntax but instead of <code class="language-plaintext highlighter-rouge">flow</code> being a <code class="language-plaintext highlighter-rouge">Dictionary</code> of decision variables indexed by tuples, we produce a <code class="language-plaintext highlighter-rouge">Map<string list, GRBDecVar></code> which is essentially the same for our purposes. I am using <code class="language-plaintext highlighter-rouge">string list</code> as the index instead of tuples because we need an indexer which has dynamic length. I could do it with tuples but it would be less straightforward.</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Create a new instance of the Gurobi Environment object</span>
<span class="c1">// to host models</span>
<span class="c1">// create: GRBEnv</span>
<span class="k">let</span> <span class="n">env</span> <span class="p">=</span> <span class="nn">Environment</span><span class="p">.</span><span class="n">create</span>
<span class="c1">// Create a new model with the environment variable</span>
<span class="c1">// create: env:GRBEnv -> name:string -> GRBModel</span>
<span class="k">let</span> <span class="n">m</span> <span class="p">=</span> <span class="nn">Model</span><span class="p">.</span><span class="n">create</span> <span class="n">env</span> <span class="s2">"netflow"</span>
<span class="c1">// Create a Map of decision variables for the model</span>
<span class="c1">// addVarsForMap: model:GRBModel -> lowerBound:float -> upperBound:float -> varType:string -> indexMap:Map<'a,float></span>
<span class="k">let</span> <span class="n">flow</span> <span class="p">=</span> <span class="nn">Model</span><span class="p">.</span><span class="n">addVarsForMap</span> <span class="n">m</span> <span class="mi">0</span><span class="p">.</span><span class="mi">0</span> <span class="nc">INF</span> <span class="nc">CONTINUOUS</span> <span class="n">costs</span>
</code></pre></div></div>
<p>Instead of using the methods on the object, functions have been provided which operate on the values that are passed in. This is more idiomatic for F#. The <code class="language-plaintext highlighter-rouge">Model</code> module in the library hosts all of the functions for working with objects of type <code class="language-plaintext highlighter-rouge">Model</code>.</p>
<p>The <code class="language-plaintext highlighter-rouge">Model.adddVarsForMap</code> function takes a <code class="language-plaintext highlighter-rouge">Map<string list, float></code> and produces a <code class="language-plaintext highlighter-rouge">Map<string list, GRBDecVar></code> for the modeler to work with. This is similar to how the Python tuples are working in the <code class="language-plaintext highlighter-rouge">gurobipy</code> library. Instead of indexing into a Python dictionary with <code class="language-plaintext highlighter-rouge">tuples</code>, F# uses a <code class="language-plaintext highlighter-rouge">string list</code> as the index.</p>
<h3 id="adding-constraints">Adding Constraints</h3>
<h4 id="python-1">Python</h4>
<p>The <code class="language-plaintext highlighter-rouge">gurobipy</code> library offers a succinct way of expressing a whole set of constraints by using generators. There is additional magic going on under the hood though that may not be obvious at first. The following method generates a set of constraints for each element in <code class="language-plaintext highlighter-rouge">arcs</code> but also creates a meaningful constraint name. The prefix for the constraint name is the last argument of the method (<code class="language-plaintext highlighter-rouge">"capacity"</code> in this instance).</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Arc capacity constraints
</span><span class="n">capacityConstraints</span> <span class="o">=</span>
<span class="n">m</span><span class="p">.</span><span class="n">addConstrs</span><span class="p">(</span>
<span class="p">(</span><span class="n">flow</span><span class="p">.</span><span class="nb">sum</span><span class="p">(</span><span class="s">'*'</span><span class="p">,</span><span class="n">i</span><span class="p">,</span><span class="n">j</span><span class="p">)</span> <span class="o"><=</span> <span class="n">capacity</span><span class="p">[</span><span class="n">i</span><span class="p">,</span><span class="n">j</span><span class="p">]</span> <span class="k">for</span> <span class="n">i</span><span class="p">,</span><span class="n">j</span> <span class="ow">in</span> <span class="n">arcs</span><span class="p">),</span> <span class="s">"capacity"</span><span class="p">)</span>
</code></pre></div></div>
<p>There is also special sauce occuring in the <code class="language-plaintext highlighter-rouge">flow.sum('*',i,j)</code> syntax. <code class="language-plaintext highlighter-rouge">flow</code> is a dictionary which is indexed by a 3 element tuple. What this <code class="language-plaintext highlighter-rouge">sum()</code> method is doing is summing across all elements in the dictionary which fit the pattern. The <code class="language-plaintext highlighter-rouge">*</code> symbol is a wildcard and will match against any element. This is a powerful way to sum across dimensions of the optimization model.</p>
<h4 id="with-gurobifsharp-1">With Gurobi.Fsharp</h4>
<p>In F# we can do something similar but instead of having a generator we pass in a lambda to create the constraints. The sinature of this function for creating the constraint set is: <code class="language-plaintext highlighter-rouge">model->string->string list->(Map<string list, Gurobi.GRBConstr)</code></p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// addConstrs: model:GRBModel -> setName:string -> setIndexes: string list list -> constraintFunc:(string list -> ConstraintTuple) -> Map<string list, GRBConstr></span>
<span class="k">let</span> <span class="n">capacityConstraints</span> <span class="p">=</span>
<span class="nn">Model</span><span class="p">.</span><span class="n">addConstrs</span> <span class="n">m</span> <span class="s2">"capacity"</span> <span class="n">arcs</span>
<span class="p">(</span><span class="k">fun</span> <span class="p">[</span><span class="n">i</span><span class="p">;</span> <span class="n">j</span><span class="p">]</span> <span class="p">-></span> <span class="p">(</span><span class="n">sum</span> <span class="n">flow</span> <span class="p">[</span><span class="s2">"*"</span><span class="p">;</span> <span class="n">i</span><span class="p">;</span> <span class="n">j</span><span class="p">]</span> <span class="o"><==</span> <span class="n">capacity</span><span class="o">.[[</span><span class="n">i</span><span class="p">;</span> <span class="n">j</span><span class="o">]]))</span>
</code></pre></div></div>
<p>The function <code class="language-plaintext highlighter-rouge">Model.addConstrs</code> takes a <code class="language-plaintext highlighter-rouge">model</code> object as its first argument (<code class="language-plaintext highlighter-rouge">m</code> in this case), the prefix for what the constraints are going to be named (<code class="language-plaintext highlighter-rouge">"capacity"</code> in this case), and the set of indices the constraints will be created over, <code class="language-plaintext highlighter-rouge">arcs</code> in this case. The key point is that the types of the indices must match the input type of the lambda.</p>
<p>The <code class="language-plaintext highlighter-rouge">addConstrs</code> function will iterate through each of the indices in the set, create a constraint from the lambda that was passed, and name the constraint appropriatly. If the first element of the <code class="language-plaintext highlighter-rouge">arcs</code> set was <code class="language-plaintext highlighter-rouge">["Detroit"; "Boston"]</code> then the name of the first constraint would be <code class="language-plaintext highlighter-rouge">capacity_Detroit_Boston</code>. This helps the modeler by maintaining a consistent naming scheme for the constraints in the model.</p>Matthew CrewsI recently attended a training event hosted by Gurobi. For those who don’t know, Gurobi produces one of the best mathematical solvers in the industry. It was a great event and we were able to spend ample time with engineers and experts in the field.Domains Run Amok2018-01-27T00:00:00+00:002018-01-27T00:00:00+00:00matthewcrews.com/domains-run-amok<p>I am a huge fan of Domain Driven Design and I have been trying to apply it more and more. I ran into a problem last week that kept beating me over the head though. I kept using a bottom up approach and kept coming up with terrible solutions. Finally, I took a more outside to in approach which cleaned up the solution. I credit <a href="http://blog.ploeh.dk/">Mark Seemann</a> for the idea to work from the outside in. I am wanting to show some of the difficulties you can run into using a bottom up approach so that others don’t make the same mistakes that I did. Hopefully this little exercise helps provide others some guidance on how to get unstuck when attempting Domain Driven Design.</p>
<h2 id="our-refactoring-problem">Our Refactoring Problem</h2>
<p>I have a project where we are rebuilding how we calculate the replenishment logic for our Supply Chain. Replenishment is the process of ordering product from Vendors for your Warehouses so that we can fill customer orders. I work for an e-commerce company so Replenishment is at the heart of what we do.</p>
<p>The current solution is a monolith application which is all fed from an Azure SQL instance. It is comprised of a large set of batch process that run in order and populate tables in the database. This mess was inherited from an old system and has been warped beyond comprehension at this point. It is so fragile we don’t dare touch it. The plan is to decompose the monolith into separate services which communicate via messages. To do this though, we need to create those separate services. At the heart of one of those services is the analysis of Time Series data. This is my attempt to create a tiny little domain for modeling this analysis and the mistakes I made along the way.</p>
<h2 id="modeling-timeseries-take-1-from-the-bottom-up">Modeling TimeSeries Take 1: From the Bottom Up</h2>
<p>All of our Replenishment logic is built on analyzing Time Series data. This data can be thought of as a array of tuples where one value is the timestamp and the other is the observed value, <code class="language-plaintext highlighter-rouge">DateTimeOffset * 'a</code>.</p>
<p>What I set out to do is create a domain model that allows us to analyze these Time Series in a robust and performant way. My initial thought was, “I know that my data will always be <code class="language-plaintext highlighter-rouge">Decimal</code> or <code class="language-plaintext highlighter-rouge">String</code> so I can think of an <code class="language-plaintext highlighter-rouge">ObservedValue</code> in my Time Series as a Discriminated Union and an <code class="language-plaintext highlighter-rouge">Observation</code> is a record with a <code class="language-plaintext highlighter-rouge">DateTimeOffset</code> and an <code class="language-plaintext highlighter-rouge">ObservedValue</code>. A <code class="language-plaintext highlighter-rouge">TimeSeries</code> is just an array of the type <code class="language-plaintext highlighter-rouge">Observation</code>. When I am done with an analysis the result will be either <code class="language-plaintext highlighter-rouge">decimal</code> or <code class="language-plaintext highlighter-rouge">string</code> so I’ll define an <code class="language-plaintext highlighter-rouge">AnalysisResult</code> type to contain the result.”</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">type</span> <span class="nc">ObservedValue</span> <span class="p">=</span>
<span class="p">|</span> <span class="nc">Decimal</span> <span class="k">of</span> <span class="n">decimal</span>
<span class="p">|</span> <span class="nc">String</span> <span class="k">of</span> <span class="kt">string</span>
<span class="k">type</span> <span class="nc">Observation</span> <span class="p">=</span> <span class="p">{</span>
<span class="nc">DateTime</span> <span class="p">:</span> <span class="nc">DateTimeOffset</span>
<span class="nc">Value</span> <span class="p">:</span> <span class="nc">ObservedValue</span>
<span class="p">}</span>
<span class="k">type</span> <span class="nc">TimeSeries</span> <span class="p">=</span> <span class="kt">array</span><span class="p"><</span><span class="nc">Observation</span><span class="p">></span>
<span class="k">type</span> <span class="nc">AnalysisResult</span> <span class="p">=</span>
<span class="p">|</span> <span class="nc">Decimal</span> <span class="k">of</span> <span class="n">decimal</span>
<span class="p">|</span> <span class="nc">String</span> <span class="k">of</span> <span class="kt">string</span>
</code></pre></div></div>
<p>This doesn’t seem bad so far. Now I need to add some basic functions for analyzing my <code class="language-plaintext highlighter-rouge">TimeSeries</code>. Some simple and obvious ones are <code class="language-plaintext highlighter-rouge">mean</code>, <code class="language-plaintext highlighter-rouge">first</code>, and <code class="language-plaintext highlighter-rouge">last</code>. There are actually many functions I will need but these will suffice to make my point. I now try to write these simple functions for my <code class="language-plaintext highlighter-rouge">TimeSeries</code> type.</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="nc">TimeSeries</span> <span class="p">=</span>
<span class="k">let</span> <span class="k">private</span> <span class="n">create</span> <span class="n">observedType</span> <span class="n">t</span> <span class="p">:</span> <span class="nc">TimeSeries</span> <span class="p">=</span>
<span class="n">t</span>
<span class="p">|></span> <span class="nn">Seq</span><span class="p">.</span><span class="n">map</span> <span class="p">(</span><span class="k">fun</span> <span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span> <span class="p">-></span> <span class="p">{</span><span class="nc">DateTime</span> <span class="p">=</span> <span class="n">t</span><span class="p">;</span> <span class="nc">Value</span> <span class="p">=</span> <span class="n">observedType</span> <span class="n">v</span><span class="o">})</span>
<span class="p">|></span> <span class="nn">Seq</span><span class="p">.</span><span class="n">toArray</span>
<span class="k">let</span> <span class="n">fromDecimal</span> <span class="n">s</span> <span class="p">:</span> <span class="nc">TimeSeries</span> <span class="p">=</span>
<span class="n">create</span> <span class="nn">ObservedValue</span><span class="p">.</span><span class="nc">Decimal</span> <span class="n">s</span>
<span class="k">let</span> <span class="n">fromString</span> <span class="n">s</span> <span class="p">:</span> <span class="nc">TimeSeries</span> <span class="p">=</span>
<span class="n">create</span> <span class="nn">ObservedValue</span><span class="p">.</span><span class="nc">String</span> <span class="n">s</span>
<span class="k">let</span> <span class="n">first</span> <span class="p">(</span><span class="n">ts</span> <span class="p">:</span> <span class="nc">TimeSeries</span><span class="p">)</span> <span class="p">=</span>
<span class="n">ts</span><span class="o">.[</span><span class="mi">0</span><span class="o">].</span><span class="nc">Value</span>
<span class="k">let</span> <span class="n">last</span> <span class="p">(</span><span class="n">ts</span> <span class="p">:</span> <span class="nc">TimeSeries</span><span class="p">)</span> <span class="p">=</span>
<span class="n">ts</span><span class="o">.[-</span><span class="mi">1</span><span class="o">].</span><span class="nc">Value</span>
<span class="k">let</span> <span class="n">mean</span> <span class="p">(</span><span class="n">ts</span> <span class="p">:</span> <span class="nc">TimeSeries</span><span class="p">)</span> <span class="p">=</span>
<span class="n">ts</span>
<span class="p">|></span> <span class="nn">Array</span><span class="p">.</span><span class="n">averageBy</span> <span class="p">(</span><span class="k">fun</span> <span class="n">x</span> <span class="p">-></span> <span class="n">x</span><span class="p">.</span><span class="nc">Value</span><span class="p">)</span> <span class="c1">// Error: The type ObservedValue does not support the operator '+'</span>
</code></pre></div></div>
<p>I have encountered my first problem with this approach. I want to be able to take the <code class="language-plaintext highlighter-rouge">mean</code> of my <code class="language-plaintext highlighter-rouge">TimeSeries</code> but the <code class="language-plaintext highlighter-rouge">ObservedValue</code> type does not support the <code class="language-plaintext highlighter-rouge">+</code> operator. I think, “No problem, I’ll just add the <code class="language-plaintext highlighter-rouge">+</code> operator.” I then look at the type again and realize I may be doing something wrong. Adding a <code class="language-plaintext highlighter-rouge">decimal</code> to a <code class="language-plaintext highlighter-rouge">decimal</code> makes sense and I also understand adding <code class="language-plaintext highlighter-rouge">string</code> to <code class="language-plaintext highlighter-rouge">string</code> but this is going to require me to have a <code class="language-plaintext highlighter-rouge">+</code> defined for <code class="language-plaintext highlighter-rouge">decimal</code> to <code class="language-plaintext highlighter-rouge">string</code> and <code class="language-plaintext highlighter-rouge">string</code> to <code class="language-plaintext highlighter-rouge">decimal</code>. That does not make any sense.</p>
<h2 id="modeling-timeseries-take-2-homogenous-values">Modeling TimeSeries Take 2: Homogenous Values</h2>
<p>My problem is that I am allowing a single <code class="language-plaintext highlighter-rouge">TimeSeries</code> to be heterogenous, containing both <code class="language-plaintext highlighter-rouge">decimal</code> and <code class="language-plaintext highlighter-rouge">string</code> values. Really a single <code class="language-plaintext highlighter-rouge">TimeSeries</code> needs to be homogeneous, containing only <code class="language-plaintext highlighter-rouge">decimal</code> or only <code class="language-plaintext highlighter-rouge">string</code>. Okay, no problem! I’ll reformulate the domain to have the <code class="language-plaintext highlighter-rouge">TimeSeries</code> be a Discriminated Union instead of the <code class="language-plaintext highlighter-rouge">ObservedValue</code>.</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">open</span> <span class="nc">System</span>
<span class="k">type</span> <span class="nc">Observation</span><span class="p"><</span><span class="k">'</span><span class="n">a</span><span class="p">></span> <span class="p">=</span> <span class="p">{</span>
<span class="nc">DateTime</span> <span class="p">:</span> <span class="nc">DateTimeOffset</span>
<span class="nc">Value</span> <span class="p">:</span> <span class="k">'</span><span class="n">a</span>
<span class="p">}</span>
<span class="k">type</span> <span class="nc">TimeSeries</span> <span class="p">=</span>
<span class="p">|</span> <span class="nc">Decimal</span> <span class="k">of</span> <span class="kt">array</span><span class="p"><</span><span class="nc">Observation</span><span class="p"><</span><span class="n">decimal</span><span class="o">>></span>
<span class="p">|</span> <span class="nc">String</span> <span class="k">of</span> <span class="kt">array</span><span class="p"><</span><span class="nc">Observation</span><span class="p"><</span><span class="kt">string</span><span class="o">>></span>
<span class="k">type</span> <span class="nc">AnalysisResult</span> <span class="p">=</span>
<span class="p">|</span> <span class="nc">Decimal</span> <span class="k">of</span> <span class="n">decimal</span>
<span class="p">|</span> <span class="nc">String</span> <span class="k">of</span> <span class="kt">string</span>
</code></pre></div></div>
<p>Now let’s try to implement our analysis functions again. Don’t judge me for what you see next. Once I wrote it, I felt a little ill. I’ll go into why after the code.</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="nc">TimeSeries</span> <span class="p">=</span>
<span class="k">let</span> <span class="k">private</span> <span class="n">create</span> <span class="n">observedType</span> <span class="n">t</span> <span class="p">:</span> <span class="nc">TimeSeries</span> <span class="p">=</span>
<span class="n">t</span>
<span class="p">|></span> <span class="nn">Seq</span><span class="p">.</span><span class="n">map</span> <span class="p">(</span><span class="k">fun</span> <span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span> <span class="p">-></span> <span class="p">{</span><span class="nc">DateTime</span> <span class="p">=</span> <span class="n">t</span><span class="p">;</span> <span class="nc">Value</span> <span class="p">=</span> <span class="n">v</span><span class="o">})</span>
<span class="p">|></span> <span class="nn">Seq</span><span class="p">.</span><span class="n">toArray</span>
<span class="p">|></span> <span class="n">observedType</span>
<span class="k">let</span> <span class="n">fromDecimal</span> <span class="n">t</span> <span class="p">=</span>
<span class="n">create</span> <span class="nn">TimeSeries</span><span class="p">.</span><span class="nc">Decimal</span> <span class="n">t</span>
<span class="k">let</span> <span class="n">fromString</span> <span class="n">t</span> <span class="p">=</span>
<span class="n">create</span> <span class="nn">TimeSeries</span><span class="p">.</span><span class="nc">String</span> <span class="n">t</span>
<span class="k">let</span> <span class="k">private</span> <span class="n">map</span> <span class="n">df</span> <span class="n">sf</span> <span class="n">ts</span> <span class="p">=</span>
<span class="k">match</span> <span class="n">ts</span> <span class="k">with</span>
<span class="p">|</span> <span class="nn">TimeSeries</span><span class="p">.</span><span class="nc">Decimal</span> <span class="n">t</span> <span class="p">-></span> <span class="n">df</span> <span class="n">t</span>
<span class="p">|</span> <span class="nn">TimeSeries</span><span class="p">.</span><span class="nc">String</span> <span class="n">t</span> <span class="p">-></span> <span class="n">sf</span> <span class="n">t</span>
<span class="k">let</span> <span class="n">first</span> <span class="p">(</span><span class="n">ts</span> <span class="p">:</span> <span class="nc">TimeSeries</span><span class="p">)</span> <span class="p">=</span>
<span class="k">let</span> <span class="n">f</span> <span class="p">=</span> <span class="k">fun</span> <span class="p">(</span><span class="n">t</span> <span class="p">:</span> <span class="kt">array</span><span class="p"><</span><span class="nc">Observation</span><span class="p"><</span><span class="k">'</span><span class="n">a</span><span class="o">>>)</span> <span class="p">-></span> <span class="n">t</span><span class="o">.[</span><span class="mi">0</span><span class="o">].</span><span class="nc">Value</span>
<span class="n">map</span> <span class="p">(</span><span class="n">f</span> <span class="o">>></span> <span class="nn">AnalysisResult</span><span class="p">.</span><span class="nc">Decimal</span><span class="p">)</span> <span class="p">(</span><span class="n">f</span> <span class="o">>></span> <span class="nn">AnalysisResult</span><span class="p">.</span><span class="nc">String</span><span class="p">)</span> <span class="n">ts</span>
<span class="k">let</span> <span class="n">last</span> <span class="p">(</span><span class="n">ts</span> <span class="p">:</span> <span class="nc">TimeSeries</span><span class="p">)</span> <span class="p">=</span>
<span class="k">let</span> <span class="n">f</span> <span class="p">=</span> <span class="k">fun</span> <span class="p">(</span><span class="n">t</span> <span class="p">:</span> <span class="kt">array</span><span class="p"><</span><span class="nc">Observation</span><span class="p"><</span><span class="k">'</span><span class="n">a</span><span class="o">>>)</span> <span class="p">-></span> <span class="n">t</span><span class="o">.[-</span><span class="mi">1</span><span class="o">].</span><span class="nc">Value</span>
<span class="n">map</span> <span class="p">(</span><span class="n">f</span> <span class="o">>></span> <span class="nn">AnalysisResult</span><span class="p">.</span><span class="nc">Decimal</span><span class="p">)</span> <span class="p">(</span><span class="n">f</span> <span class="o">>></span> <span class="nn">AnalysisResult</span><span class="p">.</span><span class="nc">String</span><span class="p">)</span> <span class="n">ts</span>
<span class="k">let</span> <span class="n">mean</span> <span class="p">(</span><span class="n">ts</span> <span class="p">:</span> <span class="nc">TimeSeries</span><span class="p">)</span> <span class="p">=</span>
<span class="k">let</span> <span class="n">df</span> <span class="p">=</span>
<span class="k">fun</span> <span class="p">(</span><span class="n">t</span> <span class="p">:</span> <span class="kt">array</span><span class="p"><</span><span class="nc">Observation</span><span class="p"><</span><span class="n">decimal</span><span class="o">>>)</span> <span class="p">-></span>
<span class="n">t</span> <span class="p">|></span> <span class="nn">Array</span><span class="p">.</span><span class="n">averageBy</span> <span class="p">(</span><span class="k">fun</span> <span class="n">x</span> <span class="p">-></span> <span class="n">x</span><span class="p">.</span><span class="nc">Value</span><span class="p">)</span> <span class="p">|></span> <span class="nn">AnalysisResult</span><span class="p">.</span><span class="nc">Decimal</span>
<span class="k">let</span> <span class="n">sf</span> <span class="p">=</span>
<span class="k">fun</span> <span class="p">(</span><span class="n">t</span> <span class="p">:</span> <span class="kt">array</span><span class="p"><</span><span class="nc">Observation</span><span class="p"><</span><span class="kt">string</span><span class="o">>>)</span> <span class="p">-></span> <span class="s2">""</span> <span class="p">|></span> <span class="nn">AnalysisResult</span><span class="p">.</span><span class="nc">String</span>
<span class="n">map</span> <span class="n">df</span> <span class="n">sf</span> <span class="n">ts</span>
</code></pre></div></div>
<p>I will readily admit this is clunky. Let me explain the thought process. I know that the <code class="language-plaintext highlighter-rouge">TimeSeries</code> type is a Discriminated Union and therefore I should have a <code class="language-plaintext highlighter-rouge">map</code> like function for easily applying the correct function, depending on which value <code class="language-plaintext highlighter-rouge">TimeSeries</code> takes on. In many cases I would use the exact same logic (Ex: <code class="language-plaintext highlighter-rouge">first</code> and <code class="language-plaintext highlighter-rouge">last</code>) so I just defined a generic function and used that for both arguments of the <code class="language-plaintext highlighter-rouge">map</code> function.</p>
<p>When I get to the <code class="language-plaintext highlighter-rouge">mean</code> function I run into another problem. It does not make sense to take the <code class="language-plaintext highlighter-rouge">mean</code> of a set of <code class="language-plaintext highlighter-rouge">string</code> observations but the code allows it. In this code I am returning an empty <code class="language-plaintext highlighter-rouge">string</code> but that is not in line with the heart of what I am going for. If something does not make sense, I don’t want to allow it. I want invalid states to be unrepresentable in the code. I don’t want myself or someone else to even be able to call <code class="language-plaintext highlighter-rouge">mean</code> with a <code class="language-plaintext highlighter-rouge">TimeSeries</code> containing <code class="language-plaintext highlighter-rouge">string</code> values.</p>
<p>It’s at this point I start to feel really dumb. How can this be so hard? Here is what I am wanting to accomplish:</p>
<ul>
<li>Model a TimeSeries made up of either <code class="language-plaintext highlighter-rouge">decimal</code> or <code class="language-plaintext highlighter-rouge">string</code></li>
<li>Reuse function logic wherever I can (DRY principle)</li>
<li>Prevent unrepresentable states</li>
</ul>
<h2 id="modeling-timeseries-take-3-generic-timeseries">Modeling TimeSeries Take 3: Generic TimeSeries</h2>
<p>I would rather not admit how long I was stumped at this point. It felt like I was missing something glaringly obvious. I mulled on this problem for awhile until the next thought came to me, “What is really going on is that I have two special cases of <code class="language-plaintext highlighter-rouge">TimeSeries<'a></code> here. I have a <code class="language-plaintext highlighter-rouge">TimeSeries<decimal></code> and a <code class="language-plaintext highlighter-rouge">TimeSeries<string></code>. Why not have a full set of functions for <code class="language-plaintext highlighter-rouge">TimeSeries<'a></code> and then have two different types for the <code class="language-plaintext highlighter-rouge">TimeSeries<decimal></code> case and the <code class="language-plaintext highlighter-rouge">TimeSeries<string></code> case which only have a subset of the functions available?” Here is what I came up with.</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">open</span> <span class="nc">System</span>
<span class="k">type</span> <span class="nc">Observation</span><span class="p"><</span><span class="k">'</span><span class="n">a</span><span class="p">></span> <span class="p">=</span> <span class="p">{</span>
<span class="nc">DateTime</span> <span class="p">:</span> <span class="nc">DateTimeOffset</span>
<span class="nc">Value</span> <span class="p">:</span> <span class="k">'</span><span class="n">a</span>
<span class="p">}</span>
<span class="k">type</span> <span class="nc">TimeSeries</span><span class="p"><</span><span class="k">'</span><span class="n">a</span><span class="p">></span> <span class="p">=</span> <span class="kt">array</span><span class="p"><</span><span class="nc">Observation</span><span class="p"><</span><span class="k">'</span><span class="n">a</span><span class="o">>></span>
<span class="k">type</span> <span class="nc">DecimalSeries</span> <span class="p">=</span> <span class="nc">TimeSeries</span><span class="p"><</span><span class="n">decimal</span><span class="p">></span>
<span class="k">type</span> <span class="nc">StringSeries</span> <span class="p">=</span> <span class="nc">TimeSeries</span><span class="p"><</span><span class="kt">string</span><span class="p">></span>
<span class="k">type</span> <span class="nc">AnalysisResult</span> <span class="p">=</span>
<span class="p">|</span> <span class="nc">Decimal</span> <span class="k">of</span> <span class="n">decimal</span>
<span class="p">|</span> <span class="nc">String</span> <span class="k">of</span> <span class="kt">string</span>
<span class="k">module</span> <span class="nc">TimeSeries</span> <span class="p">=</span>
<span class="k">let</span> <span class="k">private</span> <span class="n">create</span> <span class="n">t</span> <span class="p">:</span> <span class="nc">TimeSeries</span><span class="p"><</span><span class="k">'</span><span class="n">a</span><span class="o">>=</span>
<span class="n">t</span>
<span class="p">|></span> <span class="nn">Seq</span><span class="p">.</span><span class="n">map</span> <span class="p">(</span><span class="k">fun</span> <span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span> <span class="p">-></span> <span class="p">{</span><span class="nc">DateTime</span> <span class="p">=</span> <span class="n">t</span><span class="p">;</span> <span class="nc">Value</span> <span class="p">=</span> <span class="n">v</span><span class="p">}</span> <span class="p">)</span>
<span class="p">|></span> <span class="nn">Seq</span><span class="p">.</span><span class="n">toArray</span>
<span class="k">let</span> <span class="k">private</span> <span class="n">first</span> <span class="p">(</span><span class="n">ts</span> <span class="p">:</span> <span class="nc">TimeSeries</span><span class="p"><</span><span class="k">'</span><span class="n">a</span><span class="o">>)</span> <span class="p">=</span>
<span class="n">ts</span><span class="o">.[</span><span class="mi">0</span><span class="o">].</span><span class="nc">Value</span>
<span class="k">let</span> <span class="k">private</span> <span class="n">last</span> <span class="p">(</span><span class="n">ts</span> <span class="p">:</span> <span class="nc">TimeSeries</span><span class="p"><</span><span class="k">'</span><span class="n">a</span><span class="o">>)</span> <span class="p">=</span>
<span class="n">ts</span><span class="o">.[-</span><span class="mi">1</span><span class="o">].</span><span class="nc">Value</span>
<span class="k">let</span> <span class="k">inline</span> <span class="k">private</span> <span class="n">mean</span> <span class="p">(</span><span class="n">ts</span> <span class="p">:</span> <span class="nc">TimeSeries</span><span class="p"><</span><span class="k">'</span><span class="n">a</span><span class="o">>)</span> <span class="p">=</span>
<span class="n">ts</span>
<span class="p">|></span> <span class="nn">Array</span><span class="p">.</span><span class="n">averageBy</span> <span class="p">(</span><span class="k">fun</span> <span class="n">x</span> <span class="p">-></span> <span class="n">x</span><span class="p">.</span><span class="nc">Value</span><span class="p">)</span>
<span class="k">module</span> <span class="nc">DecimalSeries</span> <span class="p">=</span>
<span class="k">let</span> <span class="n">create</span> <span class="n">t</span> <span class="p">:</span> <span class="nc">DecimalSeries</span> <span class="p">=</span>
<span class="n">create</span> <span class="n">t</span>
<span class="k">let</span> <span class="n">first</span> <span class="p">(</span><span class="n">ds</span> <span class="p">:</span> <span class="nc">DecimalSeries</span><span class="p">)</span> <span class="p">=</span>
<span class="n">first</span> <span class="n">ds</span> <span class="p">|></span> <span class="nn">AnalysisResult</span><span class="p">.</span><span class="nc">Decimal</span>
<span class="k">let</span> <span class="n">last</span> <span class="p">(</span><span class="n">ds</span> <span class="p">:</span> <span class="nc">DecimalSeries</span><span class="p">)</span> <span class="p">=</span>
<span class="n">last</span> <span class="n">ds</span> <span class="p">|></span> <span class="nn">AnalysisResult</span><span class="p">.</span><span class="nc">Decimal</span>
<span class="k">let</span> <span class="n">mean</span> <span class="p">(</span><span class="n">ds</span> <span class="p">:</span> <span class="nc">DecimalSeries</span><span class="p">)</span> <span class="p">=</span>
<span class="n">mean</span> <span class="n">ds</span> <span class="p">|></span> <span class="nn">AnalysisResult</span><span class="p">.</span><span class="nc">Decimal</span>
<span class="k">module</span> <span class="nc">StringSeries</span> <span class="p">=</span>
<span class="k">let</span> <span class="n">create</span> <span class="n">t</span> <span class="p">:</span> <span class="nc">StringSeries</span> <span class="p">=</span>
<span class="n">create</span> <span class="n">t</span>
<span class="k">let</span> <span class="n">first</span> <span class="p">(</span><span class="n">ds</span> <span class="p">:</span> <span class="nc">StringSeries</span><span class="p">)</span> <span class="p">=</span>
<span class="n">first</span> <span class="n">ds</span> <span class="p">|></span> <span class="nn">AnalysisResult</span><span class="p">.</span><span class="nc">String</span>
<span class="k">let</span> <span class="n">last</span> <span class="p">(</span><span class="n">ds</span> <span class="p">:</span> <span class="nc">StringSeries</span><span class="p">)</span> <span class="p">=</span>
<span class="n">last</span> <span class="n">ds</span> <span class="p">|></span> <span class="nn">AnalysisResult</span><span class="p">.</span><span class="nc">String</span>
</code></pre></div></div>
<p>One thing to note, I had to add the keyword <code class="language-plaintext highlighter-rouge">inline</code> to the <code class="language-plaintext highlighter-rouge">mean</code> function in the <code class="language-plaintext highlighter-rouge">TimeSeries</code> module. This makes the compiler figure out the types at the point the function is used. Now I am still not really proud of this code yet but it is accomplishing most of my goals. I am getting code reuse while being able to control which functions can be used by which type of <code class="language-plaintext highlighter-rouge">TimeSeries</code>. Since I only define a <code class="language-plaintext highlighter-rouge">create</code> function for the <code class="language-plaintext highlighter-rouge">DecimalSeries</code> and <code class="language-plaintext highlighter-rouge">StringSeries</code> types, I don’t have to fear someone creating a random <code class="language-plaintext highlighter-rouge">TimeSeries<'a></code> if they follow the convention of using the <code class="language-plaintext highlighter-rouge">create</code> function. The functions for <code class="language-plaintext highlighter-rouge">TimeSeries</code> are also private and can only be called from the sub-modules <code class="language-plaintext highlighter-rouge">DecimalSeries</code> and <code class="language-plaintext highlighter-rouge">StringSeries</code>.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I hope my failures prove useful and an encouragement to others wandering through the process of learning Domain Driven Design. This was just one small problem that made me feel rather silly as I wrestled with it. Maybe I will come up with a more elegant solution but as of now, I like the code reuse and guarantees this is providing me. If you have a better solution, please message me on Twitter (@McCrews). When you get stuck coding, remember most of progress feels like wandering down dark halls until you come to the light. Keep calm and curry on!</p>Matthew CrewsI am a huge fan of Domain Driven Design and I have been trying to apply it more and more. I ran into a problem last week that kept beating me over the head though. I kept using a bottom up approach and kept coming up with terrible solutions. Finally, I took a more outside to in approach which cleaned up the solution. I credit Mark Seemann for the idea to work from the outside in. I am wanting to show some of the difficulties you can run into using a bottom up approach so that others don’t make the same mistakes that I did. Hopefully this little exercise helps provide others some guidance on how to get unstuck when attempting Domain Driven Design.The Divide Operator is a Lie2018-01-14T00:00:00+00:002018-01-14T00:00:00+00:00matthewcrews.com/the-divide-operator-is-a-lie<p>One of the things that most attracted me to F# is the ability to accurately model your domain. What first turned me on to this was a talk by <a href="https://www.youtube.com/watch?v=E8I19uA-wGY&t=1102s">Scott Wlaschin on Functional programming design patterns</a>. Scott has a more focused talk on <a href="https://www.youtube.com/watch?v=Up7LcbGZFuo&t=229s">Domain Modeling Made Functional</a> that he did a few years later and a <a href="https://fsharpforfunandprofit.com/books/">book with the same title</a>. This whole concept was blowing my mind. The idea of modeling your domain such that illegal states are unrepresentable sounds immensely satisfying to me.</p>
<p>This new way of looking at the world has been slowly transforming all of my code. Everywhere I look now I am asking, “Is it possible for this state to be illegal? What can I do to ensure I am covering all scenarios?” With this new focus I quickly came across an operator in F# that lies, the division operator.</p>
<h2 id="the-divide-lie">The Divide Lie</h2>
<p>If you hover over the / operator in Visual Studio you will get the following function signature</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span><span class="o">(/):</span> <span class="n">x</span><span class="p">:</span><span class="k">'</span><span class="nc">T1</span> <span class="p">-></span> <span class="n">y</span><span class="p">:</span><span class="k">'</span><span class="nc">T2</span> <span class="p">-></span> <span class="k">'</span><span class="nc">T3</span> <span class="p">(</span><span class="n">requires</span> <span class="k">member</span> <span class="o">(/))</span>
</code></pre></div></div>
<p>There is nothing surprising here. The <code class="language-plaintext highlighter-rouge">/</code> operator is expecting two values and will produce a third. Now let’s look at what the compiler says is supposed to happen when we divide two decimals. If I input the following lines into a fsx script in Visual Studio I will get the following types from the compiler.</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">a</span> <span class="p">=</span> <span class="mi">10</span><span class="nc">M</span> <span class="c1">// val a : decimal</span>
<span class="k">let</span> <span class="n">b</span> <span class="p">=</span> <span class="mi">5</span><span class="nc">M</span> <span class="c1">// val b : decimal</span>
<span class="k">let</span> <span class="n">c</span> <span class="p">=</span> <span class="n">a</span> <span class="o">/</span> <span class="n">b</span> <span class="c1">// val c : decimal</span>
</code></pre></div></div>
<p>This is where my problem is. The compiler says that taking two decimal values and dividing them will produce a third decimal value. This is not always the case though. If <code class="language-plaintext highlighter-rouge">b = 0M</code> then this will throw an exception. This runs counter to the idea of making illegal states unrepresentable. We would rather that the operator returned <code class="language-plaintext highlighter-rouge">'T option</code> which would force us to deal with both scenarios.</p>
<h2 id="defining-a-new-operator">Defining a new Operator</h2>
<p>Fortunately for us, it is easy to add operators to F# but there are a couple of gotchas I will cover here. The F# Language Reference has a great page describing the rules around <a href="https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/operator-overloading">Operator Overloading</a>. The key thing to know is that there are a limited set of characters that are permitted: <code class="language-plaintext highlighter-rouge">!</code>, <code class="language-plaintext highlighter-rouge">%</code>, <code class="language-plaintext highlighter-rouge">&</code>, <code class="language-plaintext highlighter-rouge">*</code>, <code class="language-plaintext highlighter-rouge">+</code>, <code class="language-plaintext highlighter-rouge">-</code>, <code class="language-plaintext highlighter-rouge">.</code>, <code class="language-plaintext highlighter-rouge">/</code>, <code class="language-plaintext highlighter-rouge"><</code>, <code class="language-plaintext highlighter-rouge">=</code>, <code class="language-plaintext highlighter-rouge">></code>, <code class="language-plaintext highlighter-rouge">?</code>, <code class="language-plaintext highlighter-rouge">@</code>, <code class="language-plaintext highlighter-rouge">^</code>, <code class="language-plaintext highlighter-rouge">|</code>, and <code class="language-plaintext highlighter-rouge">~</code>. <code class="language-plaintext highlighter-rouge">~</code> is a special character to be used when making a unary operator. In this case, I need a binary operator so I will avoid using it.</p>
<p>I want to create a new divide operator that will check if the divisor is <code class="language-plaintext highlighter-rouge">0</code>. If the divisor is equivalent to <code class="language-plaintext highlighter-rouge">0</code>, I want the operator to return <code class="language-plaintext highlighter-rouge">None</code>. Since I want this to be intuitive when looking at the operator I will combine the divide symbol, <code class="language-plaintext highlighter-rouge">/</code>, with the bang symbol, <code class="language-plaintext highlighter-rouge">!</code>, to make my new operator <code class="language-plaintext highlighter-rouge">/!</code>. The reason I am using the <code class="language-plaintext highlighter-rouge">!</code> symbol is because it often indicates a warning which is what I am wanting to communicate to the developer. This means my function signature needs to look like this:</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span><span class="o">(/!):</span> <span class="n">x</span><span class="p">:</span><span class="k">'</span><span class="nc">T1</span> <span class="p">-></span> <span class="n">y</span><span class="p">:</span><span class="k">'</span><span class="nc">T2</span> <span class="p">-></span> <span class="k">'</span><span class="nc">T3</span> <span class="n">option</span> <span class="p">(</span><span class="n">requires</span> <span class="k">member</span> <span class="o">(/))</span>
</code></pre></div></div>
<p>My first attempt looked like the following:</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="o">(/!)</span> <span class="n">a</span> <span class="n">b</span> <span class="p">=</span>
<span class="k">match</span> <span class="n">b</span> <span class="p"><></span> <span class="mi">0</span> <span class="k">with</span>
<span class="p">|</span> <span class="bp">true</span> <span class="p">-></span> <span class="n">a</span> <span class="o">/</span> <span class="n">b</span> <span class="p">|></span> <span class="nc">Some</span>
<span class="p">|</span> <span class="bp">false</span> <span class="p">-></span> <span class="nc">None</span>
</code></pre></div></div>
<p>When I look at the function signature of my operator though I see the following:</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span><span class="o">(/!):</span> <span class="n">x</span><span class="p">:</span><span class="kt">int</span> <span class="p">-></span> <span class="n">y</span><span class="p">:</span><span class="kt">int</span> <span class="p">-></span> <span class="kt">int</span> <span class="n">option</span>
</code></pre></div></div>
<p>This is no good. This will only work with inputs of <code class="language-plaintext highlighter-rouge">int</code> and I am wanting something that is generic. The problem is in two places. The first, and more obvious one, is that I am comparing the value of <code class="language-plaintext highlighter-rouge">b</code> with the value of <code class="language-plaintext highlighter-rouge">0</code> which is an <code class="language-plaintext highlighter-rouge">int</code>. The F# compiler is therefore restricting the input types to be <code class="language-plaintext highlighter-rouge">int</code>. I know this because I can change the value <code class="language-plaintext highlighter-rouge">b</code> is compared to and change the function signature. For example if I change <code class="language-plaintext highlighter-rouge">0</code> to <code class="language-plaintext highlighter-rouge">0M</code>, the type of <code class="language-plaintext highlighter-rouge">a</code> and <code class="language-plaintext highlighter-rouge">b</code> is restricted to <code class="language-plaintext highlighter-rouge">decimal</code>. If I change <code class="language-plaintext highlighter-rouge">0</code> to <code class="language-plaintext highlighter-rouge">0.</code>, making it a float, the type of <code class="language-plaintext highlighter-rouge">a</code> and <code class="language-plaintext highlighter-rouge">b</code> is restricted to <code class="language-plaintext highlighter-rouge">float</code>.</p>
<h2 id="making-the-operator-generic">Making the Operator Generic</h2>
<p>Fortunately, F# has a fix for this, it is called <code class="language-plaintext highlighter-rouge">GenericZero</code>. <code class="language-plaintext highlighter-rouge">GenericZero</code> is a type function which returns the <code class="language-plaintext highlighter-rouge">0</code> equivalent for any numeric type or type with a static member called <code class="language-plaintext highlighter-rouge">Zero</code>. It is contained in the F# Language Primitives, <code class="language-plaintext highlighter-rouge">Microsoft.FSharp.Core.LanguagePrimitives</code>. More information can be found in the <a href="https://msdn.microsoft.com/visualfsharpdocs/conceptual/languageprimitives.genericzero%5b%5et%5d-type-function-%5bfsharp%5d">language reference entry on GenericZero</a>.</p>
<p>The other problem with this function is that it needs to be an <code class="language-plaintext highlighter-rouge">inline</code> function. The <code class="language-plaintext highlighter-rouge">inline</code> keyword in F# tells the compiler to figure out the types for the function at the place of usage instead of restricting the types. Here is a simple example of an <code class="language-plaintext highlighter-rouge">add</code> function without the <code class="language-plaintext highlighter-rouge">inline</code> keyword.</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// non-inlined function</span>
<span class="k">let</span> <span class="n">add</span> <span class="n">a</span> <span class="n">b</span> <span class="p">=</span>
<span class="n">a</span> <span class="o">+</span> <span class="n">b</span>
<span class="c1">// val add : a:int -> b:int -> int</span>
</code></pre></div></div>
<p>You would think that the <code class="language-plaintext highlighter-rouge">add</code> function would be generic but the F# compiler will restrict this to <code class="language-plaintext highlighter-rouge">int</code> because that is the best match it can deduce from the context. Now, if we use the <code class="language-plaintext highlighter-rouge">add</code> function with <code class="language-plaintext highlighter-rouge">float</code> values it will change the function signature but it will still be restricted to only a single type. Here I show using the <code class="language-plaintext highlighter-rouge">add</code> function with <code class="language-plaintext highlighter-rouge">float</code> values before trying to use it with <code class="language-plaintext highlighter-rouge">int</code> values. F# updates the function signature to using <code class="language-plaintext highlighter-rouge">float</code> but now throws an error when we try to use <code class="language-plaintext highlighter-rouge">int</code> values.</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// non-inlined function</span>
<span class="k">let</span> <span class="n">add</span> <span class="n">a</span> <span class="n">b</span> <span class="p">=</span>
<span class="n">a</span> <span class="o">+</span> <span class="n">b</span>
<span class="c1">// val add : a:float -> b:float -> float</span>
<span class="k">let</span> <span class="n">r</span> <span class="p">=</span> <span class="n">add</span> <span class="mi">1</span><span class="p">.</span> <span class="mi">2</span><span class="p">.</span> <span class="c1">// r : float</span>
<span class="k">let</span> <span class="n">r2</span> <span class="p">=</span> <span class="n">add</span> <span class="mi">1</span> <span class="mi">2</span> <span class="c1">// compiler error</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">inline</code> keyword can be added to the beginning of the function to have the compiler deduce the types at the point the function is used.</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="k">inline</span> <span class="n">add</span> <span class="n">a</span> <span class="n">b</span> <span class="p">=</span>
<span class="n">a</span> <span class="o">+</span> <span class="n">b</span>
<span class="c1">// val add : a:'a -> b:'b -> 'c (requires member(+))</span>
<span class="k">let</span> <span class="n">r</span> <span class="p">=</span> <span class="n">add</span> <span class="mi">1</span><span class="p">.</span> <span class="mi">2</span><span class="p">.</span> <span class="c1">// r : float</span>
<span class="k">let</span> <span class="n">r2</span> <span class="p">=</span> <span class="n">add</span> <span class="mi">1</span> <span class="mi">2</span> <span class="c1">// r2 : int</span>
</code></pre></div></div>
<p>We now have all of the ingredients we need to update our new operator <code class="language-plaintext highlighter-rouge">/!</code> so that it will work with generic types.</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">open</span> <span class="nn">Microsoft</span><span class="p">.</span><span class="nn">FSharp</span><span class="p">.</span><span class="nn">Core</span><span class="p">.</span><span class="nc">LanguagePrimitives</span>
<span class="k">let</span> <span class="k">inline</span> <span class="o">(/!)</span> <span class="n">a</span> <span class="n">b</span> <span class="p">=</span>
<span class="k">match</span> <span class="n">b</span> <span class="p"><></span> <span class="nc">GenericZero</span> <span class="k">with</span>
<span class="p">|</span> <span class="bp">true</span> <span class="p">-></span> <span class="n">a</span> <span class="o">/</span> <span class="n">b</span> <span class="p">|></span> <span class="nc">Some</span>
<span class="p">|</span> <span class="bp">false</span> <span class="p">-></span> <span class="nc">None</span>
<span class="c1">// val (/!) : a:'a -> b:'b -> 'c option (requires member (/) and member get_Zero and equality)</span>
</code></pre></div></div>
<p>This is exactly what we were looking for in the beginning. Now when we use our new operator we are forced to deal with a situation where the divisor is possibly <code class="language-plaintext highlighter-rouge">0</code>. This solution for dealing with a possible <code class="language-plaintext highlighter-rouge">0</code> divisor may not be for everyone. Perhaps having to deal with the <code class="language-plaintext highlighter-rouge">None</code> scenario is too cumbersome for you. I find that I like having this additional safety in place because it forces me to write more robust code.</p>Matthew CrewsOne of the things that most attracted me to F# is the ability to accurately model your domain. What first turned me on to this was a talk by Scott Wlaschin on Functional programming design patterns. Scott has a more focused talk on Domain Modeling Made Functional that he did a few years later and a book with the same title. This whole concept was blowing my mind. The idea of modeling your domain such that illegal states are unrepresentable sounds immensely satisfying to me.My Most Expensive error2018-01-06T00:00:00+00:002018-01-06T00:00:00+00:00matthewcrews.com/my-most-expensive-error<p>The title for this may be a little over the top but it is not far from the truth. I am wanting to show how Units of Measure in F# can protect against some of the most insidious types of errors, mismatched units.</p>
<p>One of the most difficult parts of putting together algorithms has been making sure that the Units of Measure for numbers match. For example, you should not be able to add lbs and cm, it doesn’t make sense. In most programming languages though, a number is just a number. You may be working with a strict language which requires you to convert from <code class="language-plaintext highlighter-rouge">int</code> to <code class="language-plaintext highlighter-rouge">float</code> before multiplying, but many will do this implicitly.</p>
<p>When I am writing in R, Python, or C# I don’t have any kind of Units of Measure checking. This has led to a lot of frustrating debugging in the past where I missed some simple multiplication or division in my code. These types of bugs can be really nefarious because you can often get numbers which seem sensible at first but then blow up when outlier data is introduced.</p>
<h2 id="the-initial-error">The Initial Error</h2>
<p>I was tasked with writing a simple fee calculation for our products on Amazon. We need to know the impact of the new fees on our costing before they go into effect. This is such a simple thing. On my first pass I decided to just throw something together in Python. When I did this, I made a very expensive mistake. Can you see it?</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">calculate_item_fba_fee</span><span class="p">(</span><span class="n">cost_config</span><span class="p">,</span> <span class="n">item</span><span class="p">):</span>
<span class="n">weight_tiers</span> <span class="o">=</span> <span class="n">cost_config</span><span class="p">[</span><span class="n">item</span><span class="p">[</span><span class="s">'item_size'</span><span class="p">]][</span><span class="s">'WeightTiers'</span><span class="p">]</span>
<span class="n">weight_tier</span> <span class="o">=</span> <span class="p">[</span><span class="n">tier</span> <span class="k">for</span> <span class="n">tier</span> <span class="ow">in</span> <span class="n">weight_tiers</span> <span class="k">if</span>
<span class="p">(</span><span class="n">tier</span><span class="p">[</span><span class="s">'MinWeight'</span><span class="p">]</span> <span class="o"><</span> <span class="n">item</span><span class="p">[</span><span class="s">'item_weight'</span><span class="p">])</span> <span class="o">&</span> <span class="p">(</span><span class="n">tier</span><span class="p">[</span><span class="s">'MaxWeight'</span><span class="p">]</span> <span class="o">>=</span> <span class="n">item</span><span class="p">[</span><span class="s">'item_weight'</span><span class="p">])][</span><span class="mi">0</span><span class="p">]</span>
<span class="n">fee</span> <span class="o">=</span> <span class="n">weight_tier</span><span class="p">[</span><span class="s">'BaseFee'</span><span class="p">]</span> <span class="o">+</span> <span class="nb">max</span><span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">item</span><span class="p">[</span><span class="s">'item_weight'</span><span class="p">]</span> <span class="o">-</span> <span class="n">weight_tier</span><span class="p">[</span><span class="s">'weight_fee_lb_cutoff'</span><span class="p">])</span>
<span class="k">return</span> <span class="n">fee</span>
</code></pre></div></div>
<p>This function is taking a <code class="language-plaintext highlighter-rouge">Dictionary</code>, <code class="language-plaintext highlighter-rouge">cost_config</code>, which holds some configuration values and a row of a Pandas <code class="language-plaintext highlighter-rouge">DataFrame</code>, called <code class="language-plaintext highlighter-rouge">item</code>. The first line of the function looks up the weight tiers which may apply to the <code class="language-plaintext highlighter-rouge">item</code>. It then searches through the tiers to find the <code class="language-plaintext highlighter-rouge">weight_tier</code> which matches the weight of the <code class="language-plaintext highlighter-rouge">item</code>. It then calculates the fee, which is where the error is.</p>
<p>The <code class="language-plaintext highlighter-rouge">fee</code> value is composed of a <code class="language-plaintext highlighter-rouge">base_fee</code>, in US Dollars (USD), and a USD/lb fee if the weight is above the <code class="language-plaintext highlighter-rouge">weight_fee_lb_cutoff</code> value. In this case the <code class="language-plaintext highlighter-rouge">weight_fee_lb_cutoff</code> value is 2.0 lbs. So, for every lb over 2.0, the item is charged an additional fee per lb.</p>
<p>You may see the error now, I never multiply the overage weight by the <code class="language-plaintext highlighter-rouge">[USD/lb]</code>, (US Dollars / pound), fee rate. If you look at the units of the fee calculation I am adding the <code class="language-plaintext highlighter-rouge">base_fee</code>, which is in <code class="language-plaintext highlighter-rouge">[USD]</code>, to <code class="language-plaintext highlighter-rouge">[lbs]</code>. That does not make any sense. You can’t add different types of units, but most languages will let you do this all day. This was insidious because for most of our items, the fee was right. Only in cases where the item was over 2.0 <code class="language-plaintext highlighter-rouge">[lbs]</code> did we get an incorrect fee.</p>
<p>I’ll be honest, I didn’t actually catch this bug. I put this code in production but I never felt really good about it. I couldn’t explain it but there was disquiet in my soul. I was already starting to rewrite parts of our system in F# so I decided that I would rewrite this little piece while it was fresh in my mind.</p>
<h2 id="f-units-of-measure-save-the-day">F# Units of Measure Save the Day</h2>
<p>For the last several years I have been moving toward more and more strict programming languages. When I heard that F# allows you to put Units of Measure on your numbers, I fell in love. I have longed for such a feature. So many errors can be eliminated when dealing with numbers if you can track and enforce units alignment in numbers.</p>
<p>Because my soul never settled with my initial Python solution, I decided to rewrite the fee calculation. When I started I immediately declared the Units of Measure that I would need:</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Units of Measure Types</span>
<span class="p">[<</span><span class="nc">Measure</span><span class="p">>]</span> <span class="k">type</span> <span class="nc">USD</span> <span class="c">(* US Dollar *)</span>
<span class="p">[<</span><span class="nc">Measure</span><span class="p">>]</span> <span class="k">type</span> <span class="n">lb</span> <span class="c">(* Imperial pound *)</span>
</code></pre></div></div>
<p>I then wrote my fee calculation with the Units of Measure on the numbers to ensure everything matched. I then immediately saw the mistake. You will notice in this new function that I do multiply by the <code class="language-plaintext highlighter-rouge">feeRate</code>.</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// New fee function</span>
<span class="k">let</span> <span class="n">calculateWeightFee</span> <span class="p">(</span><span class="n">baseFee</span> <span class="p">:</span> <span class="n">decimal</span><span class="p"><</span><span class="nc">USD</span><span class="o">>)</span> <span class="p">(</span><span class="n">weightFeeCutoff</span> <span class="p">:</span> <span class="n">decimal</span><span class="p"><</span><span class="n">lb</span><span class="o">>)</span>
<span class="p">(</span><span class="n">feeRate</span> <span class="p">:</span> <span class="n">decimal</span><span class="p"><</span><span class="nc">USD</span><span class="o">/</span><span class="n">lb</span><span class="o">>)</span> <span class="p">(</span><span class="n">weight</span> <span class="p">:</span> <span class="n">decimal</span><span class="p"><</span><span class="n">lb</span><span class="o">>)</span> <span class="p">=</span>
<span class="n">baseFee</span> <span class="o">+</span> <span class="p">(</span><span class="n">max</span> <span class="mi">0</span><span class="nc">M</span><span class="p"><</span><span class="n">lb</span><span class="p">></span> <span class="p">(</span><span class="n">weight</span> <span class="p">-</span> <span class="n">weightFeeCutoff</span><span class="o">))</span> <span class="p">*</span> <span class="n">feeRate</span>
</code></pre></div></div>
<p>I felt pretty stupid after such an obvious mistake. Fortunately, the previous version of the code was only in production for a couple of days. Had this gone on for longer, we could have missed huge volumes of opportunity because products would have look too expensive due to the new fee.</p>
<p>Now granted, better unit testing would have caught this. Also, this post is not meant to disparage Python, or any other language, in any way. Rather, I am highlighting that F# is eliminating an entire class of errors for me and making me more productive. I much prefer the compiler barking at me about my units not matching than me spending hours or days hunting for where I missed a multiplication or a division. It feels great knowing that my units line up and that if I miss a small detail like this, the compiler will gently guide me back to sanity. Check out this wonderful <a href="https://fsharpforfunandprofit.com/posts/units-of-measure/">post by Scott Wlaschin</a> for a more detailed discussion on what can be done with F# and Units of Measure.</p>Matthew CrewsThe title for this may be a little over the top but it is not far from the truth. I am wanting to show how Units of Measure in F# can protect against some of the most insidious types of errors, mismatched units.Using F# to Parse HTML2018-01-05T00:00:00+00:002018-01-05T00:00:00+00:00matthewcrews.com/using-fsharp-to-parse-html<p>At work I have been tasked with extracting the product description information for several of our products to be used as an import for an external system. Normally I would just write a query for the database to get all of this information but in this case I do not have access to the database directly. Even if I did have it, I am not familiar with the schema so I would rather not have to spend the effort digging into if I do not have to. I have been putting off this project since it was not high priority but recently I came across an excellent talk by <a href="https://www.youtube.com/watch?v=K_AlkvZsUus&t=1219s">Evelina Gabasova at NDC Oslo</a> where she showed the use of TypeProviders to connect to IMDB to extract data on cast members of Star Wars. As I watched this I had a eureka moment, “Why not just use F# to pull the data directly from the website instead of dealing with the SQL Schema?”</p>
<h1 id="initial-attempt-with-htmlprovider">Initial Attempt with HtmlProvider</h1>
<p>Now, this may seem a little silly but for my case it has some advantages. I do not have to bother with getting permissions for the database running the company e-commerce website and it allows me to use some F#. I quickly fire up a new F# project in VS Code and stub out the following:</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">#</span><span class="nc">I</span> <span class="s2">"./packages"</span>
<span class="p">#</span><span class="n">r</span> <span class="s2">"FSharp.Data/lib/net40/FSharp.Data.dll"</span>
<span class="k">open</span> <span class="nn">FSharp</span><span class="p">.</span><span class="nc">Data</span>
<span class="k">type</span> <span class="nc">Product</span> <span class="p">=</span> <span class="nc">HtmlProvider</span><span class="p"><</span><span class="s2">"https://www.b-glowing.com/skincare/cleansers/paulas-choice-calm-redness-relief-cleanser-for-oily-skin/"</span><span class="p">></span>
<span class="k">let</span> <span class="n">test</span> <span class="p">=</span> <span class="nn">Product</span><span class="p">.</span><span class="nc">GetSample</span><span class="bp">()</span>
</code></pre></div></div>
<p>This is when I run into a problem. In the talk that <a href="http://evelinag.com/">Evelina Gabasova</a> gave the data on IMDB was in a nice table. This meant that the TypeProvider could detect it automatically and provide it as a nice property of the <code class="language-plaintext highlighter-rouge">test</code> object in the above example. My problem is that the data I need is in the Description area of the page, specifically the <code class="language-plaintext highlighter-rouge"><span></code> with the attribute <code class="language-plaintext highlighter-rouge">itemprop="description"</code>. I am trying to turn this information:</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><span</span> <span class="na">itemprop=</span><span class="s">"description"</span><span class="nt">></span>
<span class="nt"><p></span>
<span class="nt"><strong></span>WHAT IT IS<span class="ni">&nbsp;</span><span class="nt"></strong></span>
<span class="nt"><br></span>
A lightweight silky gel cleanser for Normal to Oily skin types that gently removes makeup and soothes red, sensitive skin.
<span class="nt"></p></span>
<span class="nt"><p></span>
<span class="nt"><strong></span>BENEFITS FOR YOU<span class="nt"></strong></span>
<span class="nt"><br></span>• Safe for even the most sensitive skin.
<span class="nt"><br></span>• Removes excess oil and makeup.
<span class="nt"><br></span>• Soothes and refreshes senstive, irritated<span class="ni">&nbsp;</span>skin.
<span class="nt"></p></span>
<span class="nt"><p></span>
<span class="nt"><strong></span>YOU’LL EXPERIENCE<span class="nt"></strong></span>
<span class="nt"><br></span>This lightweight gel texture lathers beautifully to remove excess oils, impurities and makeup. Skin is left calm, clean and soft.
<span class="nt"></p></span>
<span class="nt"><p></span>
<span class="nt"><strong></span>WHY IT’S GLOWING<span class="nt"></strong></span>
<span class="nt"><br></span>
The calming cleanser works wonders for those of us who<span class="ni">&nbsp;</span>experience sensitivity and redness without drying or stripping skin. The formula increases our skins natural barrier so overtime skin is less sensitive and red on its own.<span class="ni">&nbsp;</span>
<span class="nt"></p></span>
<span class="nt"></span></span>
</code></pre></div></div>
<p>into something like this:</p>
<table>
<thead>
<tr>
<th>Tag</th>
<th>Text</th>
</tr>
</thead>
<tbody>
<tr>
<td>WHAT IT IS</td>
<td>A lightweight silky gel cleanser for Normal to Oily skin types that gently removes makeup and soothes red, sensitive skin.</td>
</tr>
<tr>
<td>BENEFITS FOR YOU</td>
<td>Safe for even the most sensitive skin. Removes excess oil and makeup. Soothes and refreshes sensitive, irritated skin.</td>
</tr>
</tbody>
</table>
<p>This means that I need a different approach. Thankfully, F# delivered.</p>
<h1 id="using-html-parser">Using HTML Parser</h1>
<p>If the <code class="language-plaintext highlighter-rouge">HtmlProvider</code> does not give you what you need for HTML parsing then <code class="language-plaintext highlighter-rouge">FSharp.Data</code> also has a handy <a href="http://fsharp.github.io/FSharp.Data/library/HtmlParser.html">HTML Parser</a> which includes some excellent documentation and examples. I put together a new script to extract the data from the website.</p>
<div class="language-fsharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">#</span><span class="nc">I</span> <span class="s2">"./packages"</span>
<span class="p">#</span><span class="n">r</span> <span class="s2">"FSharp.Data/lib/net40/FSharp.Data.dll"</span>
<span class="k">open</span> <span class="nn">FSharp</span><span class="p">.</span><span class="nc">Data</span>
<span class="k">let</span> <span class="n">productHtml</span> <span class="p">=</span> <span class="nn">HtmlDocument</span><span class="p">.</span><span class="nc">Load</span><span class="p">(</span><span class="s2">"https://www.b-glowing.com/skincare/cleansers/paulas-choice-calm-redness-relief-cleanser-for-oily-skin/"</span><span class="p">)</span>
<span class="k">let</span> <span class="n">getDescription</span> <span class="p">(</span><span class="n">html</span><span class="p">:</span><span class="nc">HtmlDocument</span><span class="p">)</span> <span class="p">=</span>
<span class="n">html</span><span class="p">.</span><span class="nc">Descendants</span> <span class="p">[</span><span class="s2">"span"</span><span class="p">]</span>
<span class="p">|></span> <span class="nn">Seq</span><span class="p">.</span><span class="n">filter</span> <span class="p">(</span><span class="k">fun</span> <span class="n">x</span> <span class="p">-></span>
<span class="k">match</span> <span class="n">x</span><span class="p">.</span><span class="nc">TryGetAttribute</span><span class="p">(</span><span class="s2">"itemprop"</span><span class="p">)</span> <span class="k">with</span>
<span class="p">|</span> <span class="nc">None</span> <span class="p">-></span> <span class="bp">false</span>
<span class="p">|</span> <span class="nc">Some</span> <span class="n">att</span> <span class="p">-></span>
<span class="k">match</span> <span class="n">att</span><span class="p">.</span><span class="nc">Value</span><span class="bp">()</span> <span class="k">with</span>
<span class="p">|</span> <span class="s2">"description"</span> <span class="p">-></span> <span class="bp">true</span>
<span class="p">|</span> <span class="p">_</span> <span class="p">-></span> <span class="bp">false</span>
<span class="p">)</span>
<span class="p">|></span> <span class="nn">Seq</span><span class="p">.</span><span class="n">exactlyOne</span>
<span class="p">|></span> <span class="p">(</span><span class="k">fun</span> <span class="n">x</span> <span class="p">-></span>
<span class="n">x</span><span class="p">.</span><span class="nc">Descendants</span> <span class="p">[</span><span class="s2">"p"</span><span class="p">]</span>
<span class="p">|></span> <span class="nn">Seq</span><span class="p">.</span><span class="n">map</span> <span class="p">(</span><span class="k">fun</span> <span class="n">t</span> <span class="p">-></span>
<span class="k">let</span> <span class="n">tag</span> <span class="p">=</span>
<span class="n">t</span><span class="p">.</span><span class="nc">Descendants</span> <span class="p">[</span><span class="s2">"strong"</span><span class="p">]</span>
<span class="p">|></span> <span class="nn">Seq</span><span class="p">.</span><span class="n">exactlyOne</span>
<span class="p">|></span> <span class="p">(</span><span class="k">fun</span> <span class="n">b</span> <span class="p">-></span> <span class="n">b</span><span class="p">.</span><span class="nc">InnerText</span><span class="bp">()</span><span class="p">)</span>
<span class="k">let</span> <span class="n">text</span> <span class="p">=</span> <span class="n">t</span><span class="p">.</span><span class="nc">InnerText</span><span class="bp">()</span><span class="o">.[(</span><span class="n">tag</span><span class="p">.</span><span class="nc">Length</span><span class="o">)..(</span><span class="n">t</span><span class="p">.</span><span class="nc">InnerText</span><span class="bp">()</span><span class="p">.</span><span class="nc">Length</span> <span class="p">-</span> <span class="mi">1</span><span class="o">)]</span>
<span class="n">tag</span><span class="p">,</span> <span class="n">text</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="k">let</span> <span class="n">productDescription</span> <span class="p">=</span> <span class="n">getDescription</span> <span class="n">productHtml</span>
</code></pre></div></div>
<p>Lines 1 through 5 are just getting the HTML for the product listing. The <code class="language-plaintext highlighter-rouge">getDescription</code> function is what actually breaks down the HTML to return a tuple with the information that I am interested in. What I like most about this is that I did not have to use <code class="language-plaintext highlighter-rouge">XPath</code> or some other <code class="language-plaintext highlighter-rouge">XML</code> querying tool. While <code class="language-plaintext highlighter-rouge">XPath</code> may be powerful, I find I I have difficulty achieving what I really want. I find the F# approach shown here much more straightforward.</p>
<p>In line 8 the function is extracting every node in the HTML which is a <code class="language-plaintext highlighter-rouge">span</code>. This will obviously return spans that we are not interested in which is why we need to filter the result using <code class="language-plaintext highlighter-rouge">Seq.filter</code>. Since I know that the span I am interested in has the attribute <code class="language-plaintext highlighter-rouge">itemprop="description"</code>, I use a function to return <code class="language-plaintext highlighter-rouge">false</code> when that attribute is not present and <code class="language-plaintext highlighter-rouge">true</code> when it is present. Line 10 highlights one of my favorite features of F# which is the returning of an <code class="language-plaintext highlighter-rouge">Option</code> type. The function <code class="language-plaintext highlighter-rouge">TryGetAttribute</code> will either successfully return the attribute which is a type of <code class="language-plaintext highlighter-rouge">Some 'T</code> or it returns <code class="language-plaintext highlighter-rouge">None</code>. In the case of <code class="language-plaintext highlighter-rouge">None</code> I simply have the function return <code class="language-plaintext highlighter-rouge">false</code>. If the attribute does exist I then test if it is equal to “description” on line 14. If it does match, the function returns <code class="language-plaintext highlighter-rouge">true</code>. In all other cases the function returns <code class="language-plaintext highlighter-rouge">false</code>.</p>
<p>Since I know how the HTML is rendered on these pages, I know that there will only ever be one of these <code class="language-plaintext highlighter-rouge"><span></code> elements in the HTML so I use the function <code class="language-plaintext highlighter-rouge">Seq.exactlyOne</code> to select a single element from the sequence. Line 18 to 27 is where I actually pull out the information that I want. I have an odd problem in that the text in the <code class="language-plaintext highlighter-rouge"><strong></code> element is what I want the tag name to be for the output table and the rest of the text in the parent <code class="language-plaintext highlighter-rouge"><p></code> element is to be the text data. To do this I first extract the text in the <code class="language-plaintext highlighter-rouge"><strong></code> element on lines 21 through 24. I then extract all of the text from the parent <code class="language-plaintext highlighter-rouge"><p></code> element, which includes the <code class="language-plaintext highlighter-rouge"><strong></code> text, and then select a substring which excludes the <code class="language-plaintext highlighter-rouge"><strong></code> text on line 22. I then return a tuple of the tag name and the associated text.</p>
<p>While I am sure this is not the most elegant way to go about this, it was incredibly simple compared to some previous efforts I have had trying to get <code class="language-plaintext highlighter-rouge">XPath</code> to work on other projects. I find the F# syntax and approach much more straightforward and easier to understand. Is there a better way for me to have done this? Could the code be more idiomatic? All comments and suggestions are appreciated.</p>Matthew CrewsAt work I have been tasked with extracting the product description information for several of our products to be used as an import for an external system. Normally I would just write a query for the database to get all of this information but in this case I do not have access to the database directly. Even if I did have it, I am not familiar with the schema so I would rather not have to spend the effort digging into if I do not have to. I have been putting off this project since it was not high priority but recently I came across an excellent talk by Evelina Gabasova at NDC Oslo where she showed the use of TypeProviders to connect to IMDB to extract data on cast members of Star Wars. As I watched this I had a eureka moment, “Why not just use F# to pull the data directly from the website instead of dealing with the SQL Schema?”