{"id":13534,"date":"2018-08-07T12:00:39","date_gmt":"2018-08-07T10:00:39","guid":{"rendered":"https:\/\/medium.com\/p\/19ac7e13fd06"},"modified":"2023-03-16T16:49:26","modified_gmt":"2023-03-16T15:49:26","slug":"analyzing-2018-world-cup-match-data-with-clojure","status":"publish","type":"post","link":"https:\/\/touk.pl\/blog\/2018\/08\/07\/analyzing-2018-world-cup-match-data-with-clojure\/","title":{"rendered":"Analyzing 2018 World Cup match data with Clojure"},"content":{"rendered":"

So, Russia 2018 World Cup is over. There were lots of good moments, and some worse (especially for us, here in Poland\u2026.). But is there something we can learn from this event as programmers? Recently, I had a few free evenings and played a bit with a World Cup API<\/a>. In this post I want to show you what is Clojure way of dealing with REST APIs and when it beats other languages in this field.
\n
\nTLDR;<\/strong> The code is on GitHub<\/a>. Setup is based on Atom and proto-repl<\/a> package which makes the interactive development experience really pleasant. I highly recommend you follow its installation guide while dealing with my solution.<\/p>\n

As a problem to solve, I wanted to find something not trivial, to show some of the Clojure super-powers. Finally, I came up with this:<\/p>\n

The problem: find all matches the winning team made a come-back\u200a\u2014\u200athey lost a goal first, but managed to win.<\/p><\/blockquote>\n

For example Sweden\u200a\u2014\u200aGermany in a group stage:<\/p>\n

\"\"<\/figure>\n

Ok, so a good starting point would be getting all matches data for further processing. I\u2019m using clj-http<\/code> as an http client (with cheshire<\/code> for json parsing support):<\/p>\n

(ns worldcup.matches\r\n  (:require\r\n    [clj-http.client :as http]))\r\n\r\n(def api-root \"https:\/\/worldcup.sfg.io\")\r\n\r\n(defn get-all-matches []\r\n  (->\r\n    (http\/get (str api-root \"\/matches\") {:as :json})\r\n    :body))\r\n<\/pre>\n
The Clojure philosophy relies heavily on dealing with built-in data structures. In statically typed languages you\u2019d probably start with modelling response JSON as some POJO classes; here you just get a map and can start to investigate it right from the beginning. Huge difference.<\/p>\n
One thing that may not be clear in the above code is this -><\/code> macro<\/a>. It is really helpful for nested collection operations. Instead of nesting (xxx (yyy (zzz \u2026<\/code> calls, you can use both -><\/code> and ->><\/code> macros to make code similar to e.g. Java equivalent:<\/p>\n
(->> lst\r\n  (map op1)\r\n  (map op2)\r\n  (filter p1)\r\n  first)\r\n\r\nlst.stream()\r\n    .map(op1)\r\n    .map(op2)\r\n    .filter(p1)\r\n    .findFirst()\r\n<\/pre>\n
Ok, let\u2019s start to investigate the API response. Using proto-repl<\/code> it is trivial to examine the value of any data structure straight from your editor. Let\u2019s see the match data\u200a\u2014\u200aby using Proto REPL: Execute Block<\/code> action\u200a\u2014\u200actrl+alt+,s<\/code> shortcut by default:<\/p>\n
\"\"<\/p>\n
It\u2019s a common practice to wrap interactive invocations into comment<\/code> macro which makes them ignored when evaluating whole file at once.<\/p>\n
Ok, so we have some basic attributes of the match, the number of goals for each team and some team events. Let\u2019s investigate these:<\/p>\n
\"\"<\/figure>\n
Looks promising\u200a\u2014\u200awe have all the times of the goals scored by each team. Having this, our algorithm may look like this:<\/p>\n
    \n
  1. Get times of goals for each team<\/li>\n
  2. Sort goals by times<\/li>\n
  3. Get the team with the first goal<\/li>\n
  4. Check if the winner is the other team<\/li>\n<\/ol>\n
    Let\u2019s start with filtering the goals from team\u2019s events. First, we need to find how to filter goals from match events:<\/p>\n
    \"\"<\/p>\n
    Ok, so we have goal<\/code>, goal-penalty<\/code> and goal-own<\/code>:<\/p>\n
    defn goal? [e]\r\n  (let [type (:type_of_event e)]\r\n    (or (= \"goal\" type)\r\n        (= \"goal-penalty\" type)\r\n        (= \"goal-own\" type))))\r\n<\/pre>\n
    \"\"<\/figure>\n
    Looks good.<\/p>\n
    We\u2019ll also need the time when the goal was scored and this is in a really strange format, e.g 90'+4'<\/code>. We need to split on +<\/code> character, remove '<\/code> and trim<\/code> and then add additional time to get exact minute of the goal:<\/p>\n
    (require '[clojure.string :as string])\r\n\r\n(defn goal-time [t]\r\n  (->> \r\n    (string\/split t #\"\\+\")\r\n    (map string\/trim)\r\n    (map #(string\/replace % #\"'\" \"\"))\r\n    (map #(Integer\/parseInt %))\r\n    (reduce +)))\r\n<\/pre>\n
    \"\"<\/figure>\n
    EDIT:<\/strong><\/p>\n
    Bartek Tartanus<\/a> (thanks!) found a bug in just adding additional time to half time: suppose we had late goal in the first half and early goal in the second (45'+5'<\/code> and 46<\/code>)\u200a\u2014\u200amy algorithm will pick second half goal as earlier. We need to somehow pass half (or part for extra-time) data to sort goal times. Fortunately, Clojure sorts collections of pairs well:<\/p>\n
    \"\"<\/p>\n
    We need to calculate both half (part) and exact time of scoring a goal:<\/p>\n
    (defn part [goal]\r\n  (let [time (first goal)]\r\n    (cond\r\n      (<= time 45) 1\r\n      (<= time 90) 2\r\n      (<= time 105) 3   \r\n      (<= time 120) 4))) ;; parts 3 and 4 are for extra-time\r\n\r\n(defn parts [time-str]\r\n  (->>\r\n    (string\/split time-str #\"\\+\")\r\n    (map string\/trim)\r\n    (map #(string\/replace % #\"'\" \"\"))\r\n    (map #(Integer\/parseInt %))))\r\n\r\n(defn part-and-time [time-str]\r\n  (let [parts (parts time-str)]\r\n    [(part parts) (reduce + parts)]))\r\n<\/pre>\n
    Now we can test improved sorting:<\/p>\n
    \"\"<\/p>\n
    Seems to work! So let\u2019s try to find a team that scored a first goal. My idea was to mix both :home<\/code> and :away<\/code> goal times in one collection:<\/p>\n
    ([:home [1 38]] [:home [1 39]] [:away [1 28]])<\/pre>\n
    and then sort by times:<\/p>\n
    ([:away [1 28]] [:home [1 38]] [:home [1 39]])<\/pre>\n
    to know the side that scored first goal (:away<\/code> in this case).<\/p>\n
    (defn first-scored-side [match]\r\n  (let [goal-times\r\n          (concat\r\n            (team-goal-times (:home_team_events match) :home)\r\n            (team-goal-times (:away_team_events match) :away))]\r\n   (->> goal-times (sort-by second) first first)))\r\n<\/pre>\n
    I use first<\/code> to get the first pair from sorted collection and then first<\/code> again to get the first element from the pair:<\/p>\n
    \"\"<\/figure>\n
    To create a list with goal times and side I used this function:<\/p>\n
    (defn team-goal-times [events side]\r\n  (->> events\r\n    (filter goal?)\r\n    (map :time)\r\n    (map part-and-time)\r\n    (map vector (repeat side))))\r\n<\/pre>\n
    What\u2019s happening in this last line? So, repeat<\/code> returns an infinite lazy sequence with an element repeated and vector<\/code> just creates an indexed collection<\/a> from provided elements.<\/p>\n
    \"\"<\/figure>\n
    And map can also take two collections, applying provided function to pairs: <c1_first, c2_first>, <c1_second, c2_second><\/code>…<\/p>\n
    \"\"<\/figure>\n
    Which leads us easily to this:<\/p>\n
    \"\"<\/figure>\n
    Ok, now we are ready write this come-back?<\/code> function now:<\/p>\n
    (defn winner-side [match]\r\n  (let [winner (:winner match)]\r\n    (cond\r\n      (= winner (:home_team_country match)) :home\r\n      (= winner (:away_team_country match)) :away)))\r\n\r\n(defn come-back? [match]\r\n  (let [first-scored-side (first-scored-side match)\r\n        winner-side (winner-side match)]\r\n    (and (not (nil? winner-side))\r\n         (not= winner-side first-scored-side))))\r\n<\/pre>\n
    Let\u2019s see if it works, just filter the team\u2019s data from the match to remove noise:<\/p>\n
    (defn teams [match]\r\n  (select-keys match [:home_team_country :away_team_country :winner]))\r\n<\/pre>\n
    \"\"<\/figure>\n
    Ok, we got some results! But, hmm\u2026 Morocco\u200a\u2014\u200aIran?<\/p>\n
    \"\"<\/figure>\n
    0:1 with last-minute own goal? This definitely doesn\u2019t look like come-back\u2026. Let\u2019s look at this match events:<\/p>\n
    \"\"<\/figure>\n
    Ok, now I can see the problem. Own goals are contained in wrong side\u2019s events! This goal should be in Iran\u2019s events, otherwise our solution won\u2019t work\u2026<\/p>\n
    We can make a special case for own goals or just move own goals to correct side events. I decided to go with the second solution, knowing that Clojure is really good at transforming nested data structures e.g. with update and update-in functions:<\/p>\n
    \"\"<\/figure>\n
    So, update<\/code> takes a map, a key and a function and applies to value under that key, leaving rest of the map untouched. update-in<\/code> works the same, but you can pass a whole path to some nested element in your data structure.<\/p>\n
    Let\u2019s use update to move all the own-goals from :home_team_events<\/code> to :away_team_events<\/code> and vice-versa:<\/p>\n
    (defn own-goal? [event]\r\n  (= \"goal-own\" (:type_of_event event)))\r\n\r\n(defn own-goals [events]\r\n  (->> events (filter own-goal?)))\r\n\r\n(defn remove-own-goals [events]\r\n  (remove own-goal? events))\r\n\r\n(defn fix-match [match-to-fix]\r\n  (let [own-home-goals (own-goals (:home_team_events match-to-fix))\r\n        own-away-goals (own-goals (:away_team_events match-to-fix))]\r\n   (-> match-to-fix\r\n     (update :home_team_events remove-own-goals)\r\n     (update :away_team_events remove-own-goals)\r\n     (update :home_team_events concat own-away-goals)\r\n     (update :away_team_events concat own-home-goals))))\r\n<\/pre>\n
    And finally let\u2019s fix come-back?<\/code> function:<\/p>\n
    (defn come-back? [match-to-fix]\r\n  (let [match (fix-match match-to-fix)\r\n        first-scored-side (first-scored-side match)\r\n        winner-side (winner-side match)]\r\n    (and (not (nil? winner-side))\r\n         (not= winner-side first-scored-side))))\r\n<\/pre>\n
    Check the results again:<\/p>\n
    \"\"<\/figure>\n
    Yes, we have it! We found 9 such games, with overall leader Croatia coming back 3 times (2 by penalties).<\/p>\n","protected":false},"excerpt":{"rendered":"So, Russia 2018 World Cup is over. There were lots of good moments, and some worse (especially for…\n","protected":false},"author":5,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11],"tags":[491,649,648,647],"class_list":{"0":"post-13534","1":"post","2":"type-post","3":"status-publish","4":"format-standard","6":"category-development-design","7":"tag-clojure","8":"tag-data-analysis","9":"tag-rest-api","10":"tag-world-cup"},"_links":{"self":[{"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/posts\/13534","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/comments?post=13534"}],"version-history":[{"count":10,"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/posts\/13534\/revisions"}],"predecessor-version":[{"id":15308,"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/posts\/13534\/revisions\/15308"}],"wp:attachment":[{"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/media?parent=13534"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/categories?post=13534"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/tags?post=13534"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}