{"id":13491,"date":"2018-07-19T21:40:50","date_gmt":"2018-07-19T19:40:50","guid":{"rendered":"https:\/\/touk.pl\/blog\/?p=13491"},"modified":"2022-07-27T10:07:56","modified_gmt":"2022-07-27T08:07:56","slug":"what-really-grinds-my-gears-apache-nifi","status":"publish","type":"post","link":"https:\/\/touk.pl\/blog\/2018\/07\/19\/what-really-grinds-my-gears-apache-nifi\/","title":{"rendered":"What Really Grinds My Gears: Apache NiFi"},"content":{"rendered":"

Introduction<\/h1>\n

Complaining and doing nothing to solve a problem. That\u2019s what everybody does on the Internet. And that\u2019s precisely what I am going to do. Why? Apache NiFi has recently proved to be powerful and effective tool for processing gigabytes of data in our telco integration project. Yet, sometimes I feel like I am just clicking too much to achieve small things. It hurts just like hitting your head against a door frame every time you enter your bedroom. Let\u2019s take a closer look at it.<\/p>\n

Navigation<\/h1>\n

Dividing complex data flows into the process groups is a great way of achieving modularization and reusability, however, navigation between them is rather painful. Imagine set of process groups with nested process groups with nested process groups\u2026 Every time I want to go from A->B->C to A->E->F I have to use a well-hidden breadcrumb for A and double click to get into E and F.<\/p>\n

What is the problem? At the beginning I couldn\u2019t even find the breadcrumb! I didn\u2019t expect it to be at the very bottom of the screen as it is an untypical location for it. What I did expect however was a big shiny arrow go back<\/em>. Moreover, having navigated to A->E->F I wanted to go back to A->B->C. As there was no \u201cgo forward \/ redo\u201d button I had to search for components E and F among others which made me sacrifice my precious time.<\/p>\n

\"\"<\/p>\n

Why do I need to navigate so much you ask? Well, I am lazy and have rather short memory at the same time. What developing flows means for me is checking what I did yesterday and adjusting it to current problem. Maybe if I had a possibility to open yesterday\u2019s flow in another tab I wouldn\u2019t change those views at all? My internet browser can obviously do that but it has some limits like copy-paste. Similar idea is a screen division. I know it is all difficult to achieve but I would find it a major improvement.<\/p>\n

List queue<\/h1>\n

When I integrate with external data source all I have is documentation and hope for its correctness. In fact this rule applies to every new NiFi processor I try. When something fails I investigate the reason of failure. When everything seems right I ensure that it truly is. In both scenarios I check the first element in the queue.<\/p>\n

Again, my expectations are simple. I double click the queue to see what it contains because it is the most interesting feature of a queue: it holds elements. According to my experience 95% of times when I do anything related to a queue I check its first element. Sadly, when double clicking I see a configuration window which comes useful from time to time. Still, in my opinion it is not as important as the content itself.<\/p>\n

\"\"<\/p>\n

So how do you see elements of a queue anyway? Just like starting engine with cables: not easy but one can get used to it. Open context menu, select \u201cList queue\u201d and click the \u201cinfo\u201d button on the very left. Clicking the button is the weirdest step for me. It does not look like a button at all and I discovered its \u2018clickability\u2019 only because my friend told me about it.<\/p>\n

\"\"<\/p>\n

I am sure there must be faster way to take a quick look at the queue content. I know that the current state is consistent with the processors front-end design but I do not feel a there\u2019s such a strong link between queues and processors. What you typically do with a processor is configuration but what you typically do with a queue is checking the content details.<\/p>\n

Error display<\/h1>\n

NiFi displays error in flows with big red label located in the top right corner of the processor \/ process group. I have nothing against it. Yet, what does a true programmer do when an error occurs? He or she copies an error message and enters stack overflow :)<\/p>\n

\"\"<\/p>\n

And here comes the problem: there is no way to copy an error from NiFi GUI. My workaround is to open logs. Usually, it takes ages. Sometimes it is even impossible. As a software vendor company we struggle with the limited access to hosts and services on everyday basis. Just let me copy my own errors and i will do my job. Trust me I\u2019m an engineer.<\/p>\n

Surprisingly in the review my colleague noted that I can actually copy errors from Bulletin board<\/em> located in the main settings menu. However, I find it a bit dissapointing as the messages are incomplete and the menu itself is hidden deeply.<\/p>\n

Templates<\/h1>\n

Exporting templates from one NiFi instance to another is another unintuitive feature. What I expect is to right click on a process group and choose export<\/em> option. What I get is a sequence of actions:<\/p>\n

    \n
  1. Create template<\/li>\n
  2. Open \u201ctemplates\u201d from the main settings menu in a top-right corner<\/li>\n
  3. Choose a very small \u201cdownload\u201d button that doesn\u2019t look like a download button<\/li>\n<\/ol>\n

    I do not understand why I can\u2019t export a process group which is not a template. Of course it has to be imported as a template but exporting seems to be just an on demand XML generation. What are the benefits of such dependency?<\/p>\n

    \"\"<\/p>\n

    Summary<\/h1>\n

    I hope that this post will not be taken as offensive by any means. As a programmer I know how difficult it is to design beautiful, functional and consistent frontend even for very simple application. Therefore, I would like to think about this case study as a guidance for everyone who designs UX. Ask your users about their habits, analyse how the application is used. Automating your client\u2019s job will bring you tons of respect.<\/p>\n

    And what is your experience with NiFi GUI? I think it looks pretty good but still may be more functional and convenient. I hope it will be soon.<\/p>\n

    Finally I would like to thanks Micha\u0142 Hofman and Bart\u0142omiej Tartanus for the review.<\/p>\n","protected":false},"excerpt":{"rendered":"Introduction Complaining and doing nothing to solve a problem. That\u2019s what everybody does on the Internet. And that\u2019s…\n","protected":false},"author":73,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11],"tags":[252,440],"class_list":{"0":"post-13491","1":"post","2":"type-post","3":"status-publish","4":"format-standard","6":"category-development-design","7":"tag-big-data","8":"tag-ux"},"_links":{"self":[{"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/posts\/13491","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/users\/73"}],"replies":[{"embeddable":true,"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/comments?post=13491"}],"version-history":[{"count":27,"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/posts\/13491\/revisions"}],"predecessor-version":[{"id":13532,"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/posts\/13491\/revisions\/13532"}],"wp:attachment":[{"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/media?parent=13491"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/categories?post=13491"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/tags?post=13491"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}