I don’t know where to start with this one. I think I either need to write a book about my experience with Shiny, or nothing at all. I’ll try to do something in between. Wish me luck.
I think of this project as the one primarily responsible for helping me make the transition from junior to senior developer, in terms of mindset. At the beginning of this adventure I mostly trusted what other, more senior developers told me. If I was told X was the best way to do a thing, I did it that way. By the end of this adventure I think I realized that I now knew as much or more than those senior developers. I started pushing back when I was told to implement something a particular way and knew there was a better option.
Background
In this particular case I was told that we needed to develop analytical tools for public health and microbiome data. These tools would perform various standard analyses for their respective fields and provide visualizations of the results. Perhaps most importantly, they would be built using R Shiny. I would be responsible for the public health tools and the first one was needed for a large meeting only a few weeks away. That was about all the information I had to start.
Development
I settled on a Distributions App to start. It seemed easy enough to implement and a natural place to start if we weren’t sure where to start. It’s always good to look at the distribution of your data. I had the first iteration ready two weeks later which allowed the user to choose a variable for the X-axis and one to facet by.
From there I found myself spending probably half of my working hours either planning a new tool, developing one or adding features to an existing one. That continued until I had three mostly feature complete tools of varying complexity a couple months later.
It might sound like the story should end there, but instead I spent probably the next two years trying to get those tools to a place where they where maintainable, scalable, performant, had a clear UX and were aesthetically unoffensive. I’m not sure I ever fully succeeded. I was the only one on the project committed to the task, and one of maybe two familiar with Shiny. I had little to no help despite being a part of a team of very talented people.
Result
In the end I had three Shiny applications. The first was still Distributions, but it now had two facets, a tab of related summary statistics, the ability to isolate specific facet panels to get a larger view if needed, an easy means to filter to specific time points and detailed help documentation. There was also a new 2×2 Contingency Table application and one we called Data Summaries. Data Summaries was easily the most complicated of the three. It would show different visualizations and provide different configurable inputs depending on whether the study was longitudinal or not.
Not only were there more apps and new features in the original Distributions app, there was a dashboard sort of layout now and a bit of semi-custom styling. There was even a custom drop-down tree component I had made in order to more clearly and appropriately display our ‘variable trees’. More importantly though, the server side of these applications had been completely reworked. They were modular, managing connections to the Oracle database directly, relying on summary metadata about the available variables in each study to construct the UI, and had been exhaustively performance tested.
It still wasn’t enough. They weren’t performant enough for our data, which could have thousands of columns and multiple millions of rows. They weren’t pretty enough as compared to the rest of the site. Technical debt and bugs were piling up through the iterative changes, and I didn’t have the resources as a single developer to keep up. They weren’t async and fixing that would be costly. Perhaps worst of all though, they were technologically siloed from the rest of our software stack and it was impacting UX.
Take-Away
We needed a fundamentally different solution, one that could actually leverage the vast expertise we had on the project. Our human and technical resources weren’t being put to their best use. As a result, the tools we developed were much less than they could be. It was an important, though costly, lesson to learn. In the end we decided to retire our shiny apps and develop the EDA instead.
Shiny is a great tool for the right people with the right use case, but we weren’t those people. It took me a while to get to that conclusion though, both professionally and personally. There were times that I think I tried to hold onto Shiny just because I’d invested so much time there. I was also proud of some of the code I’d written. That still didn’t make it the right tool.
Now I prefer moving to the right tool as early as possible. I don’t believe in investing resources in something you know you’ll abandon without very strong reasons. I’d rather incrementally add features the right way than get most of the specified features half functional quickly.
