This is the first of a three-part blog post on the Jupyter Notebook ecosystem. Here, I’ll talk about the data science landscape, and the forces that pushesour tools to evolve.
Ah, Jupyter Notebooks. Love it or hate it, they’re here to stay. The lasttime I wrote about them was two years ago (it’s about running a notebookfrom a remoteserver),and the ecosystem has grown ever since—even how I interact with notebookstotally changed!
- Miranda Lambert and her husband Brendan McLoughlin were spotted celebrating their two-year wedding anniversary this month. Time does fly really fast and it does so even faster when you are in love. The couple has been in a relationship, which at that time, raised a huge amount of excitement amongst the fans.
- Shop for miranda lambert at Best Buy. Find low everyday prices and buy online for delivery or in-store pick-up.
In this multi-part blogpost, I’d review my oft-used tool in the data sciencetoolbox, Jupyter Notebooks, and how I use them in modern times. I dividedthis post into three:
- Part I: The data science landscape(This page). I’d like to look into how the practice of data science has changed for the past few years. Then, I’ll zoom into the three main forces that changed the way I use Notebooks today.
- Part II: How I use Notebooks in 2020. Given these changes, new tools in the Notebook ecosystem emerged. I’d like to share what I like (and don’t like) about them, and how I use them in my day-to-day.
- Part III: Notebooks and the future. Here I’ll share my wishlist for Notebooks, potential gaps that the community can still fill-in, and why Notebooks are still awesome!
Got a problem on my 600m. I'm running Miranda 0.3.3.1. Whenever i goto dim my screen by pressing FN+Down Arrow Key it brings up this dialog. Shop Miranda Priestly - The Devil Wears Prada the devil wears prada t-shirts designed by LiLian-Kaff as well as other the devil wears prada merchandise at TeePublic.
To give context, a little more aboutme: as a data scientist, I alternatebetween doing analyses on notebooks and writing production code for dataproducts. My work environment is highly-collaborative so I don’t just reviewcode, I also read (and attempt to reproduce) others’ notebook analyses. Withthat said, I have a strong bias to production code and softwarebest practices, yet I still use notebooks in my day-to-day.
The data science landscape today
The field of data science is rapidly changing. We’venow entered a time where phrases like “sexiest job of the 21stcentury”and “data is the newoil”have become old and replaced by more realistic business problems and groundedtechnical challenges. I see this change as two-fold: we now need to supportboth the (1) demand for productionizing analyses and experiments, and the (2)rapid adoption of the Cloud.
Figure: A simplified framework on how to think about the advancements in
the data science process for the past years
First, the need for production, i.e., creation of data products ordeploying experiment artifacts within the software engineering lifecycle, hasgrown through the years. This is evidenced by an uptake for moreengineering type of work with the rise of machine learning engineers and datascience softwaredevelopers.Furthermore, analyses aren’t confined anymore insidepublications or charts, for there is now a growing demand for experiments to bereproduced and its artifacts to be deployed.
Next, the exponential increase of data necessitates theadoption of the Cloud. We cannot just load a 1TB dataset in pandas usingour own laptops! The popularity of tools like Docker and Kubernetesallowed us to scale our data-processing workloads at unprecedented levels.Cloud adoption means that we take care of scaling, resource provisioning, andinfrastructure when managing our workloads. However, the previous JupyterNotebook ecosystem, as much as it is a staple in the data scientist toolbox,isn’t meant for these changes:
Figure: The Notebooks we know only cover a small domain of the data scienceecosystem
As I’ve said, the Jupyter Notebook we’ve come to know isn’t meant for these changes. They’re meant for exploration, not production. They’re meant to run in a singlemachine, not a fleet of pods. However, for the past five years, the JupyterNotebook ecosystem has grown: we now have JupyterLab, several plugins, new kernelsfor other languages, and third-party tooling available at our disposal. Sure,we can still run notebooks by typing jupyter notebook in our terminals, butit’s now more than that!
This then begs the question: what are the forcesthat prompted these changes?, and how can we leverage this larger notebookecosystem to respond to the changes in data science today?
The three forces of change
Miranda Laptops For Senior Citizens
Miranda Laptops For Kids
The Jupyter Notebook ecosystem is growing, and I posit that this is driven bythree forces:
- Experiment on the Cloud: big data demands large compute and storage, something that your average consumer-grade machine will not always be capable of.
- Support for developer workflow: more and more data science teams are starting to adopt software engineering best practices—version-control, gitfow, pull requests, and more.
- Quick turnaround from analysis to production: it’s not enough to test hypotheses under controlled environments. Software written for analysis should be easily reused for prod.
Figure: The growth of the Notebook ecosystem is driven by these forces
Growing towards a more Cloud-first environment means that we can performNotebook-based tasks in machines more powerful than our own. For example,managed notebook instances enabled us to run Jupyter notebooks from a remoteserver with no-ops and setup. On the other hand, growing towards a moreProduction workflow provides us with a set of tools to endow our notebook-basedtasks with software engineering practices. We’ll see more of these tools in thenext part of this post.
Finally, note that the growth of a tool doesn’t depend on a single entity ororganization. As we will see later on, filling these gaps may stem fromindividuals who contribute third-party plugins or organizations offering managedservices from notebooks.
Conclusion
In the first part of this series, we looked into the two drivers of growth inthe data science landscape: the (1) adoption of the Cloud, and the (2)increasing demand for production. We saw that Jupyter Notebooks only fill asmall part of this ecosystem, i.e., it’s often used for exploration (notproduction) and only ran in our local machines (not in the Cloud).
Then, using that same framework, we identified three forces of change thatallowed the Jupyter Notebook ecosystem to grow: increased experimentation onthe Cloud, support for developer workflow, and quicker turnaround from analysisto production. These forces may have brought in the development of new tools,plugins, and Notebook-like products to satisfy such gaps.
For the next part of this series, I’ll talk about how we can use JupyterNotebooks given these forces of change. I’ll review some of the tools andworkflows that have helped me in my day-to-day work and side-projects.
Write review about Big Ben Computers in Westfield Miranda
Help people and share your opinion about shopping in Big Ben Computers in Westfield Miranda. Did you like it? Give ratings and write reviews.
Read 2 reviews
Beatrix - Jun 22, 2020
I think there are some high expectations of a small business, especially when it comes to IT. My experience with the staff at Big Ben Computers (now at Homebush West) is nothing but great. I went to the Apple store with my out-of-warranty Apple 6S mobile. After being made to wait for 1.5hrs was told it would cost $780! I wandered down stairs and found Big Ben. The man said it could be fixed, to come back in an hour. The charge? $20. I have taken many items to this business BUT I ASK questions like ... is there anything I should know before I let you open electronic item. They are very clear with the pitfalls. I took them a hard drive (20 years of photos) and they immediately told me if they tinkered with it they could damage the internals b/c that is how this item is wired. I am still saving for the $3K I need to take it to a specialist. So, I think Big Ben Computers are great and have found their new premises after being at Miranda Westfield for 10 years due to the sky-high rent. Maybe Apple can afford the rent given they rip consumers off with ridiculous fees but Big Ben are honest, so the honest guy cannot compete.
John - May 23, 2017
Send my partner there to replace my daughter's iPad screen. iPad was working fine except the touch screen. They wanted to sell her a battery advising her that the battery needs to be changed because it was crooked. They called a day after saying that they cannot repair it because there as an issue were the battery was soldered on the iPad board. I called them and they told me that the iPad does not turn on and that the battery smells burned. I told them that I expect at least to give it to me in the same condition that I gave it to them as it was turning on. They were very rude over the phone and they hang up on me twice! I will file a complaint to Miranda Westfield tomorrow and I will report them to Fair Trading. They only want to sell and they don't care about customer service. Stay away from them.