Julian LaNeve is the Chief Technical Officer (CTO) at Astronomer, the driving force behind Apache Airflow and modern data orchestration to power all the pieces from AI to general analytics.

Julian does product and engineering at Astronomer where he focuses on developer experience, data observability, and AI. He’s also the writer of Cosmos, an Airflow provider for running dbt Core projects as Airflow DAGs.

He’s obsessed with all things data and open source as he spends his spare time doing hackathons, prototyping latest projects, and exploring the most recent in data.

Could you share your personal story of the way you became involved with software engineering, and worked your way as much as being CTO of Astronomer?

I’ve been coding since I used to be in middle school. For me, engineering has at all times been an important creative outlet: I can provide you with an idea and use whatever technology’s vital to construct towards a vision. After spending a while in engineering, though, I desired to do more. I wanted to grasp how businesses are run, how products are sold and the way teams are built –– and I desired to learn quickly.

I spent just a few years working in management consulting at BCG, where I worked on a wide range of projects in several industries. I learned a ton, but ultimately missed constructing products and dealing towards a longer-term vision. I made a decision to hitch Astronomer’s product management team, where I could still work with customers and construct strategies (the things I enjoyed from consulting), but could also get very hands on constructing out the actual product and dealing with technology.

For some time, I acted as a hybrid PM/engineer –– I’d work with customers to grasp the challenges they were facing and design products and features as a PM. Then, I’d take the product requirements and work with the engineering team to really construct out the product or feature. Over time, I did this with a bigger set of products at Astronomer, which ultimately led to the CTO role I’m now in.

For users who’re unfamiliar with Airflow, are you able to explain what makes it the perfect platform to programmatically writer, schedule and monitor workflows?

Apache Airflow is an open-source platform for developing, scheduling, and monitoring batch-oriented workflows. Airflow provides the workflow management capabilities which might be integral to modern cloud-native data platforms. It automates the execution of jobs, coordinates dependencies between tasks, and provides organizations a central point of control for monitoring and managing workflows.

Data platform architects leverage Airflow to automate the movement and processing of knowledge through and across diverse systems, managing complex data flows and providing flexible scheduling, monitoring, and alerting. All of those features are extremely helpful for contemporary data teams, but what makes Airflow the perfect platform is that it’s an open-source project –– meaning there’s a community of Airflow users and contributors who’re continually working to further develop the platform, solve problems and share best practices.

Airflow also has many data integrations with popular databases, applications, and tools, in addition to dozens of cloud services — and more are added every month.

How does Astronomer use Airflow for internal processes?

We use Airflow a ton! Naturally, we’ve got our own data team that uses Airflow to deliver data to the business and our customers. They’ve some pretty sophisticated tooling they’ve built around Airflow that we’ve used as inspiration for feature development on the broader platform.

We also use Airflow for some pretty untraditional use cases, nevertheless it performs thoroughly. For instance, our CRE team uses Airflow to watch the tons of of Kubernetes clusters and 1000’s of Airflow deployments we run on behalf of our customers. Their pipelines run continually to examine for issues, and if we notice any, we’ll open proactive support tickets on behalf of our customers.

I’ve even used Airflow for private use cases. My favorite (so far) was once I was moving to Latest York City. In the event you’ve ever lived here, you’ll know the rental market is crazy.  Apartments get rented out inside hours of them being listed. My roommates and I had a listing of criteria all of us agreed upon (location, variety of bedrooms, bathrooms, etc), and I built an Airflow DAG that ran every couple of minutes, pulled latest listings from various apartment listing sites, and texted me (thanks Twilio!) each time there was something latest that matched our criteria. The apartment I’m now living in was found due to Airflow!

Astronomer designed Astro, a contemporary data orchestration platform, powered by Airflow. Are you able to share with us how this tool enables corporations to simply place Airflow on the core of their data operations?

Astro enables organizations and more specifically, data engineers, data scientists, and data analysts, to construct, run, and grow their mission-critical data pipelines on a single platform for all of their data flows. It’s the only managed Airflow service that gives high levels of knowledge security and protection and helps corporations scale their deployments and liberate resources to deal with their overarching business goals.

One in every of our customers, Anastasia, a cutting-edge technology company, selected Astro to administer Airflow because they didn’t have enough time or resources to take care of Airflow on their very own. Astro works on the back end so teams can deal with core business activities, somewhat than spending time on undifferentiated activities like managing Airflow.

One in every of the core components of Astro is elastic scalability, could you define what that is and why it’s essential for cloud computing environments?

For us, this just means our ability to satisfy the compute demands of our customers without running a ton of infrastructure on a regular basis. Our customers use our platform for a wide range of use cases, nearly all of which have high compute requirements (training machine learning models, processing big data, etc). One in every of the core value propositions of Astronomer is that, as a customer, you don’t must think concerning the machines running your pipelines. You deploy your pipelines to Astro, and might expect that they work. We’ve built a set of features and systems that help scale our infrastructure to satisfy the changing demands of our customers, and it’s something we’re excited to maintain constructing upon in the longer term.

You were answerable for the Astronomer team constructing Ask-Astro, the LLM-powered chatbot for Apache Airflow. Are you able to share with us details on what’s Ask-Astro and the LLMs that power it?

Our team at Astronomer has a few of the most knowledgeable Airflow community members and we desired to make it easier to share their knowledge. To try this, we created a reference implementation of Andreessen Horowitz’s Emerging Architectures for LLM Applications, which shows probably the most common systems, tools, and design patterns they’ve seen utilized by AI startups and complicated tech corporations. We began with some informed opinions about this reference implementation and Apache Airflow also plays a central role within the architecture. Ask Astro is a real-life reference to indicate glue all the assorted pieces together.

Ask Astro is greater than just one other chatbot. The Astronomer team selected to develop the appliance within the open and commonly post about challenges, ideas, and solutions with the intention to develop institutional knowledge on behalf of the community. What were a few of the biggest challenges that the team faced?

The largest challenge was the dearth of clear best practices in the neighborhood. Because “cutting-edge” was redefined every week, it was tough to grasp approach certain problems (document ingestion, model selection, output accuracy measurement, etc). This was a key driver for us to construct Ask Astro within the open. We wanted to ascertain a set of practices for LLM orchestration that work well for various use cases so our customers and community could feel well-prepared to adopt LLMs and generative AI technologies.

It’s proven to be an important selection –– the tool itself gets a ton of usage, we’ve given several public talks on construct LLM applications, and we’ve even began working with a select group of shoppers to roll out internal versions of Ask Astro!

 What’s your personal vision for the longer term of Airflow and Astronomer?

I’m really excited concerning the way forward for each Airflow and Astronomer. The Airflow community continues to grow and at Astronomer, we’re committed to fostering its development, support and connection across teams and individuals.

With increasing demand for data-driven insights and an influx of knowledge sources, data engineers have a difficult job. We would like to lighten the load for these individuals and teams by empowering them to integrate and manage complex data at scale. Today, this also means supporting AI adoption and implementation. In 2023, like many other corporations, we focused on how we will speed up AI use for our customers. Our platform, Astro, accelerates AI deployment, streamlines ML development, and provides the robust compute power needed for next-gen applications. AI will proceed to be a magnet for us this 12 months and we’ll support our customers as latest technologies and frameworks emerge.

As well as, Astronomer’s an important place to work and grow a profession. As the info landscape continues evolving, working here gets increasingly more exciting. We’re constructing an important team here and have plenty of technical challenges to resolve. We also recently moved our headquarters to Latest York City where we will change into a fair greater a part of the tech community that exists there and we’ll be higher equipped to draw one of the best, most expert talent within the industry. In the event you’re fascinated with joining the team to assist us deliver the world’s data on time, reach out!