Vertica is the fastest, most advanced SQL analytics database, available on-premise, on Hadoop, and multiple clouds – all delivered via one unified platform.
We connected with Deepak Majeti, Sr. Systems Software Engineer Manager, Data Lakes/Security, and Pratibha Rana, Sr. Systems Software Engineer Manager, Cloud Storage and Machine Learning, and Stephen Walkauskas, Director, Vertica Engineering, to get an inside look at the company's technology, various projects, the team's culture, and more.
- Year Founded: 2005
- Number of employees: More than it feels like internally and fewer than you’d think, based the amount of work that gets done.
- Number of engineers: All of us, some more than others, all the way up to the General Manager of our business unit.
- Industry: Database and Data Analytics.
Can you share a summary of what Vertica does?
Vertica helps people use data to make decisions. The core of our product is a database, convenient for storing and analyzing data. It is “enterprise ready”, meaning it provides backup, restore, and data replication for disaster recovery, features which are important to mission-critical systems. Vertica’s capability extends well beyond that of a database, to a platform that facilitates access to numerous data sources. The Vertica platform enables users to query data that is external to the database and to integrate with numerous business intelligence tools, including data visualization ones. Vertica was the first to offer in-database machine learning, bringing convenient and scalable predictive analytics to the database.
Basically, Vertica makes it easy to work with data, including very large amounts of it. We’re talking multiple petabytes in some instances. To analyze that amount of data, whether using simple aggregations, analytic functions, or machine learning, requires distributing the work across multiple hardware nodes, whether on-premise or in the cloud. That is where the software development gets interesting, and for the engineers at Vertica, it's one of the many challenges that makes the job fun.
What are some of the different technologies that the engineering team gets to work with and at what scale?
One of the most exciting accomplishments for a software engineer is to design and build a useful feature from scratch, using basic core Vertica components. These basic components in the Vertica product were built from the ground up, largely implemented in C/C++, and gained stability over the years. Many of the product enhancements built today, including scalable machine learning algorithms, are built on top of these mature core components.
Vertica is architected for flexible deployment and integration options. The database can run in all of the major cloud service providers, in on-premise data centers, or embedded in other products. That’s right, many products embed a Vertica database as the brains behind their analytics.
We have projects under way on all major cloud vendors (AWS, GCP, Azure, Alibaba Cloud). Vertica supports a wide variety of storage systems (S3, Google Cloud Storage, ADLS G2, Minio, HDFS, Pure, Dell EMC ECS, HP Scality). Our product integrates with a large number of open source and proprietary tools, ones commonly found in a business intelligence stack (K8s, Kafka, Spark, Presto, Hive, Tensorflow, Tableau, and many more). If you are still left wanting for more, we have a mature SDK that supports extensions to Vertica in C++, Java, Python and R.
Needless to say, Vertica Engineers have ample opportunity for in-depth learning about interesting, often cutting edge technologies. A detailed understanding of other tools is necessary because it is not sufficient simply to get Vertica to “talk” to another tool. No, we need to find ways to transfer massive amounts of data in parallel, utilizing latency-hiding techniques, with fault tolerance and retry logic implemented. It isn’t easy to get two MPP systems to efficiently work together, but it is a lot of fun.
What are some of the interesting projects that the engineering team is tackling?
We have a project suitable for virtually every software engineer, from the low-level systems software to front-end development and everything in between. For example, we’re currently doing research and running experiments to understand the price-performance trade-off of running our product on ARM processors, compared to x86 ones. At the other end of the software stack, we’re rewriting our user interface to improve the user experience (UX), with a strong focus on ease of use for our customers.
We’ve never been content with existing functionality, that’s why the Vertica database has continued to thrive over the years. A few years ago we redesigned our database to not only run in the cloud but to take advantage of the cloud architecture. Now we’re working to bring the benefits provided by the cloud down to earth, implementing cloud-like features for on-premise deployments.
Vertica was the first to offer in-database machine learning and that continues to be a focal point of ours. We’re implementing a distributed version of the XGBoost algorithm for classification and regression. We support importing models into Vertica and we are integrating with TensorFlow, Python, Pandas, and other popular ML tools, so data scientists can continue to use tools that are familiar to them, and at the same time leverage the power of Vertica to quickly process massive amounts of data, without down sampling.
Performance and scale have always been a hallmark of Vertica. We have several performance-related projects underway, including ones related to network communication, memory allocation and memory management, query execution, and query optimization. A significant amount of research, design, and implementation is being done to ensure the success of these projects.
Vertica engineers are reimagining our user interfaces to significantly improve the UX and ease of use of our product. We’re in the early stages of delivering a database-as-a-service offering to our customers. Both projects will make it easier for our customers to deploy, monitor, and manage Vertica databases.
The projects I mentioned are just a sample of the exciting work being done by the Vertica R&D Team. In addition to these projects, we’re making it easier to migrate from other databases to Vertica by adding stored procedures and by improving our very useful Database Designer tool, which automatically optimizes storage to improve query performance. We’re making the database more secure with features such as encryption at rest. We’re improving our integration with other data-processing tools, such as Spark and Kafka. We’re very pleased to have come up with a design for executing queries on complex types (nested structs, arrays, and maps) that performs exceptionally well. We’re making it easier to operate a Vertica database with features such as support for Kubernetes. We continue to improve enterprise-grade features like backup, restore, and replication, and we’re continuously adding support for a growing number of storage systems and environments. The list goes on and on.
Does your engineering team have a chance to work on projects outside of their day-to-day responsibilities?
We believe engineers do their best work when working on projects of interest to them. We try, and typically succeed, to align the interests of our engineers with the priorities of our customers. We’re able to do that because the feature set of the product is vast. Chances are there’s a project at Vertica that will spark your curiosity. We develop a number of features, particular ones that involve integrating with other tools, in an open source Vertica GitHub repository.
We organize hackathons from time to time, typically two or more per year. These events run for a week or two and culminate with a lightning round of presentations.
We hold technical chalk talks every other Friday afternoon. These talks cover varied areas of interest and are presented by talented, technically-savvy folks, including our engineers, customers, professors and industry leaders.
Finally, we hire intelligent engineers, ones who are self-motivated. As they learn more about the product and how customers use it, they take on more responsibility and help set product direction. Sometimes this starts as a moonlighting project or as a hackathon project. Once a project gains momentum, it becomes the principal project for a team of engineers.What is the culture like at Vertica for the engineering team?
The culture at Vertica is amazing. Everyone who works here knows that our culture is special. We’ve maintained a start-up like environment. We work together, we get things done, and we measure ourselves on the success of our customers. Consequently, Vertica truly has a cult-like following from our customers. We are proud to have so many of them consider themselves Vertica champions. No matter where they go in their careers, they take Vertica with them! That makes us so proud and we’ll keep doing what we do to make sure the enthusiasm that our customers have for our product grows.
What can a potential employee expect during the interview process?
If you’ve worked with a talented team, you know how special that is. Our goal is to keep the bar high. To do that, the interview process needs to be thorough. At the same time, we wish to respect the time of each candidate. Therefore, we encourage candidates to ask questions and learn more about Vertica as they progress through the interview process. It’s a two-way street; when we get to the offer stage, we want a candidate to be as sure about us as we are about them.
Typically, for a full-time engineering position, the interview process starts with a conversation with a recruiter. This conversation covers some of the basics about a candidate’s experience, as well as some HR-related topics. It also serves as an opportunity for the candidate to ask questions about benefits, work environment, and so forth. The next step is a “take home” coding exercise which takes about an hour or two to complete. That is followed by a technical interview that gives both the candidate and the hiring team a chance to learn more about one another. The final stage of the interview process involves five technical interviews that are 45 minutes each, typically done sequentially with a short break near the middle.
The interview process involves a significant time commitment from both parties. However, it provides ample opportunity for us to learn a lot about one another. When a new hire starts at Vertica they are excited about the opportunity and we’re thrilled to have them join our amazing team.
Rapid Fire Q&A
What’s on tap?
Cold Brew & Bevi
Star Wars or Star Trek?
iPhone or Android?
Coffee - hot or iced?
Both (hot & cold)
Favorite employee perk?
Pies on Pi Day
What TV show describes the engineering team’s culture?
View from your office