Enable Data a leading analytics and engineering consultancy, recently completed a successful Databricks proof-of-concept (POC) for a billion-dollar market research firm based in Chicago, Illinois. The objective was to demonstrate:
- Migration of complex Hive workloads from internal Hadoop systems to Databricks.
- Process customer data sets in a shorter time frame.
- Lower infrastructure hosting costs.
Our consultants worked with the customer to re-factor and optimize select job functions which were then deployed to a three (3) node Azure Databricks cluster. Our benchmarks resulted in decreased run-times from 5 ½ hours to about 20 minutes for the tested code. The estimate costs savings and job performance improvements represent a substantial benefit to the business unit and is being used to support a recommendation to migrate the entire product to Databricks.